An artificial intelligence outbound call strategy iteration optimization method and system

Abstract

The application discloses an artificial intelligence outbound call strategy iteration optimization method and system, and relates to the technical field of computers. The application builds an artificial intelligence outbound call evaluation benchmark according to multi-dimensional layering, forms a fixed evaluation benchmark by allocating a unique version and periodically freezing, and enables a target outbound call strategy and a candidate strategy to carry out simulation using completely consistent evaluation conditions, thereby solving the problem that different versions are difficult to compare fairly. Failure attribution is performed on the simulation results based on layered labels, corresponding strategy assets are located, and candidate strategies are automatically generated. Regression testing comparison is completed under the same benchmark, and retention, manual review or discarding is automatically selected according to indicators, a complete automated iteration closed loop is formed, strategy optimization is completed relying on measured data, and dependence on artificial experience is eliminated. The benchmark can be additionally provided with a sales behavior sample, so as to distinguish between two types of faults, namely action decision and speech expression, and improve the accuracy of attribution and modification. The sample is an optional enhancement item and does not affect the basic operation of the iteration closed loop.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an iterative optimization method and system for artificial intelligence outbound calling strategies. Background Technology

[0002] Existing AI-powered outbound calling systems typically rely on human experience when optimizing scripts, prompts, process rules, or node strategies. This approach suffers from the following problems: strategy tuning lacks stable and unified quantitative criteria, making fair comparisons between different strategy versions difficult; modifications to local strategies may lead to decreased effectiveness in other scenarios; the cycle for problem discovery, strategy modification, verification, and deployment is lengthy; business experts need to repeatedly listen to, analyze, and adjust parameters, resulting in low efficiency; existing evaluations often focus only on outcome metrics, lacking correlation between evaluation results and specific strategy assets, making it difficult to automatically attribute failures; unified text similarity or unified judgment standards cannot distinguish between incorrect action selection and incorrect script expression; different sales actions have different requirements for objectives, style, and length, limiting the accuracy of unified scoring rules. Summary of the Invention

[0003] This application provides an iterative optimization method and system for artificial intelligence outbound calling strategies, which at least solves the problem in related technologies that there is a lack of effective evaluation basis for strategy adjustments for artificial intelligence (AI) outbound calling systems, resulting in uncontrollable modifications to various strategies and strategy evaluations that often only focus on result indicators and lack correlation analysis between failure reasons and specific strategy assets.

[0004] This application provides an iterative optimization method for artificial intelligence outbound calling strategies, including: Establish a benchmark for evaluating AI-powered outbound calling strategies; The evaluation benchmark samples within the AI ​​outbound call strategy evaluation benchmark are stratified and stratification labels are added; A unique version number is assigned to the AI ​​outbound calling strategy evaluation benchmark, and it is frozen within the target outbound calling strategy iteration cycle to obtain the fixed evaluation benchmark for this iteration; The target outbound call strategy is input into the fixed evaluation benchmark to complete the full simulation process and obtain the benchmark evaluation results. Based on the hierarchical labels, expected behaviors, and actual dialogue results of the benchmark samples, failure samples are identified, and attribution analysis is performed on the failure samples to obtain failure attribution results. Based on the benchmark evaluation results and the failure attribution results on the fixed evaluation benchmark, candidate strategies are formed; The candidate test results are compared with the benchmark evaluation results to determine whether the candidate strategy should be used to replace the target outbound call strategy.

[0005] Furthermore, the construction of the AI ​​outbound calling strategy evaluation benchmark includes: Obtain real outbound call data, perform preprocessing, and obtain standardized outbound call session data; Based on the standardized outbound call session data, user profiles, business objectives, and manual annotation results, an AI outbound call strategy evaluation benchmark is generated. The AI ​​outbound call strategy evaluation benchmark is encapsulated to obtain result-oriented, behavior-oriented, compliance-oriented, adversarial, and sales behavior-oriented samples. Each sample contains one or more of the following: user profile, historical dialogue, business scenario, target node, user input, expected behavior, prohibited behavior, scoring rules, and strategy asset mapping relationship.

[0006] Furthermore, when the AI ​​outbound calling strategy evaluation benchmark includes sales behavior samples, the steps for constructing sales behavior samples include: Clean outbound call session data, distinguish between sales roles and customer roles within the outbound call session data, and retain the original sales call wheels; Based on context mining, the behavior name, behavior description, and judgment reason corresponding to each sales turn can be obtained; Cluster and merge semantically similar behaviors, and manually review them to generate a standard set of behaviors; Configure a behavior ID, behavior description, response style, prohibited behavior, and real-person demonstration sample for each standard sales behavior in the set of standard behaviors; The standard sales behavior is written back to the original sales turn to form real-person sales script data, and sales behavior samples are generated based on the real-person sales script data.

[0007] Furthermore, the step of assigning a unique version number to the AI ​​outbound calling strategy evaluation benchmark and freezing it within the target outbound calling strategy iteration cycle to obtain a fixed evaluation benchmark for this iteration includes: Obtain the current iteration batch number and timestamp information of the AI ​​outbound call strategy evaluation benchmark, and generate a unique version number based on the iteration batch number and the timestamp; Bind the unique version number to the AI ​​outbound call strategy evaluation benchmark and write it to a read-only repository; Obtain an AI outbound call strategy evaluation benchmark bound to a unique version number, lock the samples, hierarchical rules, and judgment thresholds within the benchmark during the target outbound call strategy iteration cycle, and generate a fixed evaluation benchmark with a freeze flag and a unique version number.

[0008] Furthermore, the step of determining failure samples based on the hierarchical labels, expected behaviors, and actual dialogue results of the benchmark samples, and performing attribution analysis on the failure samples to obtain failure attribution results includes: Obtain the benchmark evaluation results and filter out the samples that do not meet the expected behavioral indicators as failure samples; Obtain the business scenario, customer group, node, or user intent in which the failed sample occurred; Determine the failure type of the failed sample. The failure type includes one or more of the following: premature advancement, failure to answer user questions, routing error, insufficient objection handling, memory usage error, compliance risk, incorrect sales behavior selection, mismatch between sales behavior and script, and unsuitable script length or style. The failure samples are mapped to relevant strategy assets, which include one or more of the following: business strategies, standard operating procedures, decision prompts, generation prompts, memory rules, intent routing, sales behavior definitions, real-person demonstration samples, and compliance rules. Summarize failure scenarios, failure types, associated strategy assets, and defect occurrence frequency, and output the failure attribution results in a structured manner.

[0009] Furthermore, the step of forming a candidate strategy based on the benchmark evaluation results and the failure attribution results on the fixed evaluation benchmark includes: Read the defects and associated strategy assets recorded in the failure attribution results, match them with the preset strategy correction rule base, and generate multiple sets of minimum scope modification plans; each set of modification plans includes one or more of the following: modification target, modification reason, associated failure sample, strategy asset to be modified, specific modification content, expected improvement indicators, scope of impact, and potential degradation risk. Predict the scope of each modification proposal, the impact on high-weight tiered scenarios, and compliance risks. For modification proposals with excessively high risks, narrow the scope of modification or mark them for manual review. The selected modification schemes are applied to the target outbound call strategy to generate one or more candidate strategies.

[0010] Furthermore, the step of comparing the candidate test results with the benchmark evaluation results to determine whether to adopt the candidate strategy to replace the target outbound calling strategy includes: The candidate strategies are subjected to regression testing using the fixed evaluation benchmark to obtain candidate test results. The regression testing method includes incremental regression or full regression. If incremental regression is performed on the candidate strategy using the fixed evaluation benchmark, failure samples, similar samples in the same layer, and related sales behavior samples are screened using hierarchical labels. Simulation tests are then conducted on the candidate strategy to obtain incremental test data and verify whether the target defect has been repaired. If the fixed evaluation benchmark is used to perform full regression on the candidate strategy, the fixed evaluation benchmark is used to complete the full sample simulation of the candidate strategy and output the candidate test results; The overall indicators, stratified indicators, sales behavior indicators, and compliance risk indicators of the candidate test results are compared with those of the benchmark evaluation results to obtain the comparison and judgment results; If the target tiered indicator improves and the overall indicator changes within the preset range, the candidate strategy will be retained or will be subject to manual review. If the target indicator improves but other tiered indicators decline, the process will proceed to manual review or require the generation of a smaller-scale modification strategy. If the overall metric declines, the risk metric increases, or the high-weighted sample degenerates, the candidate strategy is rolled back or discarded. If multiple candidate strategies pass the test, the best one will be selected based on the magnitude of metric improvement, risk of degradation, and complexity of modification.

[0011] Furthermore, if the candidate strategy is retained or enters manual review, the method further includes A / B testing, which includes: Configure the online traffic allocation ratio between the candidate strategies and the existing online target outbound call strategies; According to the traffic allocation ratio, the candidate strategy and the target outbound call strategy are distributed to real online users, and one or more of the real-time operational data of online dialogue, conversion, objection, complaint, and risk are collected simultaneously. The real-time running data during the test period is summarized to generate online test data, and the strategy gain of the candidate strategy in the real business environment is verified based on the online test data. New problems, new customer group performance, or new failure scenarios generated by the candidate strategies described in the A / B test will be used as reference data for the next round of AI outbound call strategy evaluation benchmark upgrades.

[0012] Furthermore, the method also includes: The AI ​​outbound calling strategy evaluation benchmark is upgraded during the maintenance cycle. Each upgrade of the AI ​​outbound calling strategy evaluation benchmark creates a new version, and historical version snapshots are retained. The upgrade conditions for the AI ​​outbound call strategy evaluation benchmark include one or more of the following: cleaning duplicate or low-quality samples, correcting erroneous labels, adjusting hierarchical weights, adding new business scenarios, adding new failed samples discovered through A / B testing, adding extreme adversarial samples, adding long-tail user behavior samples, and generating simulated samples based on digital twin user models.

[0013] This application also provides an artificial intelligence outbound call strategy iterative optimization system, the system comprising: The benchmark construction module is used to build benchmarks for AI outbound calling strategies. The evaluation benchmark management module is used to stratify and add stratification labels to the evaluation benchmark samples within the AI ​​outbound call strategy evaluation benchmark. The freeze and version management module is used to assign a unique version number to the AI ​​outbound call strategy evaluation benchmark and freeze it within the target outbound call strategy iteration cycle to obtain a fixed evaluation benchmark for this iteration. The automated evaluation module is used to input the target outbound call strategy into the fixed evaluation benchmark to complete the full simulation operation and obtain the benchmark evaluation results; The failure attribution module is used to determine failure samples based on the hierarchical labels, expected behaviors, and actual dialogue results of the evaluation benchmark samples, perform attribution analysis on the failure samples, and obtain failure attribution results. The candidate strategy generation module is used to generate candidate strategies based on the benchmark evaluation results and the failure attribution results on the fixed evaluation benchmark. The effect comparison decision module is used to compare the candidate test results with the benchmark evaluation results to determine whether to adopt the candidate strategy to replace the target outbound call strategy.

[0014] This application establishes a multi-dimensional, layered AI outbound calling evaluation benchmark. By assigning a unique version and periodically freezing it, a fixed benchmark is formed, ensuring that the target outbound calling strategy and candidate strategies undergo simulations under identical evaluation conditions, thus resolving the issue of unfair comparison between different versions. Failure attribution is performed on simulation results based on layered labels, identifying corresponding strategy assets and automatically generating candidate strategies. Regression testing and comparison are then conducted under the same benchmark. Based on the performance indicators, strategies are automatically selected for retention, manual review, or rejection. This entire process forms a complete automated iterative loop, relying on real-world data for strategy optimization, eliminating reliance on human experience. The benchmark can be supplemented with sales behavior samples to differentiate between action-based decision-making and script-based failures, improving the accuracy of attribution and modification. This sample is an optional enhancement and does not affect the basic operation of the iterative loop. Attached Figure Description

[0015] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating an iterative optimization method for an artificial intelligence outbound calling strategy in one embodiment of this application; Figure 2 This is a structural block diagram of an AI outbound call strategy iterative optimization device in one embodiment of this application; Figure 3 This is an internal structural diagram of a computer device in one embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] like Figure 1 As shown, embodiments of this application provide an iterative optimization method for artificial intelligence outbound calling strategies, including the following steps: Step S1: Construct an AI outbound calling strategy evaluation benchmark, and stratify each evaluation benchmark sample within the AI ​​outbound calling strategy evaluation benchmark by adding stratification labels; Step S2: Assign a unique version number to the AI ​​outbound calling strategy evaluation benchmark and freeze it within the target outbound calling strategy iteration cycle to obtain the fixed evaluation benchmark for this iteration; Step S3: Input the target outbound call strategy into the fixed evaluation benchmark to complete the full simulation operation and obtain the benchmark evaluation results; wherein the benchmark evaluation results include target achievement rate, user exit rate, node conversion rate, objection handling success rate, compliance risk, failure reason distribution and indicator performance by business, customer group and node, and if sales behavior samples are included, quantitative indicators of sales behavior are also output. Step S4: Based on the hierarchical labels, expected behaviors, and actual dialogue results of the evaluation benchmark samples, determine the failure samples, perform attribution analysis on the failure samples, and obtain the failure attribution results; Step S5: Based on the benchmark evaluation results and the failure attribution results on the fixed evaluation benchmark, a candidate strategy is formed; Step S6: Compare the candidate test results with the benchmark evaluation results to determine whether to use the candidate strategy to replace the target outbound call strategy.

[0020] The system employs a multi-dimensional, layered approach to build AI-powered outbound calling evaluation benchmarks. By assigning a unique version and periodically freezing it, a fixed benchmark is formed, ensuring that the target outbound calling strategy and candidate strategies undergo simulations under identical evaluation conditions, thus resolving the issue of unfair comparison between different versions. Failure attribution is performed on simulation results based on layered labels, identifying corresponding strategy assets and automatically generating candidate strategies. Regression testing and comparison are then conducted under the same benchmark. Based on the performance indicators, strategies are automatically selected for retention, manual review, or rejection. This entire process forms a complete automated iterative loop, relying on real-world data for strategy optimization, eliminating reliance on human experience. The benchmark can be supplemented with sales behavior samples to differentiate between action-based decision-making and script-based failures, improving the accuracy of attribution and modification. This sample is an optional enhancement and does not affect the basic operation of the iterative loop.

[0021] The AI-powered outbound calling strategy evaluation benchmark includes one or more of the following: outcome-based, behavior-based, compliance-based, adversarial, and sales behavior-based samples. The sales behavior-based samples are used to distinguish between sales action decisions and sales script delivery. For example, the AI-powered outbound calling strategy evaluation benchmark can differentiate between business scenarios such as policy renewals, adding WeChat contacts, and follow-up calls, as well as customer groups such as high-intent users, price-sensitive users, rejection-oriented users, and silent users. The samples simulate various types of users. The hierarchical labels are used for detailed indicator statistics, clustering of similar failures, and targeted incremental regression screening. Failure samples include failure scenarios, failure types, and related strategy assets.

[0022] In this embodiment, the construction of the AI ​​outbound calling strategy evaluation benchmark includes: Acquire real outbound call data, perform preprocessing, and obtain standardized outbound call session data; preprocessing includes one or more of the following: data cleaning, deduplication, and compliance filtering; Based on the standardized outbound call session data, user profiles, business objectives, and manual annotation results, an AI outbound call strategy evaluation benchmark is generated. The AI ​​outbound call strategy evaluation benchmark is encapsulated to obtain result-oriented, behavior-oriented, compliance-oriented, adversarial, and sales behavior-oriented samples. Each sample includes user profiles, historical dialogues, business scenarios, target nodes, user inputs, expected behaviors, prohibited behaviors, scoring rules, and strategy asset mapping relationships.

[0023] In this embodiment, when the AI ​​outbound calling strategy evaluation benchmark includes sales behavior samples, the step of constructing sales behavior samples includes: Clean outbound call session data, distinguish between sales roles and customer roles within the outbound call session data, and retain the original sales call wheels; Based on context mining, the behavior name, behavior description, and judgment reason corresponding to each sales turn can be obtained; Cluster and merge semantically similar behaviors, and manually review them to generate a standard set of behaviors; Configure a behavior ID, behavior description, response style, prohibited behavior, and real-person demonstration sample for each standard sales behavior in the set of standard behaviors; The standard sales behavior is written back to the original sales turn to form real-person sales script data, and sales behavior samples are generated based on the real-person sales script data.

[0024] For example: <Identify yourself and state your purpose> Hello, I am an advisor from ZhongAn Insurance.

[0025] Customer: I want to show this to my father.

[0026] <Asking about age and identity information> How old is your uncle this year? The system takes the historical dialogue preceding the target sales turn as input, and the Action of the target turn and the actual sales script as the Gold Action and Gold Response. The input must not contain information after the target turn. The training set, validation set, and test set are divided according to the complete call identifier.

[0027] The evaluation benchmark for AI-powered outbound calling strategies also includes: Each sample is configured with an evaluation profile, which includes one or more of the following: the goals to be achieved in the current turn, the behaviors that must be demonstrated and the prohibited behaviors, the style of the speech, the recommended response length, the number of sentences, the question tendency, the real person demonstration sample, and the real person speech statistical indicators; the real person speech statistical indicators include the average length, median, quantile and question ratio based on real person data statistics.

[0028] For example, information-asking actions require using short questions and asking only one key question at a time; operation guidance actions require providing only one clear operation per round; and product explanation actions require explaining a theme related to the current problem.

[0029] In this embodiment, the step of assigning a unique version number to the AI ​​outbound calling strategy evaluation benchmark and freezing it within the target outbound calling strategy iteration cycle to obtain a fixed evaluation benchmark for this iteration includes: Obtain the current iteration batch number and timestamp information of the AI ​​outbound call strategy evaluation benchmark, and generate a unique version number based on the iteration batch number and the timestamp; Bind the unique version number to the AI ​​outbound call strategy evaluation benchmark and write it to a read-only repository; Obtain an AI outbound call strategy evaluation benchmark bound to a unique version number, lock the samples, hierarchical rules, and judgment thresholds within the benchmark during the target outbound call strategy iteration cycle, and generate a fixed evaluation benchmark with a freeze flag and a unique version number.

[0030] Specifically, the AI ​​outbound calling strategy evaluation benchmark is frozen, and the samples, labels, scoring rules, and weights cannot be modified within the target outbound calling strategy iteration cycle.

[0031] The step of inputting the target outbound calling strategy into the fixed evaluation benchmark to complete the full simulation operation and obtain the benchmark evaluation results includes: The target outbound calling strategy is compared with all the benchmark samples within the fixed evaluation benchmark to complete the dialogue simulation interaction, and the dialogue process, conversion data and risk identification data output by all simulation interaction are summarized. Calculate general tiered metrics, which include one or more of the following: target achievement rate, node conversion rate, objection handling effectiveness, user churn rate, and compliance risk. If the fixed evaluation benchmark includes sales behavior samples, then sales behavior-specific indicators are calculated. These sales behavior-specific indicators include the Top-K hit rate of standard sales behavior, the reciprocal ranking MRR, the generation probability of standard sales behavior, the hit probability of multiple sampling Pass@K, and the confounding relationship between different sales behaviors. The quality of the sales script is evaluated by combining deterministic rules with a large model evaluator. The deterministic rules check the script length, number of sentences, format, and prohibited words, while the large model evaluator evaluates the consistency, factual accuracy, and compliance of the script based on the evaluation file of standard sales behavior. All indicators are combined to obtain the benchmark evaluation result.

[0032] Action probabilities can be calculated based on a fixed Action ID or the complete Action name, avoiding the need to read only the probability of the first token.

[0033] The quality of the script is evaluated using a combination of deterministic rules and an LLM Judge. The deterministic rules calculate length, number of sentences, format, and prohibited words; the LLM Judge evaluates the consistency between the action and the script, the completion of the Gold Action, contextual coherence, telephone conversational style, factual accuracy, and compliance based on the corresponding Judge Profile.

[0034] In this embodiment, the step of determining failure samples based on the hierarchical labels, expected behaviors, and actual dialogue results of the evaluation benchmark samples, and performing attribution analysis on the failure samples to obtain failure attribution results includes: Obtain the benchmark evaluation results and filter out the samples that do not meet the expected behavioral indicators as failure samples; Obtain the business scenario, customer group, node, or user intent in which the failed sample occurred; Determine the failure type of the failed sample. The failure type includes one or more of the following: premature advancement, failure to answer user questions, routing error, insufficient objection handling, memory usage error, compliance risk, incorrect sales behavior selection, mismatch between sales behavior and script, and unsuitable script length or style. The failed samples are mapped to relevant strategy assets, which include one or more of the following: business strategy, standard operating procedures, decision prompts, generation prompts, memory rules, intent routing, sales behavior definitions, real-person demonstration samples, and compliance rules; among them, failure samples of the same business, the same customer group, and the same consensus graph are batch-collected based on hierarchical tags to locate scenario defects. Summarize failure scenarios, failure types, associated strategy assets, and defect occurrence frequency, and output the failure attribution results in a structured manner.

[0035] The system determines the failure scenarios, failure types, and related strategy assets based on benchmark stratification, expected behavior, and actual results.

[0036] For Action-type samples, failures can be further categorized as follows: Incorrect Action selection or Gold Action ranking too low; The Action tag does not match the actual wording; The action is correct, but the style, length, or contextual coherence are inadequate. The script repeatedly asks questions and performs multiple actions simultaneously. The wording contains factual errors or compliance risks.

[0037] The system maps failures to business policies, SOPs, decision prompts, generated prompts, memory rules, routing rules, action definitions, and few-shot or compliance rules, thereby distinguishing between action decision problems and expression problems.

[0038] In this embodiment, forming a candidate strategy based on the benchmark evaluation results and the failure attribution results on the fixed evaluation benchmark includes: Read the defects and associated strategy assets recorded in the failure attribution results, match them with the preset strategy correction rule base, and generate multiple sets of minimum scope modification plans; each set of modification plans includes one or more of the following: modification target, modification reason, associated failure sample, strategy asset to be modified, specific modification content, expected improvement indicators, scope of impact, and potential degradation risk. Predict the scope of each modification proposal, the impact on high-weight tiered scenarios, and compliance risks. For modification proposals with excessively high risks, narrow the scope of modification or mark them for manual review. The selected modification schemes are applied to the target outbound call strategy to generate one or more candidate strategies.

[0039] Meanwhile, the system retains the original strategy version, candidate strategy versions, modification differences, reasons for modification, associated samples, and failure attribution results for subsequent traceability.

[0040] In this embodiment, comparing the candidate test results with the benchmark evaluation results to determine whether to use the candidate strategy to replace the target outbound calling strategy includes: The candidate strategies are subjected to regression testing using the fixed evaluation benchmark to obtain candidate test results. The regression testing method includes incremental regression or full regression. If incremental regression is performed on the candidate strategy using the fixed evaluation benchmark, failure samples, similar samples in the same layer, and related sales behavior samples are screened using hierarchical labels. Simulation tests are then conducted on the candidate strategy to obtain incremental test data and verify whether the target defect has been repaired. If the fixed evaluation benchmark is used to perform full regression on the candidate strategy, the fixed evaluation benchmark is used to complete the full sample simulation of the candidate strategy and output the candidate test results; The overall indicators, stratified indicators, sales behavior indicators, and compliance risk indicators of the candidate test results are compared with those of the benchmark evaluation results to obtain the comparison and judgment results; If the target stratified indicator improves and the overall indicator changes within the preset range (without a significant decrease), the candidate strategy will be retained or will proceed to manual review. If the target indicator improves but other tiered indicators decline, the process will proceed to manual review or require the generation of a smaller-scale modification strategy. If the overall metric declines, the risk metric increases, or the high-weighted sample degenerates, the candidate strategy is rolled back or discarded. If multiple candidate strategies pass the test, the best one will be selected based on the magnitude of metric improvement, risk of degradation, and complexity of modification.

[0041] In this embodiment, if the candidate strategy is retained or enters manual review, the method further includes A / B testing, which includes: Configure the online traffic allocation ratio between the candidate strategies and the existing online target outbound call strategies; According to the traffic allocation ratio, the candidate strategy and the target outbound call strategy are distributed to real online users, and one or more of the real-time operational data of online dialogue, conversion, objection, complaint, and risk are collected simultaneously. The real-time running data during the test period is summarized to generate online test data, and the strategy gain of the candidate strategy in the real business environment is verified based on the online test data. New problems, new customer group performance, or new failure scenarios generated by the candidate strategies described in the A / B test will be used as reference data for the next round of AI outbound call strategy evaluation benchmark upgrades.

[0042] Historical strategy versions can reproduce evaluation results on historical AI outbound calling strategy evaluation benchmarks, and new strategy versions can also be compared horizontally on different AI outbound calling strategy evaluation benchmark versions.

[0043] The optimizable strategy assets in this application and invention include one or more of the following: business strategy descriptions, SOP script specifications, decision prompts, generated prompts, memory-extracted prompts, memory usage rules, intent routing rules, node-policy mapping relationships, objection handling strategies, intra-node policy orchestration, node jump rules, compliance control rules, and rules for triggering manual intervention.

[0044] The aforementioned strategy assets can be optimized hierarchically, including a global strategy layer, a node strategy layer, an intent routing layer, an objection handling layer, a script generation layer, a memory rule layer, and a compliance control layer. Strategy iteration selects the minimum necessary modifications based on failure attribution results, avoiding large-scale changes to irrelevant strategies.

[0045] In this embodiment, the method further includes: The AI ​​outbound calling strategy evaluation benchmark is upgraded during the maintenance cycle. Each upgrade of the AI ​​outbound calling strategy evaluation benchmark creates a new version, and historical version snapshots are retained. The upgrade conditions for the AI ​​outbound calling strategy evaluation benchmark include one or more of the following: cleaning duplicate or low-quality samples, correcting erroneous labels, adjusting hierarchical weights, adding new business scenarios, adding new failure samples discovered through A / B testing, adding extreme adversarial samples, adding long-tail user behavior samples, and generating simulated samples based on a digital twin user model. In the above AI outbound calling strategy iterative optimization method, an AI outbound calling evaluation benchmark is built according to a multi-dimensional hierarchical structure. A fixed evaluation benchmark is formed by assigning a unique version and periodically freezing it, ensuring that the target outbound calling strategy and candidate strategies use completely consistent evaluation conditions for simulation, solving the problem of unfair comparison between different versions. Failure attribution is performed on the simulation results based on hierarchical labels, locating corresponding strategy assets and automatically generating candidate strategies. Regression testing and comparison are then performed under the same benchmark. Based on the performance of indicators, strategies are automatically selected for retention, manual review, or discard, forming a complete automated iterative closed loop. Strategy optimization is completed based on actual test data, eliminating reliance on human experience. Sales behavior samples can be added to the benchmark to distinguish between action decision-making and script expression failures, improving the accuracy of attribution and modification. This sample is an optional enhancement and does not affect the basic operation of the iterative closed loop.

[0046] This invention constructs and freezes a fixed evaluation benchmark, evaluates the original strategy version and candidate strategy versions on the same benchmark, automatically generates candidate modifications based on failure attribution, and determines whether to retain, review or roll back through regression testing, thereby achieving automated and data-driven optimization of AI outbound call strategy assets.

[0047] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0048] In one embodiment, such as Figure 2 As shown, an AI-powered outbound call strategy iteration and optimization system 10 is provided, including: an evaluation benchmark construction module 1, an evaluation benchmark management module 2, a freeze and version management module 3, an automated evaluation module 4, a failure attribution module 5, a candidate strategy generation module 6, and an effect comparison and decision module 7.

[0049] Evaluation Benchmark Construction Module 1 is used to construct evaluation benchmarks for AI outbound calling strategies; Evaluation benchmark management module 2 is used to stratify and add stratification labels to the evaluation benchmark samples within the AI ​​outbound call strategy evaluation benchmark; The Freeze and Version Management Module 3 is used to assign a unique version number to the AI ​​outbound call strategy evaluation benchmark and freeze it within the target outbound call strategy iteration cycle to obtain a fixed evaluation benchmark for this iteration. The automated evaluation module 4 is used to input the target outbound call strategy into the fixed evaluation benchmark to complete the full simulation operation and obtain the benchmark evaluation results; The failure attribution module 5 is used to determine the failure samples based on the hierarchical labels, expected behaviors and actual dialogue results of the evaluation benchmark samples, perform attribution analysis on the failure samples, and obtain failure attribution results. Candidate strategy generation module 6 is used to generate candidate strategies based on the benchmark evaluation results and the failure attribution results on the fixed evaluation benchmark; The effect comparison decision module 7 is used to compare the candidate test results with the benchmark evaluation results to determine whether to adopt the candidate strategy to replace the target outbound call strategy.

[0050] like Figure 2 As shown, the AI-powered outbound call strategy iteration and optimization system 10 further includes an A / B testing module 8. If the candidate strategy is retained or enters manual review, the A / B testing module 8 is used for: Configure the online traffic allocation ratio between the candidate strategies and the existing online target outbound call strategies; According to the traffic allocation ratio, the candidate strategy and the target outbound call strategy are distributed to real online users, and real-time operational data on online dialogue, conversion, objection, complaint and risk are collected simultaneously. The real-time running data during the test period is summarized to generate online test data, and the strategy gain of the candidate strategy in the real business environment is verified based on the online test data. New problems, new customer group performance, or new failure scenarios generated by the candidate strategies described in the A / B test will be used as reference data for the next round of AI outbound call strategy evaluation benchmark upgrades.

[0051] like Figure 2 As shown, the AI ​​outbound call strategy iterative optimization system 10 also includes: a cleaning and calibration module 9, which is used for: The AI ​​outbound calling strategy evaluation benchmark is upgraded during the maintenance cycle. Each upgrade of the AI ​​outbound calling strategy evaluation benchmark creates a new version, and historical version snapshots are retained. The upgrade conditions for the AI ​​outbound call strategy evaluation benchmark include one or more of the following: cleaning duplicate or low-quality samples, correcting erroneous labels, adjusting hierarchical weights, adding new business scenarios, adding new failed samples discovered through A / B testing, adding extreme adversarial samples, adding long-tail user behavior samples, and generating simulated samples based on digital twin user models.

[0052] The aforementioned AI-powered outbound call strategy iteration and optimization device establishes an AI outbound call evaluation benchmark based on multiple dimensions and layers. By assigning a unique version and periodically freezing it, a fixed evaluation benchmark is formed, ensuring that the target outbound call strategy and candidate strategies undergo simulations under completely consistent evaluation conditions, thus resolving the issue of unfair comparison between different versions. Based on layered labels, failures in the simulation results are attributed to specific causes, corresponding strategy assets are located, and candidate strategies are automatically generated. Regression testing and comparison are then performed under the same benchmark. Based on the performance indicators, strategies are automatically selected for retention, manual review, or rejection. The entire process forms a complete automated iterative closed loop, relying on actual test data to optimize strategies and eliminating reliance on human experience. Sales behavior samples can be added to the benchmark to differentiate between action-based decision-making and script-based failures, improving the accuracy of attribution and modification. This sample is an optional enhancement and does not affect the basic operation of the iterative closed loop.

[0053] For a description of the features in the embodiment corresponding to the AI ​​outbound call strategy iterative optimization device, please refer to the relevant description of the embodiment corresponding to the AI ​​outbound call strategy iterative optimization method, which will not be repeated here.

[0054] The following example uses the "add WeChat" outbound call scenario during price objections. The system first constructs Benchmark_V1. This benchmark is divided into users by type, such as high-intent users, price-sensitive users, low-trust users, and rejection-type users; and into dialogue nodes, such as opening, demand confirmation, value explanation, price objection handling, and adding WeChat request.

[0055] The system ran the target outbound calling strategy Strategy_V1 on Benchmark_V1 to obtain benchmark evaluation results. The evaluation found that the conversion rate of price-sensitive users at the WeChat addition node was low, and the failure samples were mainly characterized by the AI ​​not providing sufficient explanations when users inquired about prices or service value, and prematurely asking users to add WeChat.

[0056] Based on this, the failure attribution module determined that the main reasons for the failure were "insufficient handling of price objections" and "too early in pushing for adding WeChat," and mapped this problem to the price objection handling strategy, the WeChat addition node strategy orchestration, and the generation of the Prompt.

[0057] The policy iteration agent generates candidate modification schemes based on the attribution results, for example: When a user raises a question about pricing, the AI ​​should first explain the service value or pricing information. Do not directly send a request to add the user on WeChat before the user's price inquiry has been answered; Once the user expresses a willingness to learn more, then proceed to the step of adding them on WeChat. The approach to adding someone on WeChat has shifted from a pushy, directive style to an explanatory and selective one.

[0058] The system applies this modification to the strategy asset, creating a Strategy_V2_candidate. Subsequently, the system performs incremental regression on the same Benchmark_V1, first testing the original failed samples and similar samples; if successful, a full regression is then performed.

[0059] If the Strategy_V2_candidate performs better in scenarios where price-sensitive users add WeChat accounts, and there is no significant decrease in other customer groups, nodes, and risk indicators, the system will retain the candidate version or submit it for manual review. If the version leads to an increase in the risk of complaints from other customer groups or a decrease in overall indicators, the system will automatically roll back or discard the candidate version.

[0060] The following example uses outbound calls for medical insurance sales. The system discovers and merges standard Actions from real-person calls that have resulted in sales, and constructs Benchmark_V1 with Gold Action, Gold Response and Judge Profile.

[0061] After running the Strategy_V1 outbound calling strategy on Benchmark_V1, the evaluation found that the Action hit rate of "inquiring about medical insurance and health status" was high, but the script often inquired about medical insurance, age and multiple diseases at the same time, which violated the standard of "only promoting one key point in one round".

[0062] The system attributes the problem to insufficient generation of the Prompt and Action style constraints, rather than incorrect action selection, and maps the failed samples to the generation of the Prompt, Action style description, and Few-shot.

[0063] The strategy iteration module generates multiple candidate modification schemes, such as adding single-question constraints, supplementing with real-person few-shots, or compressing the response length of query-based actions. The system selects the modification scheme with the smallest impact based on the scope of impact, forming the Strategy_V2_candidate, while retaining the reason for the modification and associated failure samples.

[0064] The system first performs incremental regression on the original failed samples and related actions, and then performs full regression on the same Benchmark_V1. If the quality of the target script reaches the improvement threshold, and other customer groups, products, actions, and compliance metrics do not exceed the degradation threshold, the candidate version is retained or enters manual review; otherwise, it is rolled back or discarded.

[0065] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the artificial intelligence outbound call strategy iterative optimization method.

[0066] In one embodiment, the electronic device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores iterative optimization data for artificial intelligence outbound calling strategies. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements an iterative optimization method for artificial intelligence outbound calling strategies.

[0067] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the artificial intelligence outbound call strategy iterative optimization method.

[0068] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0069] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the artificial intelligence outbound call strategy iterative optimization method.

[0070] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the artificial intelligence outbound call strategy iterative optimization method.

[0071] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0072] The above provides a detailed description of an AI-based outbound call strategy iterative optimization method and system. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. An iterative optimization method for artificial intelligence outbound calling strategies, characterized in that, include: An evaluation benchmark for AI outbound calling strategies is constructed, and the benchmark samples within the AI ​​outbound calling strategy evaluation benchmark are stratified and stratification labels are added; A unique version number is assigned to the AI ​​outbound calling strategy evaluation benchmark, and it is frozen within the target outbound calling strategy iteration cycle to obtain the fixed evaluation benchmark for this iteration; The target outbound call strategy is input into the fixed evaluation benchmark to complete the full simulation process and obtain the benchmark evaluation results. Based on the hierarchical labels, expected behaviors, and actual dialogue results of the benchmark samples, failure samples are identified, and attribution analysis is performed on the failure samples to obtain failure attribution results. Based on the benchmark evaluation results and the failure attribution results on the fixed evaluation benchmark, candidate strategies are formed; The candidate test results are compared with the benchmark evaluation results to determine whether the candidate strategy should be used to replace the target outbound call strategy.

2. The artificial intelligence outbound call strategy iterative optimization method according to claim 1, characterized in that, The construction of the AI ​​outbound calling strategy evaluation benchmark includes: Obtain real outbound call data, perform data preprocessing, and obtain standardized outbound call session data; Based on the standardized outbound call session data, user profiles, business objectives, and manual annotation results, an AI outbound call strategy evaluation benchmark is generated. The AI ​​outbound call strategy evaluation benchmark is encapsulated to obtain result-oriented, behavior-oriented, compliance-oriented, adversarial, and sales behavior-oriented samples. Each sample contains one or more of the following: user profile, historical dialogue, business scenario, target node, user input, expected behavior, prohibited behavior, scoring rules, and strategy asset mapping relationship.

3. The AI-based outbound calling strategy iterative optimization method according to claim 2, characterized in that, When the AI ​​outbound calling strategy evaluation benchmark includes sales behavior samples, the steps for constructing sales behavior samples include: Clean outbound call session data, distinguish between sales roles and customer roles within the outbound call session data, and retain the original sales call wheels; Based on context mining, the behavior name, behavior description, and judgment reason corresponding to each sales turn can be obtained; Cluster and merge semantically similar behaviors, and manually review them to generate a standard set of behaviors; Configure one or more of the following for each standard sales behavior in the set of standard behaviors: behavior ID, behavior description, response style, prohibited behavior, and real-person demonstration sample; The standard sales behavior is written back to the original sales turn to form real-person sales script data, and sales behavior samples are generated based on the real-person sales script data.

4. The artificial intelligence outbound call strategy iterative optimization method according to claim 1, characterized in that, The process of assigning a unique version number to the AI ​​outbound calling strategy evaluation benchmark and freezing it within the target outbound calling strategy iteration cycle to obtain a fixed evaluation benchmark for this iteration includes: Obtain the current iteration batch number and timestamp information of the AI ​​outbound call strategy evaluation benchmark, and generate a unique version number based on the iteration batch number and the timestamp; Bind the unique version number to the AI ​​outbound call strategy evaluation benchmark and write it to a read-only repository; Obtain an AI outbound call strategy evaluation benchmark bound to a unique version number, lock the samples, hierarchical rules, and judgment thresholds within the benchmark during the target outbound call strategy iteration cycle, and generate a fixed evaluation benchmark with a freeze flag and a unique version number.

5. The artificial intelligence outbound call strategy iterative optimization method according to claim 1, characterized in that, The process of identifying failure samples based on the hierarchical labels, expected behaviors, and actual dialogue results of the benchmark samples, and performing attribution analysis on the failure samples to obtain failure attribution results includes: Obtain the benchmark evaluation results and filter out the samples that do not meet the expected behavioral indicators as failure samples; Obtain the business scenario, customer group, node, or user intent in which the failed sample occurred; Determine the failure type of the failed sample. The failure type includes one or more of the following: premature advancement, failure to answer user questions, routing error, insufficient objection handling, memory usage error, compliance risk, incorrect sales behavior selection, mismatch between sales behavior and script, and unsuitable script length or style. The failure samples are mapped to relevant strategy assets, which include one or more of the following: business strategies, standard operating procedures, decision prompts, generation prompts, memory rules, intent routing, sales behavior definitions, real-person demonstration samples, and compliance rules. Summarize failure scenarios, failure types, associated strategy assets, and defect occurrence frequency, and output the failure attribution results in a structured manner.

6. The artificial intelligence outbound call strategy iterative optimization method according to claim 1, characterized in that, The step of forming a candidate strategy based on the benchmark evaluation results and the failure attribution results on the fixed evaluation benchmark includes: Read the defects and associated strategy assets recorded in the failure attribution results, match them with the preset strategy correction rule base, and generate multiple sets of minimum scope modification plans; each set of modification plans includes one or more of the following: modification target, modification reason, associated failure sample, strategy asset to be modified, specific modification content, expected improvement indicators, scope of impact, and potential degradation risk. Predict the scope of each modification proposal, the impact on high-weight tiered scenarios, and compliance risks. For modification proposals with excessively high risks, narrow the scope of modification or mark them for manual review. The selected modification schemes are applied to the target outbound call strategy to generate one or more candidate strategies.

7. The artificial intelligence outbound call strategy iterative optimization method according to claim 1, characterized in that, The step of comparing the candidate test results with the benchmark evaluation results to determine whether to adopt the candidate strategy to replace the target outbound calling strategy includes: The candidate strategies are subjected to regression testing using the fixed evaluation benchmark to obtain candidate test results. The regression testing method includes incremental regression or full regression. If incremental regression is performed on the candidate strategy using the fixed evaluation benchmark, failure samples, similar samples in the same layer, and related sales behavior samples are screened using hierarchical labels. Simulation tests are then conducted on the candidate strategy to obtain incremental test data and verify whether the target defect has been repaired. If the fixed evaluation benchmark is used to perform full regression on the candidate strategy, the fixed evaluation benchmark is used to complete the full sample simulation of the candidate strategy and output the candidate test results; The overall indicators, stratified indicators, sales behavior indicators, and compliance risk indicators of the candidate test results are compared with those of the benchmark evaluation results to obtain the comparison and judgment results; If the target tiered indicator improves and the overall indicator changes within the preset range, the candidate strategy will be retained or will be subject to manual review. If the target indicator improves but other tiered indicators decline, the process will proceed to manual review or require the generation of a smaller-scale modification strategy. If the overall metric declines, the risk metric increases, or the high-weighted sample degenerates, the candidate strategy is rolled back or discarded. If multiple candidate strategies pass the test, the best one will be selected based on the magnitude of metric improvement, risk of degradation, and complexity of modification.

8. The artificial intelligence outbound call strategy iterative optimization method according to claim 1, characterized in that, If the candidate strategy is retained or enters manual review, the method further includes A / B testing, which includes: Configure the online traffic allocation ratio between the candidate strategies and the existing online target outbound call strategies; According to the traffic allocation ratio, the candidate strategy and the target outbound call strategy are distributed to real online users, and one or more of the real-time operational data of online dialogue, conversion, objection, complaint, and risk are collected simultaneously. The real-time running data during the test period is summarized to generate online test data, and the strategy gain of the candidate strategy in the real business environment is verified based on the online test data. New problems, new customer group performance, or new failure scenarios generated by the candidate strategies described in the A / B test will be used as reference data for the next round of AI outbound call strategy evaluation benchmark upgrades.

9. The artificial intelligence outbound call strategy iterative optimization method according to claim 1, characterized in that, The method further includes: The AI ​​outbound calling strategy evaluation benchmark is upgraded during the maintenance cycle. Each upgrade of the AI ​​outbound calling strategy evaluation benchmark creates a new version, and historical version snapshots are retained. The upgrade conditions for the AI ​​outbound call strategy evaluation benchmark include one or more of the following: cleaning duplicate or low-quality samples, correcting erroneous labels, adjusting hierarchical weights, adding new business scenarios, adding new failed samples discovered through A / B testing, adding extreme adversarial samples, adding long-tail user behavior samples, and generating simulated samples based on digital twin user models.

10. An artificial intelligence outbound call strategy iterative optimization system, characterized in that, The system includes: The benchmark construction module is used to build benchmarks for AI outbound calling strategies. The evaluation benchmark management module is used to stratify and add stratification labels to the evaluation benchmark samples within the AI ​​outbound call strategy evaluation benchmark. The freeze and version management module is used to assign a unique version number to the AI ​​outbound call strategy evaluation benchmark and freeze it within the target outbound call strategy iteration cycle to obtain a fixed evaluation benchmark for this iteration. The automated evaluation module is used to input the target outbound call strategy into the fixed evaluation benchmark to complete the full simulation operation and obtain the benchmark evaluation results; The failure attribution module is used to determine failure samples based on the hierarchical labels, expected behaviors, and actual dialogue results of the evaluation benchmark samples, perform attribution analysis on the failure samples, and obtain failure attribution results. The candidate strategy generation module is used to generate candidate strategies based on the benchmark evaluation results and the failure attribution results on the fixed evaluation benchmark. The effect comparison decision module is used to compare the candidate test results with the benchmark evaluation results to determine whether to adopt the candidate strategy to replace the target outbound call strategy.

Images