Evaluation and improvement for contact center performance

Abstract

Apparatus, systems, and methods, including computer programs encoded on a computer storage medium, for assessing and improving contact center performance. In some embodiments, the system obtains historical data for the contact center system and identifies a threshold reliability level for evaluating a pairing policy for the contact center system. Based on the historical data, the system determines a region of the parameter space, where the region represents a combination of parameter values for which performance improvements derived from using the second pairing policy and the first pairing policy may be identified using at least a threshold level of reliability. The system selects a usage rate of the second pairing policy based on the determined area, and performs pairing of the contact and the seat at the usage rate of the second pairing policy based on the selected usage rate.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 435,029, filed on December 23, 2022, the entire content of which is incorporated herein by reference. Background Art

[0003] There are many scenarios where tasks need to be assigned to agents. For example, when a customer calls an airline's customer service center to request a refund for a customer's ticket, it may be necessary to assign an airline agent to the call to handle the customer's request. When selecting an agent to pair with the call, a pairing strategy can be used.

[0004] Examples of pairing strategies that can be used to pair customer calls with agents include the "First In, First Out" (FIFO) strategy, the Performance - Based Routing (PBR) strategy, and the Behavior Pairing (BP) strategy. The basic principles of the FIFO strategy, the PBR strategy, and the BP strategy are known in the art, and thus this disclosure does not provide a detailed explanation of these strategies. It should be noted that there are various types of BP strategies. For example, information on different types of BP strategies is provided in U.S. Patent Nos. 9,300,802, 9,781,269, 9,787,841, 9,930,180, and 10,757,262, all of which are incorporated herein by reference.

[0005] Pairing a task (e.g., a customer's call) with an agent using an appropriate pairing strategy is crucial not only for the efficient utilization of agent and contact center computing resources but also for customer satisfaction. For example, in a scenario where Customer #1 and Customer #2 are waiting to talk to an agent and Agents #1 and #2 are currently both available, if Customer #1 contacts the customer center before Customer #2 does, and if Agent #1 becomes available before Agent #2 does, then under the FIFO strategy, Customer #1 will be paired with Agent #1 and Customer #2 will be paired with Agent #2. However, if the purpose of Customer #1's call is to ask a question about purchasing a specific product and Agent #2 (rather than Agent #1) is the salesperson for that specific product, it would be better to pair Customer #1 with Agent #2 rather than Agent #1. Thus, in this scenario, FIFO is not the best strategy for pairing.

[0006] As described above, using an appropriate pairing strategy to pair tasks with agents is crucial not only for the efficient utilization of agents and contact center resources but also for customer satisfaction, and service providers are providing improved pairing strategies for agent-task pairing to the companies to which the agents belong (e.g., airlines). It should be noted that in the present disclosure, the companies for which the service provider provides pairing strategies (e.g., the airline in the above example) are referred to as enterprise clients.

[0007] There are various types of pairing strategies. For example, a first type of pairing strategy (e.g., such as the FIFO strategy) is configured to pair a task with an agent immediately when the task and the agent become available, and this first pairing strategy can be pre-configured in the enterprise client and / or as the default pairing strategy of the enterprise client. In contrast, a second type of pairing strategy (e.g., such as the BP strategy) can be configured to wait for more tasks and / or more agents to become available and expect an agent that is more suitable for the currently available tasks (more suitable than the currently available agents) to become available later, and / or the second type of pairing strategy can be configured to pair tasks and agents in an unordered manner.

[0008] In some examples, if the service provider's pairing strategy is of the second type (e.g., the BP strategy) and the enterprise client decides to use the service provider's pairing strategy, the enterprise client can decide to use the service provider's pairing strategy in combination with its pre-configured pairing strategy. The enterprise client can rely on the service provider to determine the percentage of use of the service provider's pairing strategy and the enterprise client's pairing strategy. SUMMARY OF THE INVENTION

[0009] Techniques for evaluating and improving the performance of a contact center system are disclosed. The contact center system can perform pairing to assign agents to inbound or outbound contacts (e.g., phone calls, Internet chat sessions, emails). The contact center system can assign contacts to available agents through an algorithm to handle these contacts. Various pairing strategies can be used for the assignment of contacts to agents, and the choice of the pairing strategy (and the proportion of time using different pairing strategies) significantly affects the performance achieved by the contact center system.

[0010] When a contact center system uses multiple pairing strategies or switches between different pairing strategies, it is difficult to measure the amount of performance change resulting from using one pairing strategy rather than another. In particular, some conditions allow the performance differences between strategies to be reliably distinguished from noise and other factors, but other conditions do not allow for clear and accurate attribution of performance differences. To achieve high performance and ensure high-quality performance analysis, the system can identify values or ranges of operating parameters that will yield high performance in the contact center system and also allow for reliable attribution of performance outcomes among the multiple pairing strategies in use.

[0011] As an example, based on historical contact-agent interaction data, the system can characterize typical conditions at the contact center system and the results achieved using a first pairing strategy. With this information, the system can evaluate a parameter space (e.g., a set of multiple variables or parameters) to determine regions of the parameter space for using a second pairing strategy that meet a set of criteria (e.g., measurement conditions providing a minimum reliability criterion). For example, the set of criteria can specify conditions under which using the second pairing strategy will improve contact center system performance, and such improvement can be reliably attributed to the use of the second pairing strategy. Through this evaluation, operating parameters of the contact center system can be set, such as the ratio or proportion of using different pairing strategies. By using parameter values from the identified regions of the parameter space, the operator of the contact center system can be highly confident that using the second pairing strategy in the selected manner will improve performance, and that the resulting data will allow the performance improvement to be distinguished from noise or other factors and can be reliably attributed to the second pairing strategy.

[0012] In some embodiments, the system facilitates determining the settings of the contact center system by generating visualizations of the parameter space and the identified regions that meet the criteria for performance improvement and measurement reliability. For example, the visualization can show incremental changes in performance that are predicted outcomes of different effectiveness levels of the second pairing strategy. The visualization can mark the identified regions where the performance improvement is significant enough to meet the set reliability criteria. The visualization can also show the impact that different ratios of using the second pairing strategy will have on performance. For example, the visualization can show how different usage rates of the second pairing strategy (e.g., 40% of the time, 60% of the time, and 80% of the time, etc.) compared to using only the first pairing strategy will yield different degrees of performance improvement. Through these and other features discussed below, the visualization can clearly show combinations of parameter values that will result in verifiable performance improvement in the contact center system, which can be attributed to the use of the second pairing strategy.

[0013] In some embodiments, visualization can be provided in a user interface that includes interactive controls (e.g., sliders, input fields, dropdown boxes, etc.) that enable a user to change the value of one or more parameters of the contact center or the pairing strategy being used. For example, the controls can enable the user to change parameters such as the number of contacts that occur, the average performance of a first pairing strategy, a threshold for verifying the reliability of a performance improvement, etc. When the user applies or changes settings using the controls, the system updates the analysis and visualization to show new regions where performance improvements can be reliably verified using the conditions set by the user. In a similar manner, user interface controls can be provided that enable the user to set a target performance level. Based on an analysis of the parameter space, the system can indicate in the visualization and / or another part of the user interface whether the target performance level can be achieved in a region of verifiable performance improvement. If the target performance level can be achieved within a reliably verified region, the system can specify the parameter values at which the target can be achieved.

[0014] In some embodiments, analysis of the parameter space can be used to develop or improve pairing strategies. The analysis can be used for prediction to identify characteristics of a pairing strategy that will yield verifiable performance improvements. Even before a particular pairing strategy is used in a contact center, the analysis can show how different levels of effectiveness of the pairing strategy will impact performance outcomes. This information can be used to determine a target level of effectiveness or to determine whether a desired level of effectiveness is feasible. For example, analysis of the parameter space might indicate that under typical conditions at a contact center, a new pairing strategy needs to be at least a minimum amount (e.g., 0.1%, 0.5%, 1.0%, 1.5%, etc.) better than an existing pairing strategy in order to achieve a statistically significant level of performance improvement. This information can inform decisions such as whether to develop a customized pairing strategy for a contact center system, when to be ready to deploy a pairing strategy, and / or what proportion of the time to use a pairing strategy. For example, some pairing strategies use trained machine learning models to pair contacts with agents. When training a machine learning model, the training can be arranged to continue until the model reaches at least the minimum effectiveness required for verifiable performance improvement or another target characteristic (e.g., a level of effectiveness that provides a desired performance level).

[0015] Analysis of the parameter space can be used to set a series of operating parameters for a contact center system to provide increasingly higher performance while remaining within conditions that allow for reliable performance attribution. In many cases, as additional interaction data is received, the pairing strategy improves over time. For example, for a pairing strategy that uses a machine learning model, the results obtained over time using the pairing strategy can be used as training data to further train and improve the machine learning model. This can lead to increased effectiveness over time, including a higher level of performance improvement relative to a baseline pairing strategy or other reference. To take advantage of the increasing effectiveness over time, as different performance levels of the pairing strategy are reached, a series of different operating parameter values can be planned to incrementally improve the overall performance of the contact center system over time. However, in addition to simply improving performance, the system can also plan a series of operating parameter values to maintain the operation of the contact center system under conditions where the performance differences of the pairing strategy used can be reliably determined. For example, when multiple pairing strategies are used in an alternating manner, a series of settings can be determined that will remain within the target area or region that provides reliable performance attribution.

[0016] For example, a series of parameter values of the contact center system can be selected such that the contact center system remains operating within the region determined in the parameter space where performance improvement can be reliably verified. For example, the plan for the contact center system can specify: (1) start using the second pairing strategy at a usage rate of 40% once the estimated effectiveness is 2% greater than the first pairing strategy; (2) switch to using the second pairing strategy at a usage rate of 60% once the effectiveness is 2.5% higher than the first pairing strategy; and (3) switch to using the second pairing strategy at a usage rate of 80% once the effectiveness is 3% higher than the first pairing strategy. This series of settings can accelerate the improvement of the overall performance of the contact center system while collecting interaction data to improve the second pairing strategy, while keeping the contact center system operating within the region of the parameter space where performance improvement can be reliably verified and the performance improvement can be attributed to the second pairing strategy. In this sense, the performance of the contact center system can be optimized within constraints to ensure that performance measurements meet a predetermined reliability standard.

[0017] In one general aspect, a method performed by one or more computers includes: obtaining, by the one or more computers, historical contact-agent interaction data of a contact center system, wherein the historical contact-agent interaction data indicates the performance of the contact center system using a first pairing strategy; identifying, by the one or more computers, a threshold reliability level for evaluating a pairing strategy of the contact center system; determining, based on the historical contact-agent interaction data, by the one or more computers, a region of a parameter space representing different combinations of parameter values of the contact center system, wherein the region spans a range of values for each of a plurality of parameters, and the region indicates a combination of parameter values for which a performance improvement obtained from a combination of using a second pairing strategy and the first pairing strategy is identifiable, which is obtained by using the second pairing strategy with at least the threshold reliability level; selecting, by the one or more computers, a usage rate of the second pairing strategy, wherein the usage rate is selected based on the determined region in which the performance improvement is identifiable, which is obtained by using the second pairing strategy with at least the threshold reliability level; and pairing, by the one or more computers, contacts and agents in the contact center system using the first pairing strategy and the second pairing strategy, wherein the usage rate of the second pairing strategy in the contact center system is based on the selected usage rate.

[0018] In some embodiments, determining the region of the parameter space includes defining a boundary of the region in the parameter space, wherein the boundary is defined by a combination of parameter values providing the threshold reliability level for identifying a performance improvement, and the boundary separates the region from a region of the parameter space representing a combination of parameter values not providing the threshold reliability level for identifying a performance improvement.

[0019] In some embodiments, determining the region includes defining a curve in the parameter space that bounds the region at a parameter combination providing the threshold reliability level for identifying a performance improvement.

[0020] In some embodiments, the parameter space includes a range of parameter values including each of the following: (i) a first parameter representing an estimated improvement level of the second pairing strategy compared to the first pairing strategy, and (ii) a second parameter quantifying an incremental change in the result at the contact center system obtained from using the second pairing strategy and the first pairing strategy in combination.

[0021] In some embodiments, determining the region of the parameter space includes determining the region based on: (i) the contact capacity at the contact center system determined from the historical contact-agent interaction data and (ii) the performance metric of the contact center system achieved when using the first pairing strategy.

[0022] In some embodiments, the method includes identifying portions in the identified region corresponding to different usage rates respectively for using a second pairing strategy.

[0023] In some embodiments, the method includes determining an estimated level of performance improvement of the second pairing strategy compared to the first pairing strategy, and selecting a usage rate for the second pairing strategy including selecting a usage rate from a plurality of different usage rates, the usage rate using the estimated level of performance improvement to result in a combination of parameter values in the identified region.

[0024] In some embodiments, the performance of the contact center system includes the amount or ratio of predetermined results occurring for contacts at the contact center system, and wherein the performance improvement includes an increase in the amount or ratio when a predetermined result occurs at the contact center system.

[0025] In some embodiments, the method includes identifying a series of usage rates to apply to different estimated improvement levels provided by the second pairing strategy compared to the first pairing strategy, wherein the series of usage rates and the estimated improvement levels provide progressively higher performance of the contact center system while remaining within the identified region, in the identified region, a performance improvement from the second pairing strategy can be identified using at least a threshold reliability level.

[0026] In some embodiments, the method includes providing user interface data for visualization that differentiates the identified region from a region in the parameter space that does not provide a threshold reliability level for identifying performance improvement.

[0027] In some embodiments, the visualization indicates a combination of parameter values in the region, the combination of parameter values corresponding to different usage rates of the second pairing strategy when used in combination with the first pairing strategy.

[0028] In some embodiments, the visualization indicates performance metrics of the contact center system for each of a plurality of different performance aspects, wherein the performance metrics are interrelated such that the visualization indicates the expected performance metrics to be achieved for each of the different performance aspects in the target region.

[0029] In some embodiments, selecting a usage rate for the second pairing strategy includes selecting a usage rate that provides at least a minimum margin from the boundary of the identified region that provides a threshold reliability level for the estimated level of performance improvement provided by the second pairing strategy relative to the first pairing strategy.

[0030] In some embodiments, the second pairing strategy involves using a machine learning model to perform pairing of contacts and agents.

[0031] Other embodiments of these aspects include corresponding systems, apparatuses, and computer programs encoded on computer storage devices, configured to perform the actions of the method. The system of one or more computers can be configured by software, firmware, hardware, or a combination thereof installed on the system, causing the system to perform the actions in operation. One or more computer programs can be configured by having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0032] Details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features and advantages of the invention will become apparent from the specification, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figures 1A to 1D is a diagram showing an example of a communication system.

[0034] Figure 2 is an example of a system for evaluating and improving contact center performance.

[0035] Figures 3A to 3G is an example of a map visualization based on contact center performance analysis.

[0036] Figures 4A to 4B is an example of a user interface showing an interactive map visualization.

[0037] Figure 5 is a flowchart showing an example of a process for evaluating and improving contact center performance.

[0038] Like reference numerals and designations in the various drawings denote like elements. DETAILED DESCRIPTION

[0039] Typical contact center algorithms assign contacts arriving at the contact center to available agents to handle those contacts. Sometimes, the contact center may have available agents and be waiting to assign them to inbound or outbound contacts (e.g., phone calls, Internet chat sessions, emails). At other times, the contact center may let contacts wait in one or more queues for an agent to become available for assignment. The contact center can use a pairing strategy to pair contacts in the queue with agents available for assignment.

[0040] In addition to FIFO, PBR, and BP strategies, some contact centers may also use various other possible pairing strategies. For example, in the longest-available agent pairing strategy, the agent who has been waiting (idle) for the longest time since the end of the agent's most recent contact interaction (e.g., call) can be selected. In the least-occupied agent pairing strategy, the agent with the lowest ratio of contact interaction time to waiting or idle time (e.g., the ratio of the time spent on a call to the time outside of the call) can be selected. In the least contact interaction by agent pairing strategy, the agent with the fewest total contact interactions or calls can be selected. In the random selection of agent pairing strategy, an available agent can be randomly selected (e.g., using a pseudorandom number generator). In the sequential tagging of agent pairing strategy, agents can be tagged sequentially, and the available agent with the next tag in the sequence can be selected.

[0041] In the case where there are multiple contacts waiting in the queue and an agent becomes available to connect to one of the contacts in the queue, various pairing strategies can be used. For example, in the FIFO or longest-waiting contact pairing strategy, the agent can preferably be paired with the contact that has been waiting the longest in the queue (e.g., the contact at the front of the queue). In the random selection of contact pairing strategy, the agent can be paired with a contact randomly selected from all the contacts in the queue or a subset of the contacts. In the priority-based routing or highest-priority contact pairing strategy, the agent may be paired with a higher-priority contact even if a lower-priority contact has been waiting in the queue for a longer time.

[0042] Contact centers can measure performance based on various metrics. For example, a contact center can measure performance based on one or more of sales revenue, sales conversion rate, customer retention rate, average handle time, customer satisfaction (based on, for example, customer surveys), etc. Regardless of what metrics or combination of metrics a contact center uses to measure performance, or what pairing strategy (e.g., FIFO, PBR, BP) the contact center uses, performance can vary over time. For example, contact center performance can vary year over year when a company shrinks or grows over time, or launches new products or contact center campaigns. Contact center performance can vary month over month when a company experiences sales cycles, such as a busy holiday sales period, or a period of heavy technical support requests after a new product or upgrade launch. Contact center performance can vary day to day, for example, if customers are more likely to call on the weekend rather than on a weekday, or more likely to call on a Monday rather than on a Friday. Intra-day contact center performance can also vary. For example, compared to a little later (e.g., 9:05), more urgent, high-value contacts may be more likely to arrive in the minute when the contact center opens (e.g., 9:00 or 9:01). Contact center performance can also vary depending on the number and quality of agents working at a given time. For example, agents working the 9:00 to 5:00 PM shift may on average perform better than agents working the 5:00 to 9:00 AM shift.

[0043] These examples of variability during certain times of the day or over longer time periods can make it difficult to attribute changes in performance during a given time period to a specific pairing strategy. For example, if a contact center uses FIFO routing for a year and the average performance of the sales conversion rate is 20%, and then switches to PBR in the second year and the average performance of the sales conversion rate is 30%, then the apparent change in performance is a 50% improvement. However, if the contact center had remained using FIFO routing instead of PBR, the contact center may not have had a reliable way to know the average performance in the second year. In the real world, at least some of the 50% performance gain in the second year may be attributable to other uncontrolled or unmeasured factors or variables. For example, the contact center may have retrained its agents or hired better-performing agents, or the company may have launched an improved product that was better received in the market. Therefore, due to the challenges associated with measuring the performance gain attributable to a new pairing strategy, a contact center may have difficulty analyzing the internal rate of return or return on investment based on switching to a different pairing strategy.

[0044] In some embodiments, the contact center system may periodically switch (or "cycle") between at least two different pairing strategies (e.g., between FIFO and PBR; between PBR and BP; between FIFO, PBR, and BP). Additionally, the result of each contact-agent interaction can be recorded along with the identity of the pairing strategy (e.g., FIFO, PBR, or BP) that has been used to assign that particular contact-agent pair. By tracking which interactions produced which results, the contact center can measure the performance attributable to the first strategy (e.g., FIFO) and the performance attributable to the second strategy (e.g., PBR). In this way, the relative performance of one strategy can be benchmarked against another. Over many periods of switching between different pairing strategies, the contact center can more reliably attribute performance gains to one strategy or the other.

[0045] Several benchmarking techniques can achieve precise and measurable performance gains by reducing noise from confounding variables and eliminating biases in favor of one or the other pairing strategy. In some embodiments, the benchmarking technique may be time-based ("epoch benchmarking"). In other embodiments, the benchmarking technique may involve randomization or counting ("inline benchmarking"). In other examples, the benchmarking technique can be a hybrid of epoch and inline benchmarking.

[0046] In epoch benchmarking, the switching frequency (or cycle duration) affects the accuracy and fairness (e.g., statistical purity) of the benchmark. For example, assume a two-year period with switching between two different strategies each year. In this case, the contact center may use FIFO with a 20% conversion rate in the first year and PBR with a 30% conversion rate in the second year and measure the gain as 50%. However, the period is too long to eliminate or otherwise control for expected performance variations. Even shorter periods, such as two months with switching between strategies each month, may be similarly affected. For example, if FIFO is used in November and PBR is used in December, some of the performance improvement in December may be attributable to increased holiday sales in December rather than PBR itself.

[0047] In some embodiments, to reduce or minimize the impact of performance changes over time, the period of using the pairing strategy before switching may be much shorter than a month (e.g., less than a day, less than an hour, less than twenty minutes). As an example, a contact center system can cycle between two pairing strategies within a 10-minute period by switching the pairing strategy every five minutes. In the first five minutes (e.g., 9:00 - 9:05 AM), a first pairing strategy (e.g., BP) can be used. After five minutes, the contact center can switch to a second pairing strategy (e.g., FIFO or PBR) for the remaining five minutes of the ten-minute period (9:05 - 9:10 AM). At 9:10 AM, a second period can start, switching back to the first pairing strategy ( Figure 1A not shown). If the period is 30 minutes, the first pairing strategy can be used for the first 15 minutes, and the second pairing strategy can be used for the last 15 minutes.

[0048] Using short, intra-hour periods (10 minutes, 20 minutes, 30 minutes, etc.), it is less likely that the benchmark will be biased towards one pairing strategy or the other based on long-term changes (e.g., year-over-year growth, monthly sales cycles). However, other factors contributing to performance changes may persist. For example, if the contact center always applies the Figure 1A shown period when opening in the morning, the contact center will always use the first strategy (BP) in the first five minutes. As mentioned above, contacts arriving at the contact center at the moment it opens may have different types, urgencies, values, or distributions of type / urgency / value compared to contacts arriving at other times of the hour or day. Therefore, the benchmark may be biased towards the pairing strategy used at the start of each day (e.g., 9:00 AM).

[0049] In some embodiments, to reduce or minimize the impact of performance changes even within short time periods, the order of using the pairing strategy within each period may change. For example, the contact center can start with the second pairing strategy (e.g., FIFO or PBR) in the first five minutes and then switch to the first pairing strategy (BP) in the next five minutes.

[0050] In some embodiments, to help ensure trust and fairness in the benchmarking system, a benchmarking schedule can be established and published, or otherwise shared in advance with contact center management or other time users. In some embodiments, direct, real-time control of the benchmarking schedule can be given to contact center management or other users, such as using a computer program interface to control the loop duration and order of the pairing strategy.

[0051] Embodiments of the present disclosure can use any of a variety of techniques to vary the ordering of using pairing strategies within each period. For example, the contact center can alternate hourly (or daily or monthly) between starting from a first ordering and starting from a different second ordering. In other embodiments, the ordering can be randomly selected for each period (e.g., approximately 50% of the periods in a given day use the first ordering and approximately 50% of the periods in a given day use the second ordering, with the ordering distribution among the periods being uniform and random).

[0052] The contact center system can use multiple pairing strategies, at different rates or in different proportions. For epoch benchmarking, multiple pairing strategies can be used at different time proportions within a period. When the BP pairing strategy uses the same amount of time as another pairing strategy within each period (e.g., 5 minutes per period), the "duty cycle" of BP is 50%. However, some pairing strategies are expected to perform better than others, despite other variables affecting performance. For example, BP is expected to perform better than FIFO. Thus, the contact center may want to use BP at a greater proportion of time than FIFO - such that more pairings are done using the higher performing pairing strategy. Accordingly, the contact center may preferably have a higher duty cycle for BP (e.g., 60%, 70%, 80%, 90%, etc.), which represents more time (or a greater proportion of contacts) being paired using the higher performing pairing strategy. As an example, the contact center system can use a 10 - minute period with a duty cycle of 80% for BP. In the first eight minutes (e.g., 9:00 - 9:08 AM), the first pairing strategy (e.g., BP) can be used. After the first eight minutes, the contact center can switch to the second pairing strategy (e.g., FIFO) for the remaining two minutes of the period (9:08 - 9:10), and then switch back to the first pairing strategy again. For another example, if a 30 - minute period is used, the first pairing strategy can be used for the first twenty - four minutes (e.g., 9:00 - 9:24 AM), and the second pairing strategy can be used for the next six minutes (e.g., 9:24 - 9:30 AM).

[0053] As another example, a contact center can have six ten-minute periods within an hour. In this example, each ten-minute period has an 80% duty cycle in favor of the first pairing strategy, and the sorting within each period starts in favor of the first pairing strategy. Within an hour, the contact center system can switch pairing strategies twelve times (e.g., at 9:08, 9:10, 9:18, 9:20, 9:28, 9:30, 9:38, 9:40, 9:48, 9:50, 9:58, and 10:00). Within an hour, the first pairing strategy is used for a total of 80% of the time (48 minutes), and the second pairing strategy is used for the other 20% of the time (12 minutes). For a 30-minute period with an 80% duty cycle (not shown), within an hour, the contact center can switch pairing strategies four times (e.g., at 9:24, 9:30, 9:48, and 10:00), and the total time using the first pairing strategy is 48 minutes, and the time using the second pairing strategy is 12 minutes.

[0054] When using in-line benchmarking, the contact center system can also set the ratio or proportion of using multiple pairing strategies. Through in-line benchmarking techniques, the pairing strategy can be selected on a per-contact basis. For example, assume that approximately 50% of the contacts arriving at the contact center should be paired using the first pairing method (e.g., FIFO), and the remaining 50% of the contacts should be paired using the second pairing method (e.g., BP). Each contact can be randomly assigned to be paired using one method or the other with a probability of 50%. The selection of the pairing strategy can be shifted or weighted to provide a higher probability of selecting a specific pairing strategy (e.g., 60%, 80%, etc.), thus setting a higher usage ratio for that pairing strategy.

[0055] In other embodiments, contacts can be sequentially assigned according to a specific time period. For example, a predetermined number of contacts (e.g., the first five, or the first ten, or the first twenty, etc.) can be assigned for the FIFO strategy, and then a predetermined number of contacts (e.g., the next five, or the next ten, or the next twenty, etc.) can be assigned for the BP strategy. Other percentages and ratios can also be used, such as 60% (or 80%, etc.) paired with the BP strategy, and the other 40% (or 20%, etc.) paired with the FIFO strategy.

[0056] From time to time, a contact may return to the contact center multiple times (e.g., a callback). In particular, some contacts may require multiple "touches" (e.g., multiple interactions with one or more contact center agents) to resolve an issue. In these cases, it may be necessary to ensure that the contact is paired using the same pairing strategy each time it returns to the contact center. If the same pairing strategy is used for each touch, then benchmarking techniques will ensure that this single pairing strategy is associated with the end result (e.g., solution) of multiple contact-agent interactions. In other cases, each time a contact returns to the contact center, it may be necessary to switch the pairing strategy so that each pairing strategy has an equal chance of being used during pairing to resolve the contact's needs and produce an end result. In other cases, it may be desirable to select a pairing strategy regardless of whether the contact has contacted the contact center multiple times about the same issue.

[0057] In some embodiments, the determination of whether the same (or different) pairing strategy should be specified for a repeated contact may depend on other factors. For example, there may be a time limit such that the contact must return to the contact center within a specified period of time in order for the previous pairing strategy to be considered (e.g., within an hour, within a day, within a week). In other embodiments, the pairing strategy used in the first interaction may be considered regardless of how much time has passed since the first interaction.

[0058] For another example, repeated contacts may be limited to a particular skill queue or customer need. Consider a contact that calls the contact center and asks to speak with a customer service agent about the contact's bill. The contact hangs up the phone and calls back a few minutes later and asks to speak with a technical support agent about a technical issue with the contact. In this case, the second call may be considered a new issue rather than a second "touch" regarding the bill issue. In this second call, it may be determined that the pairing strategy used in the first call is not relevant to the second call. In other embodiments, the pairing strategy used in the first call may be considered regardless of why the contact has returned to the contact center. The contact center system may use additional techniques for selecting pairing strategies and benchmarking performance as described in U.S. Patent No. 9,774,740, which is incorporated herein by reference.

[0059] Figure 1A An example communication system 100A is shown. In this example, the communication system 100A is a contact center system. As Figure 1AAs shown, the communication system 100A may include a central switch 110. The central switch 110 may receive incoming contacts (e.g., callers) via a telecommunications network (not shown) or support an outbound connection to a contact. The central switch 110 may include contact routing hardware and software for assisting in routing contacts between one or more contact centers, or routing to one or more private branch exchanges (PBXs) and / or automatic call distributors (ACDs) or other queuing or switching components, including other Internet-based, cloud-based, or other networked contact-agent hardware or software-based contact center solutions.

[0060] The central switch 110 may not be necessary, for example, if there is only one contact center, or if there is only one PBX / ACD routing component in the communication system 100A. If more than one contact center is part of the communication system 100A, each contact center may include at least one contact center switch (e.g., contact center switches 120A and 120B). The contact center switches 120A and 120B may be communicatively coupled to the central switch 110. In embodiments, various topologies of routing and network components may be configured to implement the contact center system.

[0061] Each contact center switch of each contact center may be communicatively coupled to a plurality (or "pool") of agents. Each contact center switch may support a certain number of agents (or "seats") logging in at one time. At any given time, logged-in agents may be available and waiting to connect to a contact, or logged-in agents may be unavailable for any of a variety of reasons, such as being connected to another contact, performing certain post-call functions, such as recording information about the call or taking an interruption.

[0062] In Figure 1A the example, the central switch 110 routes contacts to one of two contact centers via the contact center switch 120A and the contact center switch 120B, respectively. Each of the contact center switches 120A and 120B is shown as having two agents. Agents 130A and 130B may log in to the contact center switch 120A, and agents 130C and 130D may log in to the contact center switch 120B.

[0063] The communication system 100A may also be communicatively coupled to integrated services from, for example, third-party providers. In Figure 1AIn the example, the pairing node 140 can be communicatively coupled to one or more switches in the switch system of the communication system 100A, such as the central switch 110, the contact center switch 120A, or the contact center switch 120B. In some embodiments, the switches of the communication system 100A can be communicatively coupled to multiple pairing nodes. In some embodiments, the pairing node 140 can be embedded within a component of the contact center system (e.g., embedded within a switch or otherwise integrated with the switch). The pairing node 140 can receive information from a switch (e.g., the contact center switch 120A) regarding agents logged into the switch (e.g., agents 130A and 130B) and regarding contacts incoming via another switch (e.g., the central switch 110) or in some embodiments from a network (e.g., the Internet or a telecommunications network) (not shown).

[0064] The contact center can include multiple pairing nodes. In some embodiments, one or more pairing nodes can be components of the pairing node 140 or one or more switches such as the central switch 110 or the contact center switches 120A and 120B. In some embodiments, the pairing node can determine which pairing node can handle the pairing of a particular contact. For example, the pairing node can alternate between enabling pairing via a behavior pairing (BP) strategy and enabling pairing using a first-in, first-out (FIFO) strategy. In other embodiments, one pairing node (e.g., the BP pairing node) can be configured to simulate other pairing strategies.

[0065] Figure 1B A second example communication system 100B is shown. As Figure 1B shown, the communication system 100B can include one or more agent endpoints 151A, 151B and one or more contact endpoints 152A, 152B. The agent endpoints 151A, 151B can include agent terminals and / or agent computing devices (e.g., laptops, mobile phones). The contact endpoints 151A, 151B can include contact terminals and / or contact computing devices (e.g., laptops, cellular phones). The agent endpoints 151A, 151B and / or the contact endpoints 152A, 152B can be connected to contact center as a service (CCaaS) 170 via the Internet or a public switched telephone network (PSTN) depending on the capabilities of the endpoint devices.

[0066] Figure 1CAn example communication system 100C with an example configuration of CCaaS 170 is shown. For example, CCaaS 170 may include multiple data centers 180A, 180B. Data centers 180A, 180B may be physically separated, even in different countries and / or continents. Data centers 180A, 180B may communicate with each other. For example, one data center is a backup of the other; thus, in some embodiments, only one of the data centers 180A or 180B receives agent endpoints 151A, 151B and contact endpoints 152A, 152B at a time.

[0067] Each of the data centers 180A, 180B includes a network fragmentation zone (DMZ) device 171A and 171B respectively, which are configured to receive agent endpoints 151A, 151B and contact endpoints 152A, 152B that are communicatively connected to CCaaS via the Internet. The network fragmentation zone (DMZ) devices 171A and 171B may operate outside the firewall to connect to the agent endpoints 151A, 151B and the contact endpoints 152A, 152B, while the remaining components of the data centers 180A, 180B may be within the firewall (except for the telephone DMZ devices 172A, 172B, which may also be outside the firewall). Similarly, each of the data centers 180A, 180B includes a telephone DMZ device 172A and 172B respectively, which are configured to receive agent endpoints 151A, 151B and contact endpoints 152A, 152B that are communicatively connected to CCaaS via the PSTN. The telephone DMZ devices 172A and 172B may operate outside the firewall to connect to the agent endpoints 151A, 151B and the contact endpoints 152A, 152B, while the remaining components of the data centers 180A, 180B (excluding the web DMZ devices 171A, 171B) may be within the firewall.

[0068] In addition, each of the data centers 180A, 180B may include one or more nodes 173A, 173B and 173C, 173D respectively. All the nodes 173A, 173B and 173C, 173D may communicate with the web DMZ devices 171A and 171B respectively and with the telephone DMZ devices 172A and 172B respectively. In some embodiments, only one node in each of the data centers 180A, 180B may communicate with the web DMZ devices 171A, 171B and with the telephone DMZ devices 172A, 172B at a time.

[0069] Each of the nodes 173A, 173B, 173C, 173D may respectively have one or more pairing modules 174A, 174B, 174C, 174D. Similar to Figure 1AThe pairing module 140 of the communication system 100A, and the pairing modules 174A, 174B, 174C, 174D can pair contacts to agents. For example, the pairing module can alternate between enabling pairing via a behavior pairing (BP) module and enabling pairing using a first-in-first-out (FIFO) module. In other embodiments, one pairing module (e.g., the BP module) can be configured to simulate other pairing strategies.

[0070] Now turning to Figure 1D , the disclosed CCaaS communication system (e.g., Figure 1B and / or Figure 1C ) can support multi-tenancy such that multiple contact centers (or contact center operations or enterprises) can operate on a shared environment. That is, each tenant can have a separated, non-overlapping set of agents. CCaaS 170 is shown in Figure 1D as including two tenants 190A and 190B. Returning to Figure 1C , for example, multi-tenancy can be supported by node 173A that supports tenant 190A, while node 173B supports 190B. In another embodiment, data center 180A supports tenant 190A, while data center 180B supports tenant 190B. In another example, multi-tenancy can be supported by a shared machine or a shared virtual machine; such that node 173A can support both tenants 190A and 190B, and nodes 173B, 173C, and 173D are similar.

[0071] In other embodiments, the system can be configured for a single tenant within a dedicated environment such as a dedicated machine or a dedicated virtual machine.

[0072] Figure 2 is a block diagram showing an example of a system 200 that employs techniques for evaluating and improving the performance of a contact center system. In system 200, CCaaS 170 communicates with contact endpoints 152A - 152B and agent endpoints 151A - 151B via network 160, as described above for Figures 1B - 1D . System 200 also includes a computer system 210 that analyzes the performance of CCaaS 170 and provides the analysis results to the client device 202 via network 160. Based on the analysis results, the operation of CCaaS 170 can be adjusted, such as setting or changing the usage rate of various pairing strategies. Figure 2 A series of stages labeled (A) through (G) are shown, which represent the processing and data flow in system 200. These stages can be executed in the indicated order or in a different order.

[0073] In an example, computer system 210 analyzes the historical performance of CCaaS 170 and estimates how various combinations of parameter values will affect the performance of CCaaS 170. Based on the analysis, computer system 210 can create visualizations (e.g., charts, graphs, etc.) and other representations that indicate which combinations of parameter values can improve the performance of CCaaS 170. In this process, computer system 210 can identify and indicate regions of the parameter space that can improve performance (e.g., values or ranges of values of various parameters), while also allowing for the desired reliability or confidence in attributing performance improvements based on the resulting interaction data.

[0074] For example, when using a first pairing strategy, computer system 210 can analyze historical data indicative of the performance of CCaaS 170. Computer system 210 can then generate data for a map visualization that indicates the operating conditions of CCaaS 170 under which it is predicted that alternating between the first pairing strategy and a second pairing strategy will improve performance, enabling the performance improvement to be reliably attributed to the second pairing strategy. The map visualization and the associated analysis provide guidance to adjust CCaaS 170 to operate in regions of performance improvement and can reliably characterize the improvement based on the tracking results of the two pairing strategies used in an alternating manner.

[0075] In many cases, the performance of a contact center system can be improved by adding additional pairing strategies for at least some contacts (e.g., at least at certain times). For example, in a contact center system using a FIFO pairing strategy or a PBR pairing strategy, the overall performance can be improved by additionally using a BP pairing strategy during the operation of the contact center. However, as described above, when using multiple pairing strategies and when multiple factors may contaminate the performance analysis, attributing performance outcomes to different pairing strategies can be challenging, as described earlier in this document. In some cases, adding a second pairing strategy can improve the performance of the contact center system, but not enough to distinguish the improvement from noise, random variations, or other measurement artifacts. When performance cannot be reliably characterized, the uncertainty regarding the effectiveness of the pairing strategy and the operating settings most suitable for the contact center system increases. On the other hand, if the performance data reliably indicates that using a second pairing strategy provides a significant performance improvement, then this outcome provides a high level of confidence for maintaining or increasing the use of the second pairing strategy. Computer system 210 provides the function of predicting the settings and conditions that will produce result data allowing for high-confidence performance attribution, often even before the second pairing strategy is used in the contact center system.

[0076] In an example, computer system 210 helps maintain the operation of CCaaS 170 under conditions where the performance contributions of different pairing strategies can be reliably distinguished. For example, computer system 210 can predictively determine which ranges of operating parameter values will allow for reliable performance attribution and which ranges will not. This allows for the setting of operating parameters such that CCaaS 170 has a high likelihood of operation in regions or areas where performance can be reliably characterized, resulting in more predictable performance outcomes and better monitoring data to support future setting of operating parameters.

[0077] Computer system 210 can be any suitable computer system, such as one or more computers of a desktop computer, a laptop computer, or a server system (e.g., a local server, a remote server, a data center, a cloud computing system, etc.).

[0078] In short, computer system 210 obtains historical interaction data regarding the operation of CCaaS 170 and then uses that data to characterize the existing conditions (e.g., the outcomes using a first pairing strategy). Computer system 210 then estimates or predicts the effects of changing the pairing strategy being used (e.g., using a second pairing strategy and the first pairing strategy). For example, computer system 210 can generate visualizations to show how different characteristics such as the usage rates of different pairing strategies will change the performance of CCaaS 170 and which conditions will provide the desired level of reliability when attributing performance to the pairing strategy. Computer system 210 can provide the results of its analysis (including the data for the visualizations) to client device 202, and then client device 202 can set the operating parameters of CCaaS 170 (e.g., the usage rate of the pairing strategy), which are estimated to achieve a desired performance level when operating at a desired level of reliability for performance attribution in the target region. In some examples, computer system 210 can provide the operating parameters to client device 202 based on the analysis and / or the data for the visualizations. In other examples, computer system 210 itself can set the operating parameters of CCaaS 170 based on the analysis and / or the data for the visualizations. Thus, these updated parameters can improve the performance achieved by CCaaS 170, as well as the quality of performance monitoring and the confidence in performance attribution for CCaaS 170.

[0079] More specifically, in stage (A), computer system 210 obtains historical interaction data 214 regarding CCaaS 170. The historical interaction data 214 can indicate, among other things, contact capacity, contact information data, agent information data, interaction result data, contact-agent pairing data, interaction timing data, interaction pairing strategy data, abandonment rate data, and other contact center data known in the art that occurred at CCaaS 170, as well as the performance achieved at CCaaS 170 through the current pairing strategy (or current pairing strategies) used at CCaaS 170. This information can be provided in various forms, such as through aggregated metrics (e.g., totals, averages, distributions, etc.) or through information regarding individual contacts, individual agents, and the resulting contact-agent interactions.

[0080] The historical interaction data 214 can describe previous contact-agent interactions with contact logs that occurred at CCaaS 170 (e.g., calls, emails, text messages, etc.) and information regarding the results of these interactions. The historical interaction data 214 can also include contact information (e.g., website purchases, in-store purchases, etc.) regarding interactions of contacts with enterprise clients associated with CCaaS 170 that did not occur at CCaaS 170. Various types of events or conditions caused by the contacts can be tracked and indicated in the historical interaction data 214. For example, the results can indicate whether a sale occurred, the number of units sold, the value amount of the transaction, whether an existing customer was retained, the user satisfaction rating, the duration of the interaction session with the agent, whether an on-site visit was made, and whether it was determined necessary, etc. Generally speaking, the historical interaction data 214 can indicate the results or outcomes of any one of the various performance dimensions that need to be monitored or improved (e.g., conversion rate, customer satisfaction, call duration, quote / resource allocation, etc.). When appropriate, the information regarding individual contact-agent interactions can specify which pairing strategy was used to assign each contact to an agent. The historical interaction data 214 can also include information specifying the identifiers of the contacts and / or agents, profile information of the contacts and / or agents, and other information regarding the interactions that occurred.

[0081] In an example, computer system 210 stores the historical interaction data 214, which includes the interaction logs of CCaaS 170, in database 212. The historical interaction data 214 describes the contacts received over a period of time (e.g., 3 months, 6 months, one year, etc.), where the contacts were paired with agents using a first pairing strategy 262 labeled "pairing strategy 1". The historical interaction data 214 also includes information regarding the results of these contacts.

[0082] In stage (B), computer system 210 analyzes historical interaction data 214 to characterize prior performance and estimate the impact of various parameter values on future performance. For example, computer system 210 uses historical interaction data 214 to characterize the volume of prior contacts and the performance outcomes of pairings using the first pairing strategy 262. For various usage rates of the second pairing strategy 264 and the first pairing strategy 262, computer system 210 also estimates or predicts how intermittently using the second pairing strategy 264 (“pairing strategy 2”) and the first pairing strategy 262 will affect performance.

[0083] Computer system 210 can use software modules, such as analysis module 220, to evaluate the interactions of various parameters on the performance of CCaaS 170. For example, analysis module 220 can determine the typical volume and characteristics of contacts that are typically processed at CCaaS 170. For example, analysis module 220 can determine historical contact capacity 221, such as the volume of contacts that occur per unit time as indicated by historical interaction data 214 (e.g., the average number of interaction events per month). Analysis module 220 can also determine a measure of the historical performance 222 of CCaaS 170 derived from contact-agent interactions as indicated by historical interaction data 214. For example, analysis module 220 can determine a conversion rate, such as the percentage of contacts described in historical interaction data 214 that result in a sale or other desired outcome. Other types of performance, such as average customer satisfaction, average interaction duration, average interaction wait time, etc., can be calculated additionally or alternatively.

[0084] When performing an analysis of CCaaS 170, computer system 210 can also calculate or obtain other values that affect how the analysis is performed. For example, analysis module 220 can identify a reliability threshold 223 that represents the minimum reliability level required for performance attribution. For example, analysis module 220 can use statistical significance to measure reliability and thus can set a threshold for the p-value (e.g., the marginal significance level in a statistical hypothesis test). For example, to set a criterion to specify whether a monitoring condition is acceptably reliable, analysis module 220 can set a p-value threshold, such as 0.1, 0.05, etc.

[0085] Analysis module 220 can retrieve and use a predetermined p-value threshold in further analysis and can consider conditions that produce a p-value below the predetermined threshold to provide acceptable reliability. For example, the p-value can be set to 0.1 to indicate that when alternating between the first pairing strategy 262 and the second pairing strategy 264, an acceptable operating condition should be able to attribute performance improvements to the second pairing strategy 264 with a p-value less than 0.1.

[0086] Even before the pairing strategies 262, 264 are used together, the analysis can determine conditions that will allow the performance improvement to be reliably attributed based on a dataset that describes the periods during which the CCaaS 170 alternates between the pairing strategies 262, 264. To accurately characterize performance, it is generally not sufficient to simply compare the performance of the second pairing strategy 264 within a single time period with the performance of the first pairing strategy 262 within a single second different time period. For example, if the first pairing strategy 262 is only used in the first month and the second pairing strategy 264 is only used in the second month, the accuracy of the performance comparison between the two is low because the conditions experienced in the CCaaS 170 (e.g., the attributes of the received contacts and available agents) may vary significantly from one month to the next.

[0087] The conditions in the contact queue can change rapidly, often on a daily or even hourly basis. Changes due to seasonality, the type of contacts received, available agents, and other factors can all make the outcomes different from one period to another, which is separate from the differences in the capabilities of the pairing strategies 262, 264 that are intended to be measured. To limit the impact of these changes on the performance comparison, it is important to measure the performance of the pairing strategies 262, 264 within similar time periods, e.g., periods during which the CCaaS 170 cycles frequently between the two pairing strategies 262 and 264 (e.g., intraday periods with multiple cycles between the two pairing strategies). This can generate an analysis of the performance outcomes of the pairing strategies 262, 264 operating under as similar conditions as possible to minimize the amount of error introduced into the performance outcomes or the analysis. In turn, the system needs to be able to reliably attribute performance based on this type of performance data (e.g., the performance outcomes of frequent cycling between the pairing strategies 262, 264 for a period of time). However, even performance data collected in this way may have characteristics that prevent reliable performance attribution under certain conditions (e.g., the usage rate of the second pairing strategy 264 is too high, the utilization rate of the second pairing strategy 264 is too low, the performance of the pairing strategies 262, 264 is very similar, etc.). As discussed further below, the computer system 210 can perform an analysis to predict the range of operating conditions for a dataset that will yield the attributes required for reliable performance attribution, and thus the CCaaS 170 can then operate under these conditions to generate a monitoring dataset with the required attributes.

[0088] Using the historical connection capacity 221, the measure of historical performance 222 (e.g., the performance when only the first pairing strategy is used), and the reliability threshold 223, the computer system 210 can analyze how combinations of various parameters affect performance and can analyze the quality of the performance monitoring data. This can involve evaluating regions of the parameter space to determine conditions under which using the first pairing strategy 262 and the second pairing strategy 264 are predicted to meet the reliability threshold 223 and potentially other criteria, such as providing at least a minimum amount of performance improvement for the CCaaS 170. For example, the analysis module 220 can consider a multi-dimensional parameter space that includes value ranges for the estimated improvement level 224 and the incremental performance change 225.

[0089] One dimension of the parameter space can represent the estimated improvement level 224 of the second pairing strategy 264 relative to the first pairing strategy. Generally, the performance level that the second pairing strategy 264 will achieve in the CCaaS 170 is initially unknown, and thus the amount of performance improvement that the second pairing strategy 264 provides relative to the first pairing strategy 262 is also initially unknown. However, the analysis module 220 can estimate how the performance of the CCaaS 170 will be affected within the range of the estimated improvement level 224 of the second pairing strategy 264 compared to the first pairing strategy 262. For example, the range of the estimated improvement level 224 can be from 1.5% to 4% to evaluate the potential effect that the second pairing strategy 264 is more effective than the first pairing strategy 262 anywhere from 1.5% to 4%.

[0090] Another dimension of the parameter space can represent the incremental performance change 225 obtained from using the second pairing strategy 264 and the first pairing strategy 262. When the second pairing strategy 264 performs better than the first pairing strategy 262, using the second pairing strategy 264 for at least some connections will improve the overall performance of the CCaaS 170. The incremental performance change 225 can indicate the amount of performance change expected to occur, e.g., an additional 100 units, 200 units, 300 units, etc. of sales or resource allocation more than the baseline level expected when only the first pairing strategy 262 is used. The actual change in performance achieved in one case (e.g., an increase in the number of desired outcomes, a decrease in the number of undesired outcomes, etc.) will depend on various factors affecting the CCaaS 170, including the connection volume, the level or improvement of the second pairing strategy 264 relative to the first pairing strategy 262, and the usage rates of the second pairing strategy 264 and the first pairing strategy 262.

[0091] The analysis module 220 can estimate combinations of values in the parameter space that meet a reliability threshold 223 (e.g., across various levels of estimated improvement 224 and incremental performance changes 225). Typically, using a minimum reliability level for certain combinations of parameter values rather than others, the performance improvement obtained from using the second pairing strategy 264 can be identifiable and attributable. The analysis module 220 can calculate boundaries through the parameter space that demarcate a region representing parameter values with appropriate reliability and one or more regions that do not provide appropriate reliability. For example, the boundary can be a curve representing the reliability at the level of the reliability threshold 223, and then this curve demarcates a target region of the parameter space in which using the first pairing strategy 262 and the second pairing strategy 264 are predicted to improve the performance of the CCaaS 170 and allow for a reliable attribution of the performance improvement.

[0092] The analysis module 220 can also determine how different usage rates 226 of the second pairing strategy 264 affect performance and the reliability of attributing performance improvements. For example, if the relative improvement in performance of the second pairing strategy 264 relative to the first pairing strategy 262 is small, using the second pairing strategy 264 for only 10% or 20% of the interactions in the CCaaS 170 may not yield a sufficient amount of performance improvement to reliably demonstrate that the second pairing strategy 264 is more effective. However, using the second pairing strategy 264 for 40% or 50% of the interactions in the CCaaS 170 can produce a sufficient amount of incremental performance change 225 to reach the target region of the parameter space that meets the reliability threshold 223. The analysis module 220 can determine the impact of different usage rates 226 of the second pairing strategy 264 across the parameter space (e.g., across a range of estimated improvement levels 224 and incremental performance changes 225). The results can indicate how each of the different usage rates, along with other parameter values, enables the CCaaS 170 to use the first pairing strategy 262 and the second pairing strategy 264 within a target region of performance improvement that meets the reliability threshold 223.

[0093] In stage (C), the computer system 210 can generate data for visualization to represent the analysis performed by the analysis module 220. The computer system 210 can include a map generator 230 that generates a map visualization for the parameter space being analyzed. For example, the map generator 230 can generate map data 232 that, when rendered, provides a map visualization 300 ( Figure 3A), the map visualization 300 identifies target regions where the parameter values provide performance improvement while meeting the reliability threshold 223. The map data 232 can encode the information of the map visualization 300 in any suitable form, such as image data (e.g., bitmap data, vector graphics, etc.), markup languages (e.g., HTML, XML, etc.), documents, data series to be plotted, etc.

[0094] Reference Figure 3A , the map visualization 300 can provide a two-dimensional chart or graph showing at least a portion of the analyzed parameter space. The map visualization 300 is based on the analysis of the analysis module 220, which can estimate that contacts will continue to occur in the CCaaS 170 in the quantity or frequency indicated by the historical contact capacity 221, which is represented by the estimated call quantity 302 in this example (in other examples, this can be the estimated number of interactions, etc.). The analysis module 220 can also estimate that the first pairing strategy 262 will yield a performance as indicated by the historical performance 222, which is represented as the conversion rate (CR) 304 of the first pairing strategy 262 in the example. This conversion rate is labeled "Off CR" to indicate that this is the expected conversion rate when the second pairing strategy 264 is turned off or disabled, thus using only the first pairing strategy 262. Using this baseline information set, the analysis module 220 can determine a combination of values for the estimated improvement level 224 and the incremental performance change 225 (e.g., change in results) that will provide a reliability metric meeting the reliability threshold 223.

[0095] Figure 3A The example has a horizontal axis 310 and a vertical axis 320. The horizontal axis 310 spans a range of values of the estimated improvement level 224, which represents the level of performance improvement provided by the second pairing strategy 264 relative to the first pairing strategy 262. For example, the horizontal axis 310 shows the percentage of estimated gain in performance that the second pairing strategy 264 can provide (e.g., from 1.5% to 4%).

[0096] The vertical axis 320 represents the range of possible incremental performance changes 225. Figure 3A The example shows performance measured in units of sales, so the vertical axis 320 indicates the increase in the number of sales units obtained from using the second pairing strategy 264 together with the first pairing strategy 262, rather than using only the first pairing strategy 262. Using only the first pairing strategy 262 is expected to result in a baseline sales level (e.g., at a rate determined from the historical interaction data 214). The baseline level is represented by zero incremental additional sales, and each higher value on the vertical axis 320 represents the net increase in sales units due to using the second pairing strategy 264 to pair some contacts.

[0097] As described above, the analysis module 220 can determine the boundary 330 through the parameter space where the reliability metric is equal to the reliability threshold 223 (e.g., the p-value is equal to 0.1). In an example, the boundary 330 is a curve that provides a boundary for the target region 332, in which the performance improvement is attributed to the second pairing strategy 264 with the required reliability level (e.g., the p-value is 0.1 or less). The target region 332 can span a range of values of the estimated improvement level 224 and a range of values of the incremental performance change 225. The target region 332 represents a set of operating conditions for the desired operation of the CCaaS 170, e.g., a region where a combination of parameter values provides a performance improvement that can be reliably attributed to the second pairing strategy 264.

[0098] The map visualization 300 differentiates the target region 332 where the reliability threshold 223 is met from another region 334 where the reliability threshold 233 is not met. In other words, in the region 334, the p-value is greater than 0.1, so the performance contribution of the second pairing strategy 264 cannot be distinguished from the results of the first pairing strategy 262 with sufficient reliability.

[0099] The map visualization 300 also includes elements representing the impact of different usage rates 226 on the resulting performance and reliability metrics. For example, the map visualization 300 shows lines passing through the parameter space to represent each of the various different usage rates 226 of the second pairing strategy 264, e.g., 226a of 20%, 226b of 40%, 226c of 60%, 226d of 80%, and 226e of 90%. These usage rates 226 represent the proportion of time the second pairing strategy 264 is turned on or active, or the proportion of contacts assigned using the second pairing strategy 264.

[0100] The lines representing the usage rate 226 and the boundary 330 of the target region 332 show how the CCaaS 170 achieves various operating results. For example, if the second pairing strategy 264 provides an estimated gain of 2.0%, the point on the 20% usage rate line is outside the target region 332 and instead within the region 334, indicating insufficient reliability metrics. However, for the same estimated gain of 2.0%, the 40% usage rate line will provide the required reliability level. Additionally, for an estimated gain of 2.0%, if it is desired to increase the number of sales units by at least 400, then a 60% usage rate will provide this increase, while a 40% usage rate will not, and an 80% or 90% usage rate will also not provide the required reliability level.

[0101] Refer again to Figure 2, in stage (D), computer system 210 sends map visualization 300 and / or map data 232 to client device 202 via network 160. Client device 202 then renders map data 232 and displays map visualization 300 on user interface 204. For example, map visualization 300 can be presented in a web browser, a document viewer, a local application, etc. In some embodiments, computer system 210 provides an interface for authorized users (e.g., administrators) to request and receive information about CCaaS 170 using client device 202 via the network. For example, computer system 210 can provide a web page or a web application including controls and a user interface to request and view information such as map visualization 300. As another example, computer system 210 can provide an application programming interface (API) that enables client device 202 to request and receive map data 232 for map visualization.

[0102] In addition to providing map data 232, computer system 210 can also provide other results of the performed analysis. For example, computer system 210 can recommend one or more parameter values, such as the usage rate of the second pairing strategy 264, to be applied at CCaaS 170. Computer system 210 can obtain a value indicating the estimated improvement level of the second pairing strategy 264 relative to the first pairing strategy 262. Based on the analysis used to generate map data 232, computer system 210 can determine one or more usage rates that provide the required reliability level at the estimated improvement level. Computer system 210 can also determine and provide the level of incremental performance change expected for the selected usage rate. For example, for an estimated gain of 2.0%, computer system 210 can recommend (1) a 40% usage rate with an expected increase of 240 sales units, and / or (2) a 60% usage rate with an expected increase of 420 sales units.

[0103] Using the same principle, computer system 210 can also receive queries from client device 202 and can generate and provide results. For example, computer system 210 can be configured to process queries that request the minimum usage rate, the maximum usage rate, or the usage rate range that meets the reliability threshold 233 for a specific estimated gain level. As another example, computer system 210 can be configured to process queries that request parameter values that can meet the minimum estimated gain or provide a certain amount of performance improvement for the reliability threshold 233.

[0104] As further discussed below, the map visualization 300 can be provided on a user interface with interactive controls (e.g., input fields, sliders, etc.) for interacting with or adjusting the visualization 300. For example, the controls can enable a user of the client device 202 or the computer system 210 to plot different combinations of parameter values on the visualization 300 to indicate whether they fall within or outside the target region 332. Similarly, the controls can allow the user to change the parameter values used to perform the analysis (e.g., change the expected contact capacity, the expected baseline performance level of the first pairing strategy 262, the reliability threshold, etc.). The computer system 210 can update the analysis based on the received input and can provide updated map data 232 for a new version of the map visualization 300 based on the user-specified parameters.

[0105] In stage (E), a user of the client device 202 can specify settings for the CCaaS 170 to adjust how the CCaaS 170 operates. In some examples, a user of the computer system 210 specifies settings for the CCaaS 170 to adjust how the CCaaS 170 operates. For example, based on the information in the map visualization 300, a user of the client device 202 or the computer system 210 can select the usage rate of the second pairing strategy 264. As an example, the user can select to start using the second pairing strategy 264 at a usage rate of 40% (e.g., 40% of the time or 40% of the contacts), while using the first pairing strategy 262 at a usage rate of 60% (e.g., the remaining 60% of the time or 60% of the contacts). The pairing settings 250 specified by the user are provided to the CCaaS 170 from the client device 202 or the computer system 210 via the network 160. As another example, if the computer system 210 recommends one or more usage rates for the second pairing strategy 264, the user can use the user interface 204 to confirm or approve the recommended usage rate, and the client device 202 or the computer system 210 can send the settings to the CCaaS 170 in response.

[0106] In stage (F), the CCaaS 170 receives the pairing settings 250 and adjusts its operation accordingly. In an example, the CCaaS 170 includes a pairing strategy selector 260 that manages the alternating use of multiple pairing strategies. The pairing strategy selector 260 can cycle between multiple pairing strategies using time-based switching (e.g., epoch benchmarking) or randomization- or count-based switching (e.g., inline benchmarking). The pairing strategy selector 260 sets the desired usage rates, e.g., 60% for the first pairing strategy 262 and 40% for the second pairing strategy 264. As a result, the pairing strategy selector 260 continues to operate the two pairing strategies 262, 264 based on the indicated proportions or ratios.

[0107] When the CCaaS 170 uses these policies 262, 264 to assign agents to contacts, it tracks the results generated from the interactions so that the performance of the CCaaS 170 can be measured. These assignments result in the creation of additional interaction data 270, which represents a record of the contact-agent interactions during the time period when the CCaaS 170 cycles between the policies 262, 264. Based on the analysis performed by the computer system 210, the usage rates of the pairing policies 262, 264 are set to levels that are expected to allow reliable performance attribution. Thus, these assignments based on the selected usage rates help generate high-quality data in which performance improvements can be reliably attributed and the machine learning model 236 itself can be trained, as further discussed herein. For example, the additional interaction data 270 indicates performance outcomes over a period of time. Since the usage rate has been set to operate under the conditions in the target region 332 (see Figure 3A ), the performance improvements achieved using the second pairing policy 264 over a period of time can be reliably determined and distinguished from the performance of the first pairing policy 262 over the same period of time.

[0108] In some embodiments, the second pairing policy 264 employs a machine learning model 236 to perform the pairing of contacts and agents. The machine learning model 236 can initially be generated or trained based on the historical interaction data 214 to learn the characteristics of pairings that lead to high performance (e.g., high sales, low error rate, lower call duration, higher customer satisfaction scores, etc.). The machine learning model 236 can be any suitable type of model, such as a neural network, classifier, decision tree, support vector machine, etc.

[0109] In an example, the computer system 210 includes a model training module 234, which can generate and train machine learning models to best fit each individual contact queue processed by the CCaaS 170. For example, there can be separate contact queues for the sales department, technical support, and customer service. For each of the three queues, a separate set of historical interaction data can be extracted and a separate machine learning model can be trained. To initially generate the machine learning model 236, the model training module 234 can use the pairings and outcomes from the historical interaction data 214 as training data. The model training module 234 can use any suitable training algorithm, such as gradient descent, Newton's method, conjugate gradient, Levenberg-Marquardt algorithm, etc.

[0110] Over time, as more interaction data becomes available, computer system 210 can further train the machine learning model 236 such that it can identify pairings that result in higher performance of the CCaaS 170. For example, in phase (G), computer system 210 (or another system) uses additional interaction data 270 to update and improve the machine learning model 236. Starting from a version of the machine learning model 236 trained based on historical interaction data 214, the model training module 234 performs further training to update the machine learning model 235 based on pairing examples in the additional interaction data 270. This process allows the machine learning model 236 to improve over time. Additionally, as the contact type changes and the behavior or preferences of the contacts change, repeated or continuous training of the machine learning model 236 enables the machine learning model 236 to be updated for new trends and patterns that emerge over time. After the machine learning model 236 is updated, the updated version 236 of the machine learning model is provided to the CCaaS 170 for use in the second pairing strategy 264.

[0111] Over time, as more interaction data becomes available for training the machine learning model 236, the performance of the machine learning model 235 and the pairing strategy 264 generally improves. Computer system 210 or an administrator can use this feature to plan a series of different operating parameter values for the CCaaS 170. For example, different levels of utilization rates can be determined for the second pairing strategy 264, as well as the conditions or criteria that vary between the utilization rates. This can provide a clear path or sequence of milestones for specifying when it is appropriate to increase the utilization rate of the second pairing strategy 264. For example, the planned levels can specify an initial utilization rate of 40%, and then switch to a higher utilization rate of 50% once it is determined that the second pairing strategy 264 provides at least a 2.5% performance improvement relative to the first pairing strategy 262. Additionally, once an improvement of at least 3.0% is achieved, a third utilization rate of 60% can be planned. Each of these parameter combinations can be selected to keep the conditions within the target region 332 that provides a predetermined level of reliability. Over time, the tracked performance can be used to determine the level of improvement actually provided by the second pairing strategy 264. Based on the calculated level of improvement, when the corresponding improvement levels (e.g., 2.0%, 2.5%, and 3.0%) are reached, the plan can be implemented to automatically change the utilization rate of the second pairing strategy 264 at the CCaaS 170 (e.g., from 40% to 50% to 60%). This level allows confidence in the effectiveness of the second pairing strategy 264 to be established first, and also allows for further improvement of the overall performance, as higher improvement levels and higher utilization rates contribute to phased optimization of the performance of the CCaaS 170.

[0112] In Figure 2 and Figure 3AIn the example, the performance metric tracked, analyzed, and presented in the map visualization 300 is the number of sales. The computer system 210 can use the same techniques to perform analysis on other aspects of performance and generate visualizations, such as, customer satisfaction ratings, customer call wait times, call durations, etc.

[0113] Figures 3A to 3G illustrates examples of various visualizations that the computer system 210 can generate based on the analysis discussed regarding Figure 2 the analysis.

[0114] As described above, Figure 3A illustrates an example of the map visualization 300 that shows how various combinations of parameter values affect performance in terms of sales units. The horizontal axis 310 shows the range of values of the percentage improvement that the second pairing strategy 264 provides more than the first pairing strategy 262. The vertical axis 320 shows the range of values of sales units as a marginal increase over using the first pairing strategy 262 alone. The boundary 330 defines the target region 332 where conditions allow the performance improvement obtained from using the second pairing strategy 264 (e.g., the marginal increase along the vertical axis 320) to be attributed in a manner that meets the reliability threshold 223. The predicted outcomes achieved from various usage rates 226 are shown as lines 226a - 226e.

[0115] Figure 3B illustrates an example of how the visualization 300 can be used to predict the results of specific operating conditions. For example, if it is predicted that the second pairing strategy 264 provides 2% more improvement than the first pairing strategy 262, this represents the range of outcomes shown by the vertical line 340. The intersections of the lines 226a - 226e with the vertical line 340 show how different usage rates are predicted to result in different amounts of incremental increases in sales. For example, the line 226b representing a 40% usage rate intersects the line 340 at point 342, which shows that using the second pairing strategy at a 40% usage rate will provide 300 more sales units of incremental increase than using the first pairing strategy 262 alone.

[0116] The positions where the utilization rate lines 226a - 226e intersect the vertical line 340 also show whether different utilization rates will provide the required reliability level when there is an estimated improvement level of 2%. For example, the point 342 is within the target region 332, indicating that the reliability threshold 223 is met. The intersection point of the utilization rate line 226c representing a 60% utilization rate also falls within the target region 332. However, the intersection points of the utilization rate lines 226a, 226d, 226e fall outside the target region, indicating that utilization rates of 20%, 80%, and 90% will not meet the reliability threshold 223 (e.g., a p - value greater than 0.1 will be obtained). In other words, for an estimated gain of 2%, the utilization rate can be increased from 40% to 60% and still provide a p - value not exceeding 0.1, but the utilization rate should not be increased to 80% because statistical significance will be lost and the gain cannot be distinguished from sampling noise.

[0117] Figure 3C A map visualization 300C is shown, which is based on the performance level of a different first pairing strategy 262 than that reflected in the visualization 300. The map visualization 300C is based on a performance level of, for example, a 6% conversion rate for the first pairing strategy 262, rather than the 5% conversion rate used for the visualization 300. This performance difference can be set based on, for example, the measured performance change for the first pairing strategy 262, historical interaction data from different queues in the contact center, to simulate the effects of different performance levels, or other differences in the contact center state or environment.

[0118] The performance level used varies from 5% to 6%, resulting in a change in the region of the parameter space that will meet the reliability threshold 223. For example, based on the analysis of the computer system 210, the target region 332c for acceptable reliability is defined by the boundary 330c (rather than the boundary 330 that defines the target region 332 in the visualization 300). Additionally, the scale of the vertical axis 320 has been adjusted to show how the increased performance of the first pairing strategy 262 will result in incremental increases in different levels of sales units. For example, the intersection point 342c (which represents a 2% estimated improvement with the second pairing strategy 264 and a 40% utilization rate) shows an incremental increase of 360 sales units. Due to the change in the conversion rate from 5% to 6%, the predicted additional 360 units is greater than the 300 additional units with the same 2% estimated improvement and 40% utilization rate.

[0119] The visualization 300c shows additional results for the changed target region 332c. For example, at an estimated improvement level of 2%, the intersection point of the 80% utilization rate line 226d and the line 340 is within the target region 332c. This shows that using an 80% utilization rate is expected to meet the reliability threshold 233, even Figure 3BUnder the conditions shown in Visualization 300, 80% utilization is expected not to meet the reliability threshold 223. This shows that with an estimated 2% improvement, 80% utilization can be used to produce 720 additional units and still meet the reliability threshold.

[0120] Generally, as shown by the examples in Figure 3B and 3C The analysis of computer system 210, as reflected in the generated visualizations 300, 300c, can reveal the conditions under which CCaaS 170 can operate in different scenarios and with different expectations or estimates of CCaaS 170 and pairing strategies 262, 264. In addition to or instead of different performance estimates (e.g., conversion rates) for the first pairing strategy 262, computer system 210 can also perform analyses and generate visualizations for different contact capacities (e.g., 500,000 contacts, 600,000 contacts, etc.).

[0121] Figure 3D An example of a map visualization 300 is shown that has a plan or roadmap 350 for setting a series of operating parameters for CCaaS 170. The analysis of computer system 210 can reveal how to set the utilization rate to improve overall performance, e.g., by gradually increasing the number of upsell units while remaining within the target region 332 that meets the reliability threshold 223. The map visualization 300 shows various operating points remaining within the target region 332, and the position along the vertical axis 320 shows the amount of additional units expected to result from operations at those points.

[0122] More specifically, roadmap 350 shows a series of parameter values, which are shown together with corresponding points 361 to 364 on visualization 300. Roadmap 350 shows that as the performance of the second pairing strategy 264 improves over time (e.g., due to refinement based on the collected data, further machine learning training, as described elsewhere herein, etc.), the usage rate also increases. As indicated by point 360, roadmap 350 indicates that the expected improvement of the second pairing strategy 264 is 2%, and a usage rate of 40% should be used. Roadmap 350 indicates that the usage rate of 40% should continue to be used until the second pairing strategy 264 can provide an improvement of 2.5% (represented by point 361), and in response, the usage rate should increase to 80% (represented by point 362). Roadmap 350 indicates that CCaaS 170 should continue to operate at a usage rate of 80% of the second pairing strategy 264 until the estimated improvement reaches 3.0% (represented by point 363), and then the usage rate should increase to 90% (represented by point 364). This sequence of usage rates provides an ever-increasing level of performance, which is represented by the increasing additional unit amounts along the progression from point 360 to point 364. During this process, the operation of CCaaS 170 remains within the target region 332 that is expected to meet the reliability threshold 223 for performance attribution.

[0123] Figure 3D An exemplary roadmap 350 is shown. As can be readily determined by those skilled in the art, various other feasible roadmaps can be determined based on the target region 332 and the region 334 (not shown) that does not meet the reliability threshold 223.

[0124] In some embodiments, computer system 210 can determine the roadmap or series of operating parameters to use, as well as the corresponding conditions or criteria for changing parameters such as the usage rate. The selected operating points can be restricted by constraints or preferences specified by the user, such as a preference for limiting the number of usage rate changes (e.g., no more than 5 changes, no more than 3 changes, etc.), a constraint to reach a minimum additional unit level predicted at the start of the sequence, a constraint to reach at least a target level by the end of the sequence (or at another point in the sequence), etc.

[0125] Once the roadmap 350 is defined, whether input by the computer system 210 or by an administrator of the client device 202, the CCaaS 170 or a connected system such as the computer system 210 can evaluate the achieved performance results and change the operation of the CCaaS 170 accordingly. For example, after the CCaaS 170 operates using the second pairing strategy 264 at a usage level of 40% (as indicated by point 360), the actual performance improvement achieved can be calculated regularly. When the performance results indicate a gain of 2.5% or higher, the usage rate of the second pairing strategy 264 can be automatically adjusted from 40% to 80%. In other words, the CCaaS 170 or another system can detect when the operation of the CCaaS 170 reaches the condition represented by point 361 and adjust the CCaaS 170 settings to operate under the condition represented by point 362. Since the operation of the CCaaS 170 at 40% usage rate provides the required level of reliability, the performance gain obtained due to the use of the second pairing strategy 264 can be reliably quantified and attributed to the use of the second pairing strategy 264, and there is a high confidence to justify increasing the use of the second pairing strategy 264. In a similar manner, other conditions or criteria of the roadmap (such as a threshold improvement level, a threshold additional unit quantity, etc.) can be checked and used to trigger changes to the applied usage rate or other operating characteristics of the CCaaS 170.

[0126] Figure 3E Another example of the map visualization 300 is shown, where an additional vertical axis scale 370 is superimposed to display additional performance metrics. In addition to showing additional units on the vertical axis, one or more other performance metrics can be associated with the additional unit quantity and presented in the visualization. In the example, the scale 370 shows the revenue (in thousands of dollars) corresponding to different levels of additional units. In the example, there is a fixed or linear correspondence between the units and the revenue. However, the scale 370 can show a non - linear relationship, such as a step function or different levels, where, for example, there may be an increasing revenue for an incremental improvement in the unit quantity (e.g., in some cases, the unit revenue for 0 to 100 additional units may be less than the unit revenue for 101 to 200 units).

[0127] By representing additional exported attributes or additional result types in the map visualization 300, the system can show the user the points in the parameter space that can provide the desired attributes or results. For example, in the case where the income scale 370 is related to the unit scale of the vertical axis 320, the map visualization 300 shows the combinations of parameter values that can achieve a specific income result. For example, the points along the horizontal line 371 show the conditions under which an income of $100,000 can be achieved. The portion of the line 371 within the target area 332 shows the combination of the expected improvement percentage value (along the horizontal axis 310) and the usage rate (at lines 226a to 226e) that can achieve this income result.

[0128] Figure 3F Another example of the map visualization 300F is shown. This example shows how the computer system 210 performs an analysis to determine the combinations of parameter values that can achieve certain performance goals while meeting the reliability threshold 223, and how to generate the map visualization 300F to present these results.

[0129] In some cases, the computer system 210 provides an interface for the user to input the target values of the performance metrics that the CCaaS 170 is to achieve. For example, the user has specified a target 380 of 400 additional units. The computer system 210 can then determine the combinations of parameter values (e.g., estimated improvement amount and usage rate) that can provide the indicated target amount of additional units while remaining within the target area 332f that meets the reliability threshold 223. For example, the computer system 210 provides Table 381, which shows the level of estimated improvement required to reach the target at each different usage rate (e.g., for a 30% usage rate, the second pairing strategy 264 needs to provide 3.56% more improvement than the first pairing strategy 262; for a 40% usage rate, 2.67% of the usage rate is required). The operating conditions represented in the table can be reflected in the table as the intersection points of the usage rate lines and the target line 382 representing the set target.

[0130] Through target analysis, the computer system 210 can solve the conditions that satisfy various user input constraints and the reliability threshold 223. The computer system 210 can focus the map visualization 300F on the identified area in the parameter space where the target performance results can be achieved. When the analysis indicates the estimated improvement level of the second pairing strategy 264, this can provide a reference for whether achieving the target is feasible.

[0131] Figure 3GShows another example of the map visualization 300G, where the improvement in performance is a reduction in the number of undesired outcomes. The horizontal axis 390 indicates the improvement of the second pairing strategy 264. The vertical axis 391 indicates the number of unnecessary on-site service visits, e.g., the change in the number of unnecessary service visits compared to the expected number when using only the first pairing strategy 262. Generally, if no on-site visit is required to solve a problem, then dispatching a technician to that location is undesirable and costly. Therefore, reducing the number of unnecessary visits is a valuable improvement in the performance of the call center system. This is reflected in the map visualization 300G, which shows a significant reduction in the number of unnecessary service visits, representing an improvement in performance.

[0132] Similar to the other visualizations described above, the map visualization 300G has boundaries 330g that specify the limits to the target area 332g, which represents the condition that the loop between the two pairing strategies 262, 264 is predicted to meet the reliability threshold 223. In the example, the user specified an indication of the combination of parameters to be tested, an estimated improvement of 2.0%, and a usage rate of 50%. The computer system 210 indicates this combination of parameters with a point 395 that falls within the area 334g that does not meet the required performance attribution reliability level and thus outside the target area 332g that would provide the required performance attribution reliability level. The computer system 210 shows the predicted outcomes that would be achieved with this combination of parameters specified by the user, including a reduction of 229 in unnecessary service visits and a p-value of 0.125. Other points 396, 397 have been selected by the user, and thus, information about the predicted effects of using these combinations of parameter values is shown.

[0133] Figure 4A and 4B Shows an example of the map visualization 400, which is interactive to allow the user to change the characteristics of the analysis being performed. The map visualization 400 can be provided via a web page, a web application, or other interactive user interfaces. The map visualization 400 has one or more associated interactive controls for adjusting one of the parameters used to generate the map visualization. For example, the control 410 provides the user with the ability to adjust the expected performance level of the first pairing strategy 262. Then, in response to user input that changes the performance level setting, the map visualization 400 is automatically updated. This allows the user to simulate or explore the effects of different values that the computer system 210 uses to perform its analysis. The user receives updated visualization data that shows a set of changed parameter value combinations in the parameter space that would meet the reliability threshold 223, e.g., by changing the size, shape, and / or location of the target area for acceptable reliability.

[0134] In Figure 4AIn [the example], control 410 is set such that the estimated performance of the first pairing strategy 262 is a conversion rate of 1.0% (e.g., “Off CR”, representing the performance when using the first pairing strategy 262 without using the second pairing strategy 264). This setting results in a bound 430a along a path where the reliability equals a threshold value (e.g., a curve with a p-value equal to 0.1). Bound 430a defines a target region 432a in which the combination of the expected improvement value and the value of additional sales units for the second pairing strategy 264 meets the reliability threshold 423. The various usage rate lines 426a to 426e show the conditions that can be achieved for different usage rates of the second pairing strategy 264.

[0135] From Figure 4A the perspective of the map visualization 400 shown, the user interacts with control 410 to change the conversion rate from 1.0% to 1.2%, which triggers an update to Figure 4B the map visualization 400 shown. The change in the conversion rate changes the target region 432a, the bound 430a, and the usage rate lines 426a to 426e. As a result, Figure 4B a new target region 432b, a bound 430b, and usage rate lines 427a to 427e are shown. For example, the usage rate line 427a and the target region 432b show that, compared to using a conversion rate of 1.0% with an estimated improvement of approximately 4.5%, using a conversion rate of 1.2% can provide the required reliability level at an estimated improvement of 4% for a 50% usage rate.

[0136] In response to user input to control 410 and other user interface controls, the client device 202 can provide the user input to the computer system 210 via the network 160. The computer system 210 can perform updated analysis and generate updated map data based on the newly specified parameter values by the user. The computer system 210 can then return the updated map data to the client device 202 via the network 160, where the updated map data can be rendered to display the updated map visualization 400.

[0137] In some embodiments, the initial map data provided by the computer system 210 can include associated code, scripts, or functions, rather than relying on client-server interaction, which enable the client device 202 to calculate the effects of the changed parameters specified through the interactive controls. For example, a web page, a web application, or other interactive documents can include script content or interpretable or executable code that runs in a web browser and recalculates the content of the visualization 400 when the user changes the parameter values.

[0138] In some embodiments, user input to the control 410 and / or other user interface controls can be provided directly to the computer system 210.

[0139] Figure 5 FIG. 4 is a flowchart showing an example of a process 500 for evaluating and improving contact center performance. The process 500 can be executed by one or more computers, such as the computer system 210, the client device 202, or another suitable computing device.

[0140] The process 500 includes obtaining historical contact-agent interaction data (502) of the contact center system. The historical contact-agent interaction data can indicate the performance of the contact center system using a first pairing strategy.

[0141] The process 500 includes identifying a threshold reliability level (504) for a pairing strategy for evaluating the contact center system. For example, the threshold reliability level can be a threshold value of a reliability metric such as a level of statistical significance (e.g., p-value). The reliability level can be the reliability for attributing performance improvement among multiple pairing strategies, based on a dataset describing the results of alternating or cycling through multiple pairing strategies over a period of time.

[0142] The process 500 includes determining a region (506) of a parameter space representing different combinations of parameter values of the contact center system. For example, based on the historical contact-agent interaction data, the computer system 210 can determine a region spanning the range of values for each of a plurality of parameters. The determined region indicates combinations of parameter values that allow for reliable performance attribution of performance improvement obtained by using a combination of a second pairing strategy and the first pairing strategy. For example, the determined region can be a target region in which performance improvement is identifiable because it is obtained by using the second pairing strategy with at least the threshold reliability level. In other words, the determined region can define conditions under which performance improvement using the second pairing strategy with at least a minimum level of confidence or statistical significance can be attributed. In some embodiments, the parameter space is a two-dimensional space representing different scenarios that can occur in the contact center system based on the expected attributes of contacts (e.g., contact volume) at the contact center system and historical characteristics of contacts at the contact center system.

[0143] To determine the target region of the parameter space, computer system 210 can define the boundaries of the region in the parameter space. The boundaries can be defined by combinations of parameter values that provide reliability at a threshold level. The boundaries define the region and separate it from other regions of the parameter space that represent combinations of parameter values that do not provide the threshold reliability level for identifying performance improvements. The boundaries can be curves or edges in the parameter space that define the target region where the operation of the contact center system will provide statistically significant performance attribution.

[0144] The parameter space can include a range of parameter values for a first parameter that represents the estimated improvement level of a second pairing strategy compared to a first pairing strategy. For example, the first parameter can be the estimated improvement level provided by the second pairing strategy relative to the first pairing strategy (e.g., the estimated improvement percentage as Figure 3A shown). The parameter space can also include a range of parameter values for a second parameter that quantifies the incremental change in the results at the contact center system obtained from the combined use of the second pairing strategy and the first pairing strategy. For example, the second parameter can be the incremental amount of a desired result achieved (e.g., the additional sales units as Figure 3A shown), the incremental amount of an undesired result avoided (e.g., the amount of unnecessary service calls avoided as Figure 3E shown), or a measure of some other metric (e.g., customer satisfaction rating, transaction value, call duration, wait time before a customer reaches an agent, etc.).

[0145] Process 500 includes selecting a usage rate for the second pairing strategy (508). The usage rate is selected based on the determined region in which performance improvement is identifiable because it is obtained by using the second pairing strategy with at least a threshold reliability level.

[0146] Process 500 includes pairing contacts and agents in the contact center system using the first pairing strategy and the second pairing strategy based on the selected usage rate (510). For example, the usage rate of the second pairing strategy in the contact center system is based on the selected usage rate. The contact center system is operated to cycle between using the first pairing strategy and the second pairing strategy, where the proportion of time or contacts processed by the second pairing strategy is set based on the selected usage rate. Thus, the contact center system can operate under conditions that are most likely to (i) have performance improvement compared to using the first pairing strategy alone, and (ii) provide result records that enable these improvements to be attributed to the use of the second pairing strategy.

[0147] A variety of embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. For example, using the various forms of processes described above, the steps can be reordered, added, or deleted.

[0148] Embodiments of the invention and all functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the invention can be implemented as one or more computer program products, e.g., one or more computer program instruction modules encoded on a computer-readable medium for execution by, or to control the operation of, a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter affecting a machine-readable propagated signal, or a combination of one or more of them. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus can also include code that creates an execution environment for the computer program being discussed, e.g., code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is used to encode information for transmission to a suitable receiver apparatus.

[0149] A computer program (also referred to as a program, software, a software application, a script, or code) can be written in any form of programming language, including a compiled or interpreted language, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program does not necessarily correspond to a file in a file system. The program can be stored in a part of a file that contains other programs or data (e.g., one or more scripts in a markup language document), in a single file dedicated to the program being discussed, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0150] The processes and logical flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0151] For example, processors suitable for executing computer programs include both general and special purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer will also include or be operatively coupled to receive data from, or transfer data to, one or more mass storage devices for storing data (e.g., magnetic disks, magneto-optical disks, or optical disks), or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device, such as a tablet computer, a mobile phone, a personal digital assistant (PDA), a mobile audio player, a global positioning system (GPS) receiver, etc. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0152] To provide interaction with a user, embodiments of the present invention may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and a pointing device, such as a mouse or a trackball, by which the user may provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, speech, or tactile input.

[0153] Embodiments of the present invention may be implemented in a computing system that includes backend components, such as a data server, or includes middleware components, such as an application server, or includes frontend components, such as a client computer having a graphical user interface or a web browser through which a user may interact with embodiments of the present invention, or any combination of one or more such backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0154] A computing system may include a client and a server. The client and the server are typically remote from each other and typically interact via a communication network. The relationship between the client and the server is created by computer programs running on respective computers and has a client-server relationship with each other.

[0155] Although this specification contains many details, these details should not be construed as limiting the scope of the invention or what may be claimed, but rather as descriptions of features of particular embodiments of the invention. Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although the above features may be described as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0156] Similarly, although operations are described in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed to achieve a desirable result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0157] Particular embodiments of the invention have been described. Other embodiments are within the scope of the following claims. For example, the steps recited in the claims may be performed in a different order and still achieve a desirable result.

Claims

1. A method performed by one or more computers, the method comprising: obtaining, by the one or more computers, historical contact-agent interaction data for a contact center system, wherein the historical contact-agent interaction data indicates the performance of the contact center system using a first pairing strategy; identifying, by the one or more computers, a threshold reliability level for evaluating a pairing strategy for the contact center system; determining, by the one or more computers based on the historical contact-agent interaction data, a region of a parameter space representing different combinations of parameter values for the contact center system, wherein the region spans a range of values for each of a plurality of parameters, and the region indicates combinations of parameter values for which a performance improvement can be identified using at least the threshold reliability level obtained by using a second pairing strategy in combination with the first pairing strategy; selecting, by the one or more computers, a usage rate of the second pairing strategy, wherein the usage rate is selected based on the determined region in which a performance improvement can be identified using at least the threshold reliability level obtained by using the second pairing strategy; and pairing, by the one or more computers, contacts and agents in the contact center system using the first pairing strategy and the second pairing strategy, wherein the usage rate of the second pairing strategy in the contact center system is based on the selected usage rate.

2. The method according to any one of the preceding claims, wherein determining the region of the parameter space includes: defining a boundary of the region in the parameter space, wherein the boundary is defined by a combination of parameter values providing the threshold reliability level for identifying a performance improvement, and the boundary separates the region from a region of the parameter space representing combinations of parameter values that do not provide the threshold reliability level for identifying a performance improvement.

3. The method according to any one of the preceding claims, wherein, Determining the region includes: defining a curve in the parameter space that bounds the region at parameter combinations providing the threshold reliability level for identifying a performance improvement.

4. The method according to any one of the preceding claims, wherein, The parameter space includes a range of parameter values for each of the following: (i) a first parameter that represents an estimated improvement level of the second pairing strategy compared to the first pairing strategy, and (ii) a second parameter that quantifies an incremental change in the result at the contact center system from using the second pairing strategy in combination with the first pairing strategy.

5. The method according to any one of the preceding claims, wherein, Determining the region of the parameter space includes determining the region based on: (i) the contact capacity at the contact center system determined from the historical contact-agent interaction data, and (ii) the performance metric of the contact center system achieved when using the first pairing strategy.

6. The method according to any one of the preceding claims, further comprising: Identify portions within the identified region that respectively correspond to different utilization rates for using the second pairing strategy.

7. The method according to any one of the preceding claims, further comprising: Determine an estimated level of performance improvement provided by the second pairing strategy compared to the first pairing strategy; Wherein, selecting the utilization rate for the second pairing strategy includes: selecting a utilization rate from a plurality of different utilization rates, the utilization rate using the estimated level of performance improvement to result in a combination of parameter values in the identified region.

8. The method according to any one of the preceding claims, wherein, The performance of the contact center system includes the amount or ratio of a predetermined result occurring for a contact at the contact center system, and, Wherein, the performance improvement includes an increase in the amount or ratio of the predetermined result occurring at the contact center system.

9. The method according to any one of the preceding claims, further comprising: Identify a series of utilization rates for application to different estimated improvement levels provided by the second pairing strategy compared to the first pairing strategy, Wherein, the series of utilization rates and estimated improvement levels provide progressively higher performance of the contact center system while remaining within the identified region, in the identified region, at least using the threshold reliability level to be able to identify performance improvement from the second pairing strategy.

10. The method according to any one of the preceding claims, further comprising: Provide user interface data for visualization, the visualization distinguishing the identified region from a region in the parameter space that does not provide the threshold reliability level for identifying performance improvement.

11. The method according to claim 10, wherein, The visualization indicates a combination of parameter values in the region, the parameter values corresponding to different utilization rates of the second pairing strategy when used in combination with the first pairing strategy.

12. The method according to claim 10 or claim 11, wherein, The visualization indicates a measure of the performance of the contact center system for each of a plurality of different performance aspects, wherein, The measures of the performance are interrelated such that the visualization indicates the expected measures of performance to be achieved in the target region for each of the different performance aspects.

13. The method according to any one of the preceding claims, wherein, Selecting the utilization rate for the second pairing strategy includes selecting a utilization rate that provides at least a minimum margin for the boundary of the identified region provided for the threshold reliability level with respect to the estimated level of performance improvement provided by the second pairing strategy relative to the first pairing strategy.

14. The method according to any one of the preceding claims, wherein, The second pairing strategy involves using a machine learning model to perform the pairing of contacts and agents.

15. A system, comprising: One or more computers; And One or more computer-readable media storing instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform the operations of the method according to any one of claims 1 to 14.

16. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, are operable to cause the one or more computers to perform the operations of the method according to any one of claims 1 to 14.

Images