Systems and methods for validating pairing models by verifying outcomes of multitouch data points

Abstract

Systems and methods for validating contact-agent pairing models prior to deployment in contact centers by verifying outcomes of multitouch data points. For example, the system may obtain a plurality of models and a plurality of contact-agent interactions. The system may perform a respective first validation for each model of the plurality of models based on a first metric, wherein the first metric is based on a respective first measurement of each contact-agent interaction of the plurality of contact-agent interactions, and wherein the respective first measurement is based on measuring a respective outcome associated with each contact-agent interaction. The system may perform a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on measuring a contact center state after each contact-agent interaction. The system may select a model of the plurality of models based on the respective first validation for each model and the respective second validation for each model.

Description

SYSTEMS AND METHODS FOR VALIDATING PAIRING MODELS BY VERIFYING OUTCOMES OF MULTITOUCH DATA POINTSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 508,501, filed on June 15, 2023. The content of the foregoing application is incorporated herein in its entirety by reference.BACKGROUND

[0002] Systems for connecting parties in a communication, irrespective of whether the format of the communication comprises in-person discussions, electronic transmissions, and / or other forms through which information is exchanged, are increasingly focusing on streamlining the connection process and / or increasing the efficiency of the resulting communications. To achieve this outcome, systems increasingly rely on one or more models for selecting the parties for the communication. These models may invoke one or more algorithms and / or pairing strategies which comprise a deliberate approach or plan for combining or matching items, parties, and / or entities for a particular purpose or desired outcome. In some cases, these algorithms may also involve the use of artificial intelligence, including, but not limited to, machine learning, deep learning, etc. (referred to collectively herein as artificial intelligence models, machine learning models, or simply models).

[0003] Regardless of the algorithms and / or pairing strategies used, effectively pairing parties to communications involves overcoming several technical challenges inherent to the pairing process. For example, pairing parties with complementary skills and knowledge is crucial for achieving intended outcomes; however, finding the right pairing can be challenging, especially when there is a wide range of expertise and / or when specific domain knowledge is required. Moreover, even if skill sets are known, coordinating the schedules of those parties to facilitate pairing may be difficult, particularly when the parties have different commitments and / or work on different projects. Failure of the system to account for misaligned availability may lead to delays and interruptions, reducing the effectiveness of the pairing strategy. These technical problems may be further exacerbated when attempting to use a model-based solution in dynamic environments featuring parties with a diverse spectrum of skills, requests featuring unknown skill sets, and parties with varying schedules, such as a contact center.SUMMARY

[0004] Systems and methods are described herein for novel uses and / or improvements to modeling applications. As one example, systems and methods are described herein for improvements to modeling applications used in dynamic environments such as contact centers. The systems and methods provide these improvements through the use of multipoint validation.

[0005] For example, modeling systems rely on aggregating copious amounts of data for training and / or evaluating a model. This data typically comprises a current modeling (or pairing) strategy and the results of interactions (e.g., contact-agent interactions) based on the current modeling strategy. Based on the results, the system may determine an effectiveness (e.g., measure as a percentage of interactions comprising a particular result) of the pairing strategy. However, in dynamic environments such as contact centers, the effectiveness of a given model may be subject to inaccuracies due to the corruption of its underlying data. This corruption may be a result of inaccuracies in the creation or collection of this data as participants directly affect the generation of the result and may have motives to introduce bias into the results. In other examples, the corruption may be a result of the time frame in which the result is measured; a result measured too close in time to the initial interaction may appear to be successful, although the success may corrode overtime, and would be more accurately measured further in time from the initial interaction.

[0006] To overcome these technical deficiencies in adapting artificial intelligence models for this practical benefit, systems and methods introduce multipoint validation to validate model results. For example, in order to eliminate the bias in the results, the system performs two validations on distinct, but corresponding data. For example, the system performs a respective first validation for each model of a plurality of models based on a first metric, wherein the first metric is based on a respective first measurement of each contact-agent interaction of the plurality of contact-agent interactions, and wherein the respective first measurement is based on measuring a respective outcome associated with each contact-agent interaction (e.g., an outcome directly following or during a first contact-agent interaction). The system then performs a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on measuring a contact center state after each contact-agent interaction or further in time from the first contact-agent interaction (e.g., an outcome subsequently following a second contact-agent interaction corresponding to the first). The system then selects a model of the plurality of models based on the respective first validation for each model and the respective second validation for each model.

[0007] In some aspects, systems and methods for validating contact-agent pairing models prior to deployment in contact centers by verifying outcomes of multitouch data points are described. For example, the system may obtain a plurality of models and a plurality of contact-agent interactions. The system may perform a respective first validation for each model of the plurality of models based on a first metric, wherein the first metric is based on a respective first measurement of each contact-agent interaction of the plurality of contact-agent interactions, and wherein the respective first measurement is based on measuring a respective outcome associated with each contact-agent interaction (e.g., the respective outcome is during or close in time to the contact-agent interaction). The system may perform a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on measuring a contact center state after each contact-agent interaction (e.g., measuring the contact center state at a point in time after the first measurement). The system may select a model of the plurality of models based on the respective first validation for each model and the respective second validation for each model.

[0008] Various other aspects, features, and advantages of the invention will be apparent through the detailed description of the invention and the drawings attached hereto. It is also to be understood that both the foregoing general description and the following detailed description are examples and are not restrictive of the scope of the invention. As used in the specification and in the claims, the singular forms of “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. In addition, as used in the specification and the claims, the term “or” means “and / or” unless the context clearly dictates otherwise. Additionally, as used in the specification, “a portion” refers to a part of, or the entirety of (i.e., the entire portion), a given item (e.g., data) unless the context clearly dictates otherwise.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1A shows an example communication system, in accordance with one or more embodiments.

[0010] FIG. IB shows an example communication system, in accordance with one or more embodiments.

[0011] FIG. 1C shows an example communication system, in accordance with one or more embodiments.

[0012] FIG. ID shows an example communication system, in accordance with one or more embodiments.

[0013] FIG. 2 shows a system featuring a model configured to facilitate pairing strategies, in accordance with one or more embodiments.

[0014] FIG. 3 shows graphical representations of artificial neural network models for facilitating pairing strategies, in accordance with one or more embodiments.

[0015] FIG. 4 shows a flowchart for validating contact-agent pairing models prior to deployment in contact centers by verifying outcomes of multitouch data points, in accordance with one or more embodiments.

[0016] FIG. 5 shows a flowchart for validating contact-agent pairing models, in accordance with one or more embodiments.DETAILED DESCRIPTION OF THE DRAWINGS

[0017] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the invention. It will be appreciated, however, by those having skill in the art that the embodiments of the invention may be practiced without these specific details or with an equivalent arrangement. In other cases, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the embodiments of the invention.

[0018] FIG. 1A illustrates an example communication system 100A. In this example, communication system 100A may comprise a contact center to facilitate communications between contact 102 and agents (e.g., agent 130A and agent BOB) via one or more user interfaces 104. For example, a contact center may comprise a centralized facility or department within an organization that handles incoming and outgoing communications with users, customers, and / or other parties (e.g., referred to herein as “contacts”). As referred to herein, a “user interface” may comprise a human-computer interaction and communication in a device, and may include display screens, keyboards, a mouse, and the appearance of a desktop. For example, a user interface may comprise a way for an agent to interact with an application, website, or contact, and / or to communicate content. As referred to herein, “content” should be understood to mean anelectronically consumable and / or transmittable data such as Internet content (e.g., streaming content, downloadable content, Webcasts, etc.), video clips, audio, content information, pictures, rotating images, documents, playlists, websites, articles, books, electronic books, blogs, advertisements, chat sessions, social media content, applications, games, and / or any other media or multimedia and / or combination of the same.

[0019] In some embodiments, communication system 100A may provide a correction to a performance evaluation of multiple pairing strategies in a contact center by more accurately accounting for the “multitouch” problem. As described herein, a contact center may comprise a primary point of contact for contacts (e g., contact 102) to interact with one or more agents (e.g., agent 130A and agent 130B) to ask questions, seek support, or make inquiries. As referred to herein, an agent may comprise a representative of a contact center, which may include a human and / or non-human entity that may interact with a contact during a contact-agent interaction.

[0020] As described herein, contact centers may use various communication channels such as phone calls, emails, webchat, social media, video calls, text chat, text messages, and / or other communication means to facilitate interactions between contacts and agents (e.g., contact-agent interactions). In some embodiments, contact centers often utilize advanced technologies like interactive voice response (IVR) systems, customer relationship management (CRM) software, call routing and queuing systems, knowledge bases, and analytics tools to streamline operations, improve efficiency, and enhance the overall customer experience.

[0021] Models for a contact center system may be validated based on actual outcomes of each contact-agent interaction and based on a measurement of contact center state after the contactagent interaction. As described herein, a contact-agent interaction may comprise a distinct session or a series of communications between a contact (e.g., contact 102) and an agent (e.g., agent 130A). A given contact-agent interaction may be distinguished from another contact-agent interaction based on one or more characteristics. As described herein, “interaction data” may comprise a feature or quality (whether quantitative or qualitative) used to identify a contact-agent interaction and / or distinguish the contact-agent interaction from another contact-agent interaction.

[0022] In some embodiments, interaction data may comprise: a time or date (or other temporal characteristic) related to the contact-agent interaction; content at issue and / or discussed during the contact-agent interaction; an identity (or identifier) for an agent and / or contact (e.g., such as a Billing Telephone Number (BTN)), including profile data related to the agent and / or contactinvolved in the contact-agent interaction; a model and / or pairing strategy, including one or more metrics thereof, used to connect parties in the contact-agent interaction; a result or outcome of a contact-agent interaction related to the contact-agent interaction; a contact center state before, during, and / or after the contact-agent interaction; a contact account state, including types of data and / or values thereof, before, during, and / or after the contact-agent interaction; and / or a relationship of the contact-agent interaction to another contact-agent interaction, including whether or not the contact-agent interaction, and / or any interaction data thereof, corresponds to (e.g., is equal to, is within a proximity to, is within a range of, matches to a certain degree and / or confidence, and / or involves similar interaction data) another contact-agent interaction and / or any interaction data thereof.

[0023] In some embodiments, interaction data may also include transcript data that may include information that was sent between a contact / agent (e.g., emails / texts). For example, interaction data may comprise a written or printed version of material originally presented in another medium. In some embodiments, interaction data may include time data that may indicate an interaction start time, interaction end time, estimated wait time, handle time, etc.

[0024] For example, the system may record interaction data for one or more contact-agent interactions. In some embodiments, the system may use the interaction data (and / or its relationship to one or more contact-agent interactions) to generate historical contact-agent interaction data (e.g., which may be used for modeling purposes).

[0025] In some embodiments, the interaction data may include profile data. The system may monitor for and / or record contact-agent interaction characteristics to generate profile data. As referred to herein, “a profile” and / or “profile data” may comprise actively and / or passively collected data about a contact-agent interaction, contact, agent, model, contact center, and / or collection (or subset thereof). For example, the profile data may comprise interaction data generated by, at, and / or in response to the contact-agent interaction.

[0026] In some embodiments, interaction data may comprise time-series data. As described herein, “time-series data” may include a sequence of data points that occur in successive order over some period of time. In some embodiments, time-series data may be contrasted with cross-sectional data, which captures a point in time. A time series can be taken on any variable that changes over time (e.g., a value in a contact account). The system may use a time series to track the variable (e.g., account status) of an account over time (e.g., across multiple contact-agent interactions). This canbe tracked over the short term, such as whether an account was “saved” during a contact-agent interaction, or the long term, such as whether the account continued to be saved over a given period of time and / or after successive contact-agent interactions. In some embodiments, the system may generate a time-series analysis and / or may use the time-series analysis for one or more metrics (e g., as described below).

[0027] For example, in a diagonal method of behavioral pairing, agents are ranked based on measurements of their performance, and pairings are made based on said measurements of agent performance. For example, agent performance can be based on any metric, such as revenue generation, sales, offer resource allocation if the agent is able to successfully sell harder-to-sell offers, contact retention, influenceability, training, etc. Ranking an agent incorrectly can be damaging to the model / pairing strategy because the wrong agent will receive the wrong calls, and performance of a model for a pairing strategy will decrease relative to the incumbent pairing strategy. Ranking the agents correctly is a complex problem that involves many statistical techniques, such as Bayesian mean regression based on the amount of data available, and properly tracking the outcome associated with each contact-agent interaction.

[0028] Additionally, there are special challenges when a contact (e g., contact 102) repeatedly connects to a contact center. For example, the contact center may employ techniques to ensure that the contact is paired with the same pairing strategy and rarely is paired with the same agent (e.g., agent 13 OB in a second contact-agent interaction if a first contact-agent interaction involved agent 130A); in other examples, the contact may not be paired with the same pairing strategy during a first contact-agent interaction and a second contact-agent interaction. When looking at a retention metric for agent performance, for example, the contact may be “saved” on an initial interaction with agent 130A at the contact center, but the contact may later cancel on a subsequent interaction with agent 13 OB at the contact center. A proper evaluation of the pairing strategies needs to reflect that the first pairing happened with a first pairing strategy associated with agent I 30A and the second pairing happened with a second pairing strategy associated with agent 13 OB, with gain apportioned among the pairing strategies; similarly, when data is used to train models for the pairing strategies, there need to be techniques in place to ensure that a “save” and a “loss” are properly accounted for in the training data. Accordingly, the system may apportion gain to each pairing strategy in order to fairly apportion the associated revenue and to properly validate eachpairing strategy as well as determine whether a pairing strategy is performing poorly in order to train models correctly.

[0029] As shown in FIG. 1 A, communication system 100A includes two contact-agent interactions corresponding to contact 102. For example, contact 102 may be involved with contact-agent interaction 106 (e.g., at a first time point) and contact-agent interaction 108 (e.g., at a second time point). Contact-agent interaction 106 and contact-agent interaction 108 may comprise corresponding interaction data of a first type (e.g., a contact involved in the contact-agent interaction, a contact account involved, etc.) and may comprise interaction data of other types that does not correspond (e.g., time point at which the contact-agent interaction occurs, an agent involved in the contact-agent interaction, a pairing strategy used to connect the contact and the agent in the contact-agent interaction, etc.). The system may then validate models based on the two contact-agent interactions. For example, contact-agent interaction 106 may correspond to agent 130A and contact-agent interaction 108 may correspond to agent 130B.

[0030] If the model does not reflect multipoint validation, there may be agents who are ranked highly who appear to be better-performing agents because this set of agents does not have cancellations associated with them. For example, if agent 130A knows that contact 102 is going to cancel the contact account, agent I30A may tell contact 102 to call back later, ending contactagent interaction 106 with what appears to be a positive outcome (e.g., the account is saved) or with the absence of a negative outcome (e.g., no account loss / cancellation is associated with the contact-agent interaction). Then, when contact 102 calls back later to cancel (e.g., during contactagent interaction 108), contact 102 may be routed to a different agent (e.g., agent 130B), and the cancellation is logged against agent BOB; in such a case, it would be more accurate for the cancellation to be logged against agent BOA. In another example, when contact 102 calls the contact center in order to cancel an account during contact-agent interaction 106, agent BOA may attempt to save the account by providing some offer to contact 102. The contact 102 may initially agree to maintain the account during contact-agent interaction 106, but may later return to the contact center to cancel the account during subsequent contact-agent interaction 108, unsatisfied even with the offer from agent BOA. In this case, a cancellation may be logged against agent BOB during contact-agent interaction 108, but the cancellation may be more properly apportioned to contact-agent interaction 106 because agent BOA failed to actually maintain the account, and instead just temporarily deferred the cancellation.

[0031] This data presents a technical problem for modeling the results of the pairing strategies, which pairing strategies rank higher “performing” agents better than regular- or lower-performing agents. However, if the rankings are incorrect (as the rankings would be incorrect in a case where an agent logs no cancellations because agent 130A is ending the calls abruptly when agent 130A knows contact 102 will cancel or in cases where the cancellation was merely deferred), a model for a pairing strategy (e.g., a diagonal method pairing strategy) will perform worse or wrongly, because the corresponding calls will keep being routed to the “higher-performing-but-actually- lower-performing” agent BOB. That is, the behavior of agent BOB is being improperly obscured, and a model cannot correctly assign contacts to agent BOA because agent BOA’s true behavior is yet unknown. Moreover, if a pairing strategy wrongly assigns cancellation calls to agent BOA, who always “saves” a contact (but is actually just always deferring a cancellation) without attempting a legitimate save for the contact, then a model’s “callback rate” may increase relevant to other pairing strategies deployed at the contact center environment.

[0032] To overcome these technical challenges, the system ensures correct tagging of data and proper validations of models before the model is deployed in a contact center environment. For example, in order to eliminate the bias in the results, the system performs two validations on distinct, but corresponding data. For example, the system performs a respective first validation for each model of a plurality of models (e.g., models corresponding to pairing strategies) based on a first metric, wherein the first metric is based on a respective first measurement of each contactagent interaction of the plurality of contact-agent interactions, and wherein the respective first measurement is based on measuring a respective outcome associated with each contact-agent interaction (e.g., an outcome directly following a first contact-agent interaction).

[0033] As referred to herein, an outcome may comprise a qualitative or quantitative metric for assessing whether a contact account state, including, but not limited to, a percentage of, a ratio of, etc., changes from one value to another in response to a contact-agent interaction. For example, the system may receive outcomes indicating daily values for agents, contact centers, products, accounts, contacts, and / or groups thereof. In some embodiments, the outcome may comprise a value in a contact account that is changed (or not changed). For example, the outcome may indicate whether or not an account was gained or saved; the outcome may be binary, and / or may correspond to data (or the absence of data) associated with a contact-agent interaction

[0034] The system performs a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on measuring a contact center state after each contact-agent interaction (e.g., an outcome subsequently following a second contact-agent interaction corresponding to the first or a state of the contact center system at a point in time after the first measurement). The system then selects a model of the plurality of models based on the respective first validation for each model and the respective second validation for each model.

[0035] As referred to herein, a contact center state may be a qualitative or quantitative metric for assessing whether a contact account state, including, but not limited to, a percentage of, a ratio of, etc., changes from one value to another. The system may determine an initial value (or change in value) from an initial interaction and determine whether that initial value (or the change in value) is maintained during a subsequent interaction. For example, the system may determine whether a change to an account occurred during a first threshold proximity (e.g., during a first interaction). The system may also determine whether a change occurred during a second threshold proximity (e.g., during a second interaction). By determining whether the change occurred during either proximity, the system may determine whether a change (or lack thereof) was permanent or temporary. For example, by measuring the contact center state after each contact-agent interaction, the system may retrieve the respective contact account and determine whether the respective contact account state for the respective contact account changes from the first value to the second value within a second threshold time proximity of the respective first contact-agent interaction.

[0036] In some embodiments, the system may select a date range for the data to catch multitouch contacts and / or confirm whether the first respective measurement was stable (e.g., the value of the measurement did not change within the date range). For example, the date range may be 30 days (or 15, 60, 45, etc.). In some embodiments, the date range may be either floating or fixed. If floating, then each contact may have a look after within a predetermined period of time to see if a result was stable (e.g., does not change); for example, the predetermined period of time may be 5 days, 10 days, 15 days, 20 days, 30 days, 45 days, 60 days, 90 days, one month, two months, three months, etc. If fixed, then all contacts in a given time period (e.g., 5 days, 10 days, 15 days, 20 days, 30 days, 45 days, 60 days, 90 days, one month, two months, three months, etc.) are evaluated to determine whether the result changed within the fixed period time period.

[0037] The system also may use metrics determined for each validation. For example, the system performs two validations. The first validation comprises an outcome of the contact-agent interaction (e.g., whether the first touch (e.g., the first interaction of one or more agents with the contact regarding the same subject matter) was a save or a cancellation), and a second validation is performed for each model based on a second metric, where the second metric is based on a future state of the contact center system (e.g., whether there was a callback, whether the future callback was a cancellation). The system may then use the two validations to perform one validation, where the metric is a combination of an on-the-call measurement and an after-the-call measurement, or to otherwise compare each model based on the first validation and the second validation.

[0038] As shown in FIG. 1A, the communication system 100A may include a central switch 110. The central switch 110 may receive incoming contacts (e.g., callers) or support outbound connections to contacts via a telecommunications network (not shown). The central switch 110 may include contact routing hardware and software for helping to route contacts among one or more contact centers, or among one or more contact center switches 120A and 120B. For example, the central switch 110 is a load balancer. Contact center switches 120A and 120B may include a Private Branch Exchanges (PBXs) and / or Automatic Call Distributors (ACDs) or other queuing or switching components, including other Internet-based, cloud-based, or otherwise networked contact-agent hardware or software-based contact center solutions.

[0039] In should be noted that the central switch 110 or load balancer may not be necessary such as if there is only one contact center, or if there is only one switch or PBX / ACD routing component, in the communication system 100A (not shown). If more than one contact center is part of the communication system 100A, each contact center may include at least one contact center switch (e.g., contact center switches 120A and 120B). For the purposes of this example embodiment, contact center switches 120A and 120B are both routed to the same contact center system. The contact center switches 120 A and 120B may be communicatively coupled to the central switch 110. In embodiments, various topologies of routing and network components may be configured to implement the contact center system. 00 0 / Each contact center switch for each contact center may be communicatively coupled to a plurality (or “pool”) of agents (e.g., agent 130A and agent 130B). Each contact center switch may support a certain number of agents (or “seats”) to be logged in at one time. At any given time, a logged-in agent may be available and waiting to be connected to a contact, or the logged-in agentmay be unavailable for any of a number of reasons, such as being connected to another contact, performing certain post-call functions such as logging information about the call, or taking a break.

[0041] In some embodiments, a contact may be included in a contact group. For example, a contact group or contact type may be a partition or grouping of contacts based on historical contactagent interaction data. As one example, contacts who arrive at a contact center may be assigned to or associated with a specific contact group or type. In some embodiments, a contact group may be defined as a contact percentile range. For example, percentiles may comprise a division of a ranked data set into a set number of (e.g., 100) equal parts. In one example, each (ranked) data set has 99 percentiles that divide it into 100 equal parts. For example, the kth percentile is denoted by Pk, where k is an integer in the range 1 to 99. In some embodiments, the contact percentile range may include a group of percentiles and / or these groups may be ranked. For example, a percentile rank of a score refers to the percentage of metrics in its frequency distribution that are lower than it or equal to. For example, a metric that is greater than 75% of the metrics of results of contacts in contact-agent interactions may be at the 75th percentile. For example, having a sufficient number of contact groups ensures that there is enough diversity in the data to test behavioral pairing strategies in the contact center environment.

[0042] In the example of FIG. 1 A, the central switch 110 routes contacts via contact center switch 120 A and contact center switch 120B, respectively. The contact center switches 120 A and 120B are shown and may be connected with two agents each. Agents 130A and 130B may be logged in to contact center switch 120 A, and agents 130C and I 30D may be logged in to contact center switch 120B.

[0043] The communication system 100A may also be communicatively coupled to an integrated service from, for example, a third-party vendor. In the example of FIG. 1A, a pairing node 140 may be communicatively coupled to one or more switches in the switch system of the communication system 100A, such as central switch 110, contact center switch 120A, and / or contact center switch 120B. In some embodiments, switches of the communication system 100A may be communicatively coupled to multiple pairing nodes (e.g., each switch is connected to a different pairing node). In some embodiments, pairing node 140 may be embedded within a component of a contact center system (e.g., embedded in or otherwise integrated with a switch). The pairing node 140 may receive information from a switch (e.g., contact center switch 120A) about agents logged in to the switch (e.g., agents 130A and DOB) and about incoming contactsvia another switch (e.g., central switch 110) or, in some embodiments, from a network (e.g., the Internet or a telecommunications network) (not shown).

[0044] A contact center may include multiple pairing nodes. In some embodiments, one or more pairing nodes may be components of pairing node 140 or one or more switches such as central switch 110 or contact center switches 120A and 120B. In some embodiments, a pairing node may determine which pairing node may handle pairing for a particular contact. For example, a first pairing node may alternate between enabling pairing via a Behavioral Pairing (BP) strategy via the first pairing node and enabling pairing with a First-in-First-out (FIFO) strategy via a second pairing node. In other embodiments, one pairing node (e.g., the BP pairing node) may itself be configured to emulate other pairing strategies.

[0045] FIG. IB illustrates a second example communication system 100B. As shown in FIG. IB, the communication system 100B may include one or more agent endpoints 151A, 151B and one or more contact endpoints 152A, 152B. The agent endpoints 151A, 15 IB may include an agent terminal and / or an agent computing device (e.g., laptop, cellphone). The contact endpoints 152A, 152B may include a contact terminal and / or a contact computing device (e.g., laptop, cellphone). Agent endpoints 151A, 151B and / or contact endpoints 152A, 152B may connect to a Contact Center as a Service (CCaaS) 170 through either the Internet or a public switched telephone network (PSTN) 160, according to the capabilities of the endpoint device.

[0046] FIG. 1C illustrates an example communication system 100C with an example configuration of a CCaaS 170. For example, CCaaS 170 may include multiple data centers 180A, 180B. The data centers 180A, 180B may be separated physically, even in different countries and / or continents. The data centers 180A, 180B may communicate with each other. For example, one data center is a backup for the other data center, so that, in some embodiments, only one data center 180A or 180B receives agent endpoints 151 A, 15 IB and contact endpoints 152A, 152B at a time.data center 180A, 180B includes web demilitarized zone (DMZ) equipment 171 A and 171B, respectively, which is configured to receive the agent endpoints 151A, 151B and contact endpoints 152A, 152B, which are communicatively connecting to CCaaS via the Internet. Web DMZ equipment 171 A and 17 IB may operate outside a firewall to connect with the agent endpoints 151A, 151B and contact endpoints 152A, 152B while the rest of the components of data centers 180 A, 180B may be within said firewall (besides the telephony DMZ equipment 172A, 172B, which may also be outside said firewall). Similarly, each data center 180A, 180B includestelephony DMZ equipment 172A and 172B, respectively, which is configured to receive agent endpoints 151 A, 15 IB and contact endpoints 152A, 152B, which are communicatively connecting to CCaaS via the PSTN. Telephony DMZ equipment 172A and 172B may operate outside a firewall to connect with the agent endpoints 151 A, 15 IB and contact endpoints 152A, 152B while the rest of the components of data centers 180A, 180B (excluding web DMZ equipment 171 A, 17 IB) may be within said firewall.

[0048] Further, each data center 180A, 180B may include one or more nodes 173A, 173B and 173C, 173D, respectively. All nodes 173 A, 173B and 173C, 173D may communicate with web DMZ equipment 171A and 171B, respectively, and with telephony DMZ equipment 172A and 172B, respectively. In some embodiments, only one node in each data center 180A, 180B may be communicating with web DMZ equipment 171A, 171B and with telephony DMZ equipment 172A, 172B at a time.

[0049] Each node 173A, 173B, 173C, 173D may have one or more pairing modules 174A, 174B, 174C, 174D, respectively. Similar to pairing node 140 of communication system 100A of FIG. 1A, pairing modules 174A, 174B, 174C, 174D may pair contacts to agents. For example, the pairing module may alternate between enabling pairing via a BP module and enabling pairing with a FIFO module. In other embodiments, one pairing module (e.g., the BP module) may be configured to emulate other pairing strategies.

[0050] Turning now to FIG. ID and example communication system 100D, the disclosed CCaaS communication systems (e.g., FIGs. IB and / or 1C) may support multi-tenancy such that multiple contact centers (or contact center operations or businesses) may be operated on a shared environment. That is, each tenant may have a separate, non-overlapping pool of agents. CCaaS 170 is shown in FIG. ID as comprising two tenants 190A and 190B. Turning back to FIG. 1C, for example, multi-tenancy may be supported by node 173 A supporting tenant 190A while node 173B supports tenant 190B. In another embodiment, data center 180 A supports tenant 190 A while data center 180B supports tenant 190B. In another example, multi-tenancy may be supported through a shared machine or shared virtual machine such that node 173 A may support both tenants 190A and 190B, and similarly for nodes 173B, 173C, and 173D. In other embodiments, the system may be configured for a single tenant within a dedicated environment such as a private machine or private virtual machine.

[0051] FIG. 2 shows a system featuring a model configured to facilitate pairing strategies, in accordance with one or more embodiments. In some embodiments, system 200 may use one or more models, including artificial intelligence-based and non-artificial intelligence-based, to select a model for pairing one or more contacts and / or agents, pair one or more contacts and / or agents, and / or evaluate (or validate) another model (e.g., used to pair one or more contacts and / or agents). For example, as shown in FIG. 2, system 200 may generate a prediction and / or determination using model 202. The determination may be output shown as output on a user interface (e.g., interface 104 (FIG. 1A)) and / or may direct a pairing using one or more components (e.g., pairing node 140 (FIG. 1A)) or one or more switches (such as central switch 110 (FIG. 1A) or contact center switches 120A and 120B (FIG. 1A)). The system may include one or more pairing strategies, neural networks (e.g., as discussed in relation to FIG. 3), and / or other models.

[0052] As an example, with respect to FIG. 2, model 202 may take inputs 204 and provide outputs 206. The inputs may include multiple datasets such as a training dataset and a test dataset (which may be based on historical interaction data). The datasets may represent one or more contact-agent interactions and / or interaction data related thereto. In some embodiments, outputs 206 may be fed back to model 202 as input to train model 202 (e.g., alone or in conjunction with user indications of the accuracy of outputs 206, labels associated with the inputs, or other reference feedback information). Alternatively or additionally, model 202 may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of its prediction (e.g., outputs 206) and reference feedback information (e.g., user indication of accuracy, reference labels, additional interaction data, or other information). Alternatively or additionally, where model 202 is a neural network, connection weights may be adjusted to reconcile differences between the neural network’s prediction and the reference feedback. In a further use case, one or more neurons (or nodes) of the neural network may require that their respective errors be sent backward through the neural network to them to facilitate the update process (e.g., b ackpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the model 202 may be trained to generate better predictions and / or determinations.

[0053] In some embodiments, model 202 may comprise one or more versions. For example, different versions of model 202 (and / or other models referred to herein) may refer to variations or iterations of a particular model that have been developed over time. These versions may involvechanges to the model’s architecture, parameters, and / or training data (e.g., as a result of newly available data, improvements to the model’s performance, and / or updates to address specific requirements). Here are a few common types of model versions. In some embodiments, the system may use an initial baseline model, which may be used by the system as a starting point for further improvements and / or comparisons of the model and / or its results. The versions of model 202 may also have one or more iterative versions as a result of the model undergoing a series of incremental changes and / or iterations. In some embodiments, system 200 may store a plurality of models. This plurality of models may include multiple versions of the same model as well as different models. The models may differ based on algorithms used for the model, pairing strategies corresponding to the model, and / or datasets upon which the model was trained.

[0054] Model 202 may be trained to select a model for pairing one or more contacts and / or agents, pair one or more contacts and / or agents, and / or evaluate (or validate) another model (e.g., used to pair one or more contacts and / or agents). For example, model 202 may receive a first dataset (e.g., historical interaction data) and / or one or more classifications (e.g., available pairing strategies, available agents for assigning a contact-agent interaction, a metric for a model, etc.). Model 202 is then trained based on a first dataset to classify a second dataset (e.g., contacts awaiting assignment to an agent, etc.).

[0055] In some embodiments, model 202 may generate and / or record one or more metrics for a model. As referred to herein, a model metric may comprise data about a model and / or its results. In some embodiments, metrics may comprise evaluation metrics that are measures used to assess the performance and effectiveness of the model and / or a version thereof. For example, the metrics may provide quantitative information about how well the model is performing and / or may be used to compare different models and / or evaluate the same model under different conditions.

[0056] In some embodiments, the metrics may comprise one or more values based on statistical and / or mathematical operations. For example, the metric may reflect an accuracy of a model (e.g., a proportion of correct predictions made by the model over the total number of predictions), precision (e.g., a proportion of true positive predictions over the total number of positive predictions), recall (e.g., a proportion of true positive predictions over the total number of actual positive instances), Fl score (e.g., a harmonic mean of precision and recall), specificity (e.g., a proportion of true negative predictions over the total number of actual negative instances), mean squared error (e.g., an average squared difference between predicted and actual values), meanabsolute error (e.g., an average absolute difference between predicted and actual values), R- squared (e.g., a proportion of the variance in the dependent variable that is predictable from the independent variables), contact (or agent) percentile or ranking (e.g., a percentile or ranking at which a contact or agent is assigned), standard deviation (e.g., a measure of how dispersed the data is in relation to the mean), and / or mean absolute deviation (e.g., a measure of variability that indicates the average distance between each prediction and a mean of the total number of predictions). In some embodiments, the metrics may be determined using specific formulas and / or criteria. For example, a contact group may be assigned to a percentile range (e.g., 10-20%). A contact associated to that contact group may then be assigned within that range randomly.

[0057] Additionally or alternatively, the metrics for one or more models (or versions thereof), samples of the one or more models’ results, and / or results from running one or more models may be aggregated. The system may aggregate the models to determine means, medians, modes, and / or other averaging functions to determine an average metric for one or more models.

[0058] Additionally or alternatively, the metrics for a model may be based on, a derivative of, and / or related to a contact account (or value thereof) as well as one or more contact-agent interactions. For example, a metric may relate to an average amount, a number of, and / or a percentile of contact accounts that comprise a particular value for a particular category and / or a change in the value. For example, a metric may comprise an average amount, a number of, and / or a percentile of contact accounts that maintain, change, and / or gain a value before, during, and / or after a client interaction. For example, in the context of a call center, a metric for a model may be based on the number of contact-agent interactions that resulted in an account being maintained, an account making a purchase, or any indicator of a successful outcome. For example, in the context of a call center, a metric for a model may be based on the number of contact-agent interactions that resulted in an account being cancelled.

[0059] Additionally or alternatively, the metric for a model may be in relation to another model. For example, a metric for a model may relate to a lift or gain experienced as the result of a model. Lift may be a measure of an effectiveness of a predictive model calculated as the difference between the results of a first model versus a second model (e.g., the ratio of contact accounts that were lost using a first model for pairing strategies subtracted from the ratio of client accounts that were lost using a second model for pairing strategies). Gain may be the ratio between the lift and data measured ratio of a given pairing strategy (e.g., the ratio of contact accounts that were lostusing the first model). For example, the system may determine the lift or gain attributed to one model in relation to another model.

[0060] Additionally or alternatively, the system may determine weights and / or effects on an evaluation and / or a validation of a model (e.g., in relation to another model) based on additional factors. For example, a metric corresponding to gain or lift may be increased (or decreased) based on whether multiple contacts, agents, contact-agent interactions, etc. corresponded to a given model. For example, a metric may be based on a difference or lack of a difference between a respective outcome (e.g., a respective outcome of a contact-agent interaction under a first model) and a contact center state (e.g., a contact center state after contact-agent interaction under the same or different model). The system may determine groupings of outcomes and contact center states based on common contacts and / or contact-agent interactions within a given time period.

[0061] In one example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state (e.g., a change or lack thereof in an account) under a different model. For example, the metric may be based on whether an account was eventually canceled under a second model after initially being saved under a first model. Alternatively, the metric may be based on whether an account was eventually saved under a second model after initially being canceled under a first model.

[0062] In another example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state (e.g., a change or lack thereof in an account) under the same model. For example, the metric may be based on whether an account was eventually canceled under a first model after initially being saved under the first model. Alternatively, the metric may be based on whether an account was eventually saved under a first model after initially being canceled under the first model.

[0063] In some embodiments, the metric may be based on instances in which the same model provided particular results (e.g., two contact-agent interactions that resulted in a save, cancelation, etc.). The metric in such instances may be based on additional weighting and / or emphasis. In some embodiments, the metric may be based on instances in which the same model provided particular frequencies of subsequent contact-agent interactions (e.g., a model that generates additional contact-agent interactions). The metric may in such instances be based on additional weighting and / or emphasis.

[0064] For example, when determining a metric for a first model during a period of time, the first model may be associated with various sets of calls, e.g.,: (i) a first set of calls, wherein each call in the first set of calls is a first touch, and has a value associated with a particular outcome; (ii) a second set of calls, wherein each call is a second touch, wherein the first touch for each call in the second set of calls was associated with a second model, and wherein the second touch for each call in the second set of calls was associated with the first model, and wherein each call in the second set of calls is associated with a value associated with a particular outcome after the second touch; (iii) a third set of calls, wherein each call is a second touch, wherein the first touch for each call in the third set of calls was associated with the first model, and wherein the second touch for each call in the third set of calls was associated with the second model, and wherein each call in the third set of calls is associated with a value associated with a particular outcome after the second touch; (iv) a fourth set of calls, wherein each call is a second touch, wherein the first touch for each call in the fourth set of calls was associated with the first model, and wherein the second touch for each call in the fourth set of calls was associated with the first model, and wherein each call in the fourth set of calls is associated with a value associated with a particular outcome after the second touch. For example, the metric may be based on a ratio / percentage / count of the first set of calls, the second set of calls, the third set of calls, and the fourth set of calls. In some examples, the fourth set of calls may be associated with a weighting to increase its weight relative to the other sets of calls. For example, the outcome may be successful (e.g., the metric may be associated with a save rate or a conversion rate) or the outcome may be unsuccessful (e.g., the metric may be associated with a cancel rate). In another example, a gain or lift metric may be determined based on at least one of the first set of calls, the second set of calls, the third set of calls, and the fourth set of calls. In another example, such a gain or lift metric may further include a weighting on the fourth set of calls to increase its weight relative to the other sets of calls. For example, such a weight is 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75, 3, etc.

[0065] In another example, the metric may be based on a callback rate for each model (e.g., callback rate corresponds to a number / percentage / ratio of contacts who returned to the contact center within a threshold period of time after a first interaction of that contact with the contact center system).

[0066] In some embodiments, the system may generate a time-series analysis and / or may use the time-series data for one or more metrics. For example, a time-series analysis may be useful to seehow a given contact account or value therein changes (or does not change) over time and / or subsequent contact-agent interactions. The system may also use a time-series analysis to examine how the changes associated with the chosen data point compare to shifts in other variables (e.g., model, agent, contact center performance) over the same time period and / or subsequent contactagent interactions. For example, with regard to accounts saved or gained, the system may receive time-series data for the various sub-segments indicating daily values for agents, contact centers, products, accounts, contacts, and / or groups thereof.

[0067] The time-series analysis may determine various trends such as secular trends (which describe movements along the term), seasonal variations (which represent seasonal changes), cyclical fluctuations (which correspond to periodic but not seasonal variations), and irregular variations (which are other nonrandom sources of series variations). The system may maintain correlations for this data during modeling. In particular, the system may maintain correlations through non-normalization as normalizing data inherently changes the underlying data, which may render correlations, if any, undetectable and / or lead to the detection of false positive correlations. For example, modeling techniques (and the predictions generated by them), such as rarefying (e.g., resampling as if each sample has the same total counts), total sum scaling (e.g., dividing counts by the sequencing depth), and others, and the performance of some strongly parametric approaches, depends heavily on the normalization choices. Thus, normalization may lead to lower model performance and more model errors. The use of a non-parametric bias test alleviates the need for normalization, while still allowing the methods and systems to determine a respective proportion of error detections for each model of the plurality of time-series data component models.

[0068] FIG. 3 shows graphical representations of artificial neural network models for facilitating pairing strategies, in accordance with one or more embodiments. Model 300 illustrates an artificial neural network. Model 300 includes input layer 302. Input layer 302 may receive contacts and / or agents waiting to be paired, metrics used to select a model for pairing one or more contacts and / or agents, etc. Model 300 also includes one or more hidden layers (e.g., hidden layer 304 and hidden layer 306). Model 300 may be based on a large collection of neural units (or artificial neurons). Model 300 loosely mimics the manner in which a biological brain works (e.g., via large clusters of biological neurons connected by axons). Each neural unit of a model 300 may be connected with many other neural units of model 300. Such connections can be enforcing or inhibitory in their effect on the activation state of connected neural units. In some embodiments, each individualneural unit may have a summation function which combines the values of all of its inputs together. In some embodiments, each connection (or the neural unit itself) may have a threshold function that the signal must surpass before it propagates to other neural units. Model 300 may be selflearning and trained, rather than explicitly programmed, and can perform significantly better in certain areas of problem solving, as compared to traditional computer programs. During training, output layer 308 may correspond to a classification of model 300 (e.g., contacts awaiting assignment to an agent, etc.), and an input known to correspond to that classification may be input into input layer 302. In some embodiments, model 300 may include multiple layers (e.g., where a signal path traverses from front layers to back layers). In some embodiments, backpropagation techniques may be utilized by model 300 where forward stimulation is used to reset weights on the “front” neural units. In some embodiments, stimulation and inhibition for model 300 may be more free-flowing, with connections interacting in a more chaotic and complex fashion. Model 300 also includes output layer 308. During testing, output layer 308 may indicate whether or not a given input corresponds to a classification of model 300 (e.g., pair one or more contacts and / or agents, select a model for pairing one or more contacts and / or agents, and / or evaluate (or validate) another model (e.g., used to pair one or more contacts and / or agents)).

[0069] FIG. 3 also includes model 350, which is a convolutional neural network. The convolutional neural network is an artificial neural network that features one or more convolutional layers. Convolutional layers extract features from an input. Convolution preserves the relationship between the data input by learning features using partitions of the input data. As shown in model 350, input layer 352 may proceed to convolution blocks 354 and 356 before being output to convolutional output 360. In some embodiments, model 350 may itself serve as an input to model 300.

[0070] In some embodiments, model 350 may implement an inverted residual structure where the input and output of a residual block (e.g., block 354) are thin bottleneck layers. A residual layer may feed into the next layer and directly into layers that are one or more layers downstream. A bottleneck layer (e.g., block 358) is a layer that contains few neural units compared to the previous layers. Model 350 may use a bottleneck layer to obtain a representation of the input with reduced dimensionality. An example of this is the use of autoencoders with bottleneck layers for nonlinear dimensionality reduction. Additionally, model 350 may remove nonlinearities in a narrow layer (e.g., block 358) in order to maintain representational power. In some embodiments, the design ofmodel 350 may also be guided by the metric of computational complexity (e.g., the number of floating point operations). In some embodiments, model 350 may increase the feature map dimension at all units to involve as many locations as possible instead of sharply increasing the feature map dimensions at neural units that perform downsampling. In some embodiments, model 350 may decrease the depth and increase the width of residual layers in the downstream direction. Model 350 may then generate output 360.

[0071] FIG. 4 shows a flowchart for validating contact-agent pairing models prior to deployment in contact centers by verifying outcomes of multitouch data points, in accordance with one or more embodiments. For example, the system may use process 400 (e.g., as implemented on one or more system components described above) in order to validate contact-agent pairing models prior to deployment in contact centers by verifying outcomes of multitouch data points.

[0072] At step 402, process 400 (e.g., using one or more components described above) obtains a plurality of models. For example, the system may obtain a plurality of models and a plurality of contact-agent interactions. By doing so, the system may retrieve one or more models that correspond to a particular pairing strategy and / or a version of a model trained on specific data (e.g., data corresponding to a specific date range).

[0073] At step 404, process 400 (e.g., using one or more components described above) performs a respective first validation for each model of the plurality of models based on a first metric. For example, the system may perform a respective first validation for each model of the plurality of models based on a first metric, wherein the first metric is based on a respective first measurement of each contact-agent interaction of the plurality of contact-agent interactions, wherein the respective first measurement is based on measuring a respective outcome associated with each contact-agent interaction. For example, the system may use a metric that represents an on-the-call measurement. In one example, the metric is based on a percentage / ratio / count of the number of saves associated with the model. This metric may be normalized or otherwise based on the total number of interactions associated with the model. That is, the first metric may be based on an outcome (e.g., whether it was a save or a cancellation) of the contact-agent interaction (e.g., represented by a first touch point).

[0074] In some embodiments, the system may determine a multitouch data point by selecting a date range for the data to catch multitouch contacts. For example, the date range may comprise a time period that includes a first time point (e.g., corresponding to a first metric and / ormeasurement) and a second time point (e.g., corresponding to a second or subsequent metric and / or measurement). For example, the first time point may comprise an initial interaction between a contact and an agent. For example, the system may receive a first time period for determining a multitouch data point. The system may determine a first time point based on the first time period. The system may determine the respective outcome (e.g., which may be based on the respective first measurement) associated with each contact-agent interaction at the first time point.

[0075] In some embodiments, measuring the respective outcome associated with each contactagent interaction may comprise determining the respective outcome associated with each contactagent interaction at a respective first time point and / or measuring the contact center state after each contact-agent interaction at a second time point. For example, the second time point may comprise a subsequent interaction between a contact and the same agent or a different agent.

[0076] In some embodiments, the system may receive a date range based on a user selection of a time period (e.g., 5 days, 10 days, 15 days, 20 days, 30 days, 45 days, 60 days, one month, two months, three months). The system may automatically select the current date to correspond to the first time point and a subsequent date corresponding to the time period (e.g., a month later) as the second time point. For example, the system may receive the first time period for determining the multitouch data point, and / or the system may receive a first user input and determine the first time period based on the first user input.

[0077] In some embodiments, the system may automatically determine a date range based on a predetermined setting, schedule, and / or other known frequency. For example, the system may continuously validate models by retrieving data at a predetermined schedule or frequency. In one example, the system may receive the first time period for determining the multitouch data point, and / or the system may receive a frequency at which to validate the plurality of models and determine the first time period based on the frequency.

[0078] In some embodiments, the system may measure the respective outcome associated with each contact-agent interaction based on whether or not the respective outcome is changed. To do so, the system may record attributes about each contact-agent interaction (e.g., a time, an account, and / or any changes to the account state) as well as an identifier for the interaction. For example, measuring the respective outcome associated with each contact-agent interaction may comprise the system determining a respective contact account corresponding to each contact-agent interaction. The system may determine a respective first contact time point corresponding to eachcontact-agent interaction. The system may determine whether a respective contact account state for the respective contact account changes from a first value to a second value within a first threshold proximity of the respective first contact time point.

[0079] In some embodiments, the system may measure the respective outcome associated with each contact-agent interaction based on whether or not the respective outcome is changed. To do so, the system may record attributes about a pairing strategy used to pair the contact and the agent. For example, measuring the respective outcome associated with each contact-agent interaction may comprise the system determining a respective contact account corresponding to each contactagent interaction. The system may determine a respective first contact time point corresponding to each contact-agent interaction. The system may determine whether a respective contact account state for the respective contact account changes from a first value to a second value within a first threshold proximity of the respective first contact time point.

[0080] In some embodiments, the system may measure the respective outcome associated with each contact-agent interaction based on whether or not the respective outcome is changed. To do so, the system may record attributes about a pairing strategy used to pair the contact and the agent. For example, measuring the respective outcome associated with each contact-agent interaction may comprise the system determining a first model of the plurality of models under which a first contact-agent interaction of the plurality of contact-agent interactions was paired and assigning a first outcome associated with the first contact-agent interaction to the first model.

[0081] In some embodiments, the first metric may be based on a comparison of outcomes from corresponding contact-agent interactions. For example, the system may determine the first metric based on a respective first measurement of each contact-agent interaction of the plurality of contact-agent interactions. In one embodiment, for example, the respective first measurement is based on measuring a respective outcome associated with each contact-agent interaction at the first time point, and a respective second measurement of each contact-agent interaction, wherein the respective second measurement is based on measuring, at the second time point, a contact center state after each contact-agent interaction, wherein the contact center state after each contact-agent interaction is based on whether the respective outcome associated with each contact-agent interaction is changed.

[0082] In some embodiments, the first measurement may be based on aggregating the measurements corresponding to each of the contact-agent interactions. For example, the systemmay generate a score based on a number, percentage, and / or other quantitative or qualitative assessment of the contact-agent interactions. For example, the system may aggregate the respective first measurement of each contact-agent interaction and determine the first metric based on aggregating the respective first measurement of each contact-agent interaction.

[0083] In some embodiments, when aggregating the measurements corresponding to each of the contact-agent interactions, the system may determine whether the interactions occurred under the same model. For example, the system may aggregate a first outcome and a second outcome in response to determining whether both outcomes relate to the same model. For example, the system may assign a first outcome associated with a first contact-agent interaction to a first model of the plurality of models in response to determining that the first contact-agent interaction occurred under the first model. The system may then assign a second outcome associated with a second contact-agent interaction to the first model in response to determining that the second contactagent interaction occurred under the first model. The system may then aggregate the first outcome and the second outcome to determine the first metric.

[0084] At step 406, process 400 (e.g., using one or more components described above) performs a respective second validation for each model of the plurality of models based on a second metric. For example, the system may perform a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on measuring a contact center state after each contact-agent interaction. For example, the system may use a metric that represents an after-the-call measurement. That is, the second metric may be based on a future state (e g., represented by the presence or absence of a second touch point during a threshold period of time) of the contact center system (e.g., whether there was a callback, whether the future callback was a cancellation, etc.).

[0085] In some embodiments, the second validation may include determining which contact-agent interactions corresponded to the same contact and / or contact account. For example, the system may determine that a retention metric for agent performance should be adjusted irrespective of whether the contact was “saved” on an initial interaction with an agent but then was canceled on a subsequent interaction with an agent. For example, performing the respective second validation for each model of the plurality of models based on the second metric may comprise the system assigning a first contact center state associated with a third contact-agent interaction to the first model in response to determining that a first contact corresponds to both the first contact-agentinteraction and the third contact-agent interaction. The system may then determine the second metric for the first model based on determining that the first contact corresponds to both the first contact-agent interaction and the third contact-agent interaction.

[0086] In some embodiments, the system may measure a contact center state. The contact center state may be a qualitative or quantitative metric. In some embodiments, the contact center state many correspond to a metric used for an outcome and / or result of a contact-agent interaction. The system may determine an initial value (or change in value) from an initial interaction and determine whether that initial value (or the change in value) is maintained during a subsequent interaction. For example, the system may determine whether a change to an account occurred during a first threshold proximity (e.g., during a first interaction). The system may also determine whether a change occurred during a second threshold proximity (e.g., during a second interaction). By determining whether the change occurred during either proximity, the system may determine whether a change (or lack thereof) was permanent or temporary. For example, by measuring the contact center state after each contact-agent interaction, the system may retrieve the respective contact account and determine whether the respective contact account state for the respective contact account changes from the first value to the second value within a second threshold proximity of the respective first contact time point. The system may perform a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on measuring a contact center state after each contact-agent interaction or further in time from the first contact-agent interaction (e.g., an outcome subsequently following a second contact-agent interaction corresponding to the first). For example, there may not be a second contact-agent interaction, but the second metric may still be taken 5 or 30 days after the initial interaction.

[0087] In some embodiments, the system may measure the contact center state based on changes to an account state of a contact. For example, the system may measure the contact center state based on whether the account state has increased in value or decreased in value. For example, the system may measure the contact center state after each contact-agent interaction by determining a respective contact account corresponding to each contact-agent interaction. The system may then determine a respective first contact time point corresponding to each contact-agent interaction. The system may then determine a first value of a respective contact account state for the respective contact account at the respective first contact time point. The system may then determine a secondvalue of the respective contact account state for the respective contact account at a second contact time point. The system may then determine whether the first value equals the second value.

[0088] In some embodiments, the system may measure the contact center state based on comparing outcomes from a first contact-agent interaction and a second contact-agent interaction. For example, when measuring the contact center state after each contact-agent interaction, the system may determine whether each contact-agent interaction corresponds to a subsequent contact-agent interaction and determine a subsequent respective outcome associated with the subsequent contactagent interaction.

[0089] In some embodiments, the system may perform the model validations based on one or more metrics. These metrics may be based on determining a value (e.g., whether an account was saved or canceled) after a contact-agent interaction. For example, the system may determine a first value of a contact account after a first contact-agent interaction, wherein the first contact-agent interaction occurred under a first model of the plurality of models. The system may then determine the first metric for the first model based on the first value. The system may determine a second value of the contact account after a second contact-agent interaction. The system may determine the second metric for the first model based on the second value.

[0090] In some embodiments, the system may increase (or decrease) the weight of a value based on whether or not multiple contact-agent interactions occurred under a model (e.g., the same pairing strategy) for the same contact. For example, the system may include additional weights (or penalties) to reflect whether a pairing strategy succeeded (or failed) multiple times. For example, determining the second metric for the first model based on the second value may comprise the system determining whether the second contact-agent interaction occurred under the first model. The system may then determine a weight (or other effect) of the second value on the second metric based on determining that the second contact-agent interaction occurred under the first model.

[0091] Vo example, a metric may be based on a difference or lack of a difference between a respective outcome (e.g., a respective outcome of a contact-agent interaction under a first model) and a contact center state (e.g., a contact center state after contact-agent interaction under the same or different model). The system may determine groupings of outcomes and contact center states based on common contacts and / or contact-agent interactions within a given time period.

[0092] In one example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state(e.g., a change or lack thereof in an account) under a different model. For example, the metric may be based on whether an account was eventually canceled under a second model after initially being saved under a first model. Alternatively, the metric may be based on whether an account was eventually saved under a second model after initially being canceled under a first model.

[0093] In another example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state (e.g., a change or lack thereof in an account) under the same model. For example, the metric may be based on whether an account was eventually canceled under a first model after initially being saved under the first model. Alternatively, the metric may be based on whether an account was eventually saved under a first model after initially being canceled under the first model.

[0094] In some embodiments, the metric may be based on instances in which the same model provided particular results (e.g., two contact-agent interactions that resulted in a save, cancelation, etc.). The metric may in such instances be based on additional weighting and / or emphasis. In some embodiments, the metric may be based on instances in which the same model provided particular frequencies of subsequent contact-agent interactions (e.g., a model that generates additional contact-agent interactions). The metric may in such instances be based on additional weighting and / or emphasis.

[0095] For example, when determining a metric for a first model during a period of time, the first model may be associated with various sets of calls, e.g.,: (i) a first set of calls, wherein each call in the first set of calls is a first touch, and has a value associated with a particular outcome; (ii) a second set of calls, wherein each call is a second touch, wherein the first touch for each call in the second set of calls was associated with a second model, and wherein the second touch for each call in the second set of calls was associated with the first model, and wherein each call in the second set of calls is associated with a value associated with a particular outcome after the second touch; (iii) a third set of calls, wherein each call is a second touch, wherein the first touch for each call in the third set of calls was associated with the first model, and wherein the second touch for each call in the third set of calls was associated with the second model, and wherein each call in the third set of calls is associated with a value associated with a particular outcome after the second touch; (iv) a fourth set of calls, wherein each call is a second touch, wherein the first touch for each call in the fourth set of calls was associated with the first model, and wherein the secondtouch for each call in the fourth set of calls was associated with the first model, and wherein each call in the fourth set of calls is associated with a value associated with a particular outcome after the second touch. For example, the metric may be based on a ratio / percentage / count of the first set of calls, the second set of calls, the third set of calls, and the fourth set of calls. In some examples, the fourth set of calls may be associated with a weighting to increase its weight relative to the other sets of calls. For example, the outcome may be successful (e.g., the metric may be associated with a save rate or a conversion rate) or the outcome may be unsuccessful (e.g., the metric may be associated with a cancel rate). In another example, a gain or lift metric may be determined based on at least one of the first set of calls, the second set of calls, the third set of calls, and the fourth set of calls. In another example, such a gain or lift metric may further include a weighting on the fourth set of calls to increase its weight relative to the other sets of calls.

[0096] For example, such a weight is 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75, 3, etc.

[0097] In another example, the metric may be based on a callback rate for each model (e.g., callback rate corresponds to a number / percentage / ratio of contacts who returned to the contact center within a threshold period of time after a first interaction of that contact with the contact center system).

[0098] At step 408, process 400 (e.g., using one or more components described above) selects a model based on the respective first validation and the respective second validation. For example, the system may select a model of the plurality of models based on the respective first validation for each model and the respective second validation for each model. For example, the system may use the two validations to perform a single, combined validation, where the metric used is a combination of an on-the-call measurement and an after-the-call measurement.

[0099] In some embodiments, the system may generate one or more recommendations for a model. The system may select the recommendation based on a ranking for the model. For example, the system may rank the plurality of models and select the model with the highest rank. For example, the system may select the model of the plurality of models based on the respective first validation for each model and the respective second validation for each model by ranking the plurality of models based on the respective first validation for each model and the respective second validation for each model. The system may determine a rank for the model based on ranking the plurality of models. The system may generate for display, on a user interface, a recommendation for the model based on the rank.

[0100] It is contemplated that the steps or descriptions of FIG. 4 may be used with any other embodiment of this disclosure. In addition, the steps and descriptions described in relation to FIG. 4 may be done in alternative orders or in parallel to further the purposes of this disclosure. For example, each of these steps may be performed in any order, in parallel, or simultaneously to reduce lag or increase the speed of the system or method. Furthermore, it should be noted that any of the components, devices, or equipment discussed in relation to the figures above could be used to perform one or more of the steps in FIG. 4.

[0101] FIG. 5 shows a flowchart for validating contact-agent pairing models, in accordance with one or more embodiments. For example, the system may use process 500 (e.g., as implemented on one or more system components described above) in order to validate contact-agent pairing models.

[0102] At step 502, process 500 (e.g., using one or more components described above) obtains a plurality of models and contact-agent interactions. For example, the system may obtain a plurality of models and a plurality of contact-agent interactions. By doing so, the system may retrieve one or more models that correspond to a particular pairing strategy and / or a version of a model trained on specific data (e.g., data corresponding to a specific date range).

[0103] At step 504, process 500 (e.g., using one or more components described above) determines a first measurement of each contact-agent interaction. For example, the system may determine a first measurement of each contact-agent interaction, wherein the first measurement is based on an existence and / or value of an outcome associated with each contact-agent interaction. For example, the first measurement may be according to any of the metrics discussed herein, including conversion rate, cancellation, or other successful or unsuccessful outcomes.

[0104] At step 506, process 500 (e.g., using one or more components described above) determines a second measurement of each contact-agent interaction. For example, the system may determine a second measurement of each contact-agent interaction, wherein the second measurement is based on a measurement of contact center state after each contact-agent interaction. For example, the system may use a metric to determine the second measurement that represents an after-the-call measurement. That is, the second metric may be based on a future state (e.g., represented by the presence or absence of a second touch point during a threshold period of time) of the contact center system (e.g., whether there was a callback, whether the future callback was a cancellation, etc.).

[0105] In some embodiments, the system may determine when two contact-agent interactions correspond. For example, the system may select a date range for the data to catch multitouch contacts and / or confirm whether the first respective measurement was stable (e.g., the value of the measurement did not change within the date range). For example, the date range may be 30 days (or 15, 60, 45, etc.). In some embodiments, the date range may be either floating or fixed. If floating, then each contact may have a look after within a predetermined period of time to see if a result was stable (e.g., does not change); for example, the predetermined period of time may be 5 days, 10 days, 15 days, 20 days, 30 days, 45 days, 60 days, 90 days, one month, two months, three months, etc. If fixed, then all contacts in a given time period (e.g., 5 days, 10 days, 15 days, 20 days, 30 days, 45 days, 60 days, 90 days, one month, two months, three months, etc.) are evaluated to determine whether the result changed within the fixed period time period.

[0106] In some embodiments, the system may determine a multitouch data point by selecting a date range for the data to catch multitouch contacts. For example, the date range may comprise a time period that includes a first time point (e.g., corresponding to a first metric and / or measurement) and a second time point (e.g., corresponding to a second or subsequent metric and / or measurement). For example, the first time point may comprise an initial interaction between a contact and an agent. For example, the system may receive a first time period for determining a multitouch data point. The system may determine a first time point based on the first time period. The system may determine the respective outcome (e.g., which may be based on the respective first measurement) associated with each contact-agent interaction at the first time point.

[0107] In some embodiments, measuring the respective outcome associated with each contactagent interaction may comprise determining the respective outcome associated with each contactagent interaction at a respective first time point and / or measuring the contact center state after each contact-agent interaction at a second time point. For example, the second time point may comprise a subsequent interaction between a contact and the same agent or a different agent.

[0108] In some embodiments, the system may receive a date range based on a user selection of a time period (e.g., 5 days, 10 days, 15 days, 20 days, 30 days, 45 days, 60 days, 90 days, one month, two months, three months). The system may automatically select the current date to correspond to the first time point and a subsequent date corresponding to the time period (e.g., a month later) as the second time point. For example, the system may receive the first time period for determiningthe multitouch data point, and / or the system may receive a first user input and determine the first time period based on the first user input.

[0109] At step 508, process 500 (e.g., using one or more components described above) performs a validation process for each model of the plurality of models based on the plurality of contactagent interactions. For example, the system may perform a validation process for each model of the plurality of models based on the plurality of contact-agent interactions, the first measurement associated with each contact-agent interaction, and the second measurement associated with each contact-agent interaction.

[0110] In some embodiments, the validation process may comprise analyzing a metric that reflects instances in which the same model provided particular results (e.g., two contact-agent interactions that resulted in a save, cancelation, etc.). For example, a metric may be based on a difference or lack of a difference between a respective outcome (e.g., a respective outcome of a contact-agent interaction under a first model) and a contact center state (e.g., a contact center state after contactagent interaction under the same or different model).

[0111] In one example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state (e.g., a change or lack thereof in an account) under a different model. For example, the metric may be based on whether an account was eventually canceled under a second model after initially being saved under a first model. Alternatively, the metric may be based on whether an account was eventually saved under a second model after initially being canceled under a first model.

[0112] In another example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state (e.g., a change or lack thereof in an account) under the same model. For example, the metric may be based on whether an account was eventually canceled under a first model after initially being saved under the first model. Alternatively, the metric may be based on whether an account was eventually saved under a first model after initially being canceled under the first model.

[0113] In some embodiments, the metric may be based on instances in which the same model provided particular results (e.g., two contact-agent interactions that resulted in a save, cancelation, etc.). The metric may in such instances be based on additional weighting and / or emphasis. In some embodiments, the metric may be based on instances in which the same model provided particularfrequencies of subsequent contact-agent interactions (e.g., a model that generates additional contact-agent interactions). The metric may in such instances be based on additional weighting and / or emphasis.

[0114] For example, when determining a metric for a first model during a period of time, the first model may be associated with various sets of calls, e.g.,: (i) a first set of calls, wherein each call in the first set of calls is a first touch, and has a value associated with a particular outcome; (ii) a second set of calls, wherein each call is a second touch, wherein the first touch for each call in the second set of calls was associated with a second model, and wherein the second touch for each call in the second set of calls was associated with the first model, and wherein each call in the second set of calls is associated with a value associated with a particular outcome after the second touch; (iii) a third set of calls, wherein each call is a second touch, wherein the first touch for each call in the third set of calls was associated with the first model, and wherein the second touch for each call in the third set of calls was associated with the second model, and wherein each call in the third set of calls is associated with a value associated with a particular outcome after the second touch; (iv) a fourth set of calls, wherein each call is a second touch, wherein the first touch for each call in the fourth set of calls was associated with the first model, and wherein the second touch for each call in the fourth set of calls was associated with the first model, and wherein each call in the fourth set of calls is associated with a value associated with a particular outcome after the second touch. For example, the metric may be based on a ratio / percentage / count of the first set of calls, the second set of calls, the third set of calls, and the fourth set of calls. In some examples, the fourth set of calls may be associated with a weighting to increase its weight relative to the other sets of calls. For example, the outcome may be successful (e.g., the metric may be associated with a save rate or a conversion rate) or the outcome may be unsuccessful (e.g., the metric may be associated with a cancel rate). In another example, a gain or lift metric may be determined based on at least one of the first set of calls, the second set of calls, the third set of calls, and the fourth set of calls. In another example, such a gain or lift metric may further include a weighting on the fourth set of calls to increase its weight relative to the other sets of calls.

[0115] For example, such a weight is 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75, 3, etc.

[0116] In another example, the metric may be based on a callback rate for each model (e.g., callback rate corresponds to a number / percentage / ratio of contacts who returned to the contactcenter within a threshold period of time after a first interaction of that contact with the contact center system).[Oil 7] In one example, the validation process is a combination of multiple metrics, e.g., including a first metric based on an existence and / or value of an outcome associated with each contact-agent interaction and a second metric based on a measurement of contact center state after each contactagent interaction (e.g., an after-the-call metric). In some embodiments, there may be weighting or normalizing applied to each metric.

[0118] At step 510, process 500 (e.g., using one or more components described above) select of model of the plurality of models. For example, the system may select a model of the plurality of models based on the validation process of step 508. For example, the system may rank the plurality of models based on the validation process of step 508 and select the model with the highest rank. The system may determine a rank for each model of the plurality of models based on ranking the plurality of models. The system may generate for display, on a user interface, a recommendation for a model of the plurality of models based on a rank associated with the model.

[0119] The above-described embodiments of the present disclosure are presented for purposes of illustration and not of limitation, and the present disclosure is limited only by the claims which follow. Furthermore, it should be noted that the features and limitations described in any one embodiment may be applied to any embodiment herein, and flowcharts or examples relating to one embodiment may be combined with any other embodiment in a suitable manner, done in different orders, or done in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and / or methods described above may be applied to, or used in accordance with, other systems and / or methods.

[0120] The present techniques will be better understood with reference to the following enumerated embodiments:1. A method for validating contact-agent pairing models prior to deployment in contact centers by verifying outcomes of multitouch data points.2. The method of the preceding embodiment, further comprising: obtaining a plurality of models and a plurality of contact-agent interactions; performing a respective first validation for each model of the plurality of models based on a first metric, wherein the first metric is based on a respective first measurement of each contact-agent interaction of the plurality of contact-agent interactions, wherein the respective first measurement is based on measuring a respective outcomeassociated with each contact-agent interaction; performing a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on measuring a contact center state after each contact-agent interaction; and selecting a model of the plurality of models based on the respective first validation for each model and the respective second validation for each model.3. The method of any one of the preceding embodiments, wherein measuring the respective outcome associated with each contact-agent interaction comprises determining the respective outcome associated with each contact-agent interaction at a respective first time point; and wherein measuring the contact center state after each contact-agent interaction comprises measuring the contact center state after each contact-agent interaction at a second time point.4. The method of any one of the preceding embodiments, wherein measuring the respective outcome associated with each contact-agent interaction further comprises: determining a respective contact account corresponding to each contact-agent interaction; determining a respective first contact time point corresponding to each contact-agent interaction; and determining whether a respective contact account state for the respective contact account changes from a first value to a second value within a first threshold proximity of the respective first contact time point.5. The method of any one of the preceding embodiments, wherein measuring the respective outcome associated with each contact-agent interaction further comprises: determining a first model of the plurality of models under which a first contact-agent interaction of the plurality of contact-agent interactions was paired; and assigning a first outcome associated with the first contact-agent interaction to the first model.6. The method of any one of the preceding embodiments, wherein measuring the contact center state after each contact-agent interaction further comprises: retrieving the respective contact account; and determining whether the respective contact account state for the respective contact account changes from the first value to the second value within a second threshold proximity of the respective first contact time point.7. The method of any one of the preceding embodiments, wherein determining the first metric based on aggregating the respective first measurement of each contact-agent interaction further comprises: assigning a first outcome associated with a first contact-agent interaction to a first model of the plurality of models in response to determining that the first contact-agent interaction occurred under the first model; assigning a second outcome associated with a second contact-agentinteraction to the first model in response to determining that the second contact-agent interaction occurred under the first model; and aggregating the first outcome and the second outcome to determine the first metric.8. The method of any one of the preceding embodiments, wherein performing the respective second validation for each model of the plurality of models based on the second metric further comprises: assigning a first contact center state associated with a third contact-agent interaction to the first model in response to determining that a first contact corresponds to both the first contactagent interaction and the third contact-agent interaction; and determining the second metric for the first model based on determining that the first contact corresponds to both the first contact-agent interaction and the third contact-agent interaction.9. The method of any one of the preceding embodiments, further comprising: determining a first value of a contact account after a first contact-agent interaction, wherein the first contactagent interaction occurred under a first model of the plurality of models; determining the first metric for the first model based on the first value; determining a second value of the contact account after a second contact-agent interaction; and determining the second metric for the first model based on the second value.10. The method of any one of the preceding embodiments, wherein determining the second metric for the first model based on the second value further comprises: determining whether the second contact-agent interaction occurred under the first model; and determining a weight of the second value on the second metric based on determining that the second contact-agent interaction occurred under the first model.11. The method of any one of the preceding embodiments, wherein measuring the contact center state after each contact-agent interaction further comprises: determining a respective contact account corresponding to each contact-agent interaction; determining a respective first contact time point corresponding to each contact-agent interaction; determining a first value of a respective contact account state for the respective contact account at the respective first contact time point; determining a second value of the respective contact account state for the respective contact account at a second contact time point; and determining whether the first value equals the second value.12. The method of any one of the preceding embodiments, wherein measuring the contact center state after each contact-agent interaction further comprises: determining whether eachcontact-agent interaction corresponds to a subsequent contact-agent interaction; and determining a subsequent respective outcome associated with the subsequent contact-agent interaction.13. The method of any one of the preceding embodiments, wherein selecting the model of the plurality of models based on the respective first validation for each model and the respective second validation for each model further comprises: ranking the plurality of models based on the respective first validation for each model and the respective second validation for each model; determining a rank for the model based on ranking the plurality of models; and generating for display, on a user interface, a recommendation for the model based on the rank.14. The method of any one of the preceding embodiments, further comprising: aggregating the respective first measurement of each contact-agent interaction; and determining the first metric based on aggregating the respective first measurement of each contact-agent interaction.15. A non-transitory, computer-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising those of any of embodiments 1-14.16. A system comprising one or more processors; and memory storing instructions that, when executed by the processors, cause the processors to effectuate operations comprising those of any of embodiments 1-14.17. A system comprising means for performing any of embodiments 1-14.

Claims

WHAT IS CLAIMED IS:

1. A system for validating contact-agent pairing models prior to deployment in contact centers by verifying outcomes of multitouch data points, the system comprising: one or more processors; and a non-transitory, computer-readable medium comprising instructions recorded thereon that, when executed by the one or more processors, cause operations comprising: storing, in a first database, a plurality of models and a plurality of contact-agent interactions, wherein each model of the plurality of models corresponds to a respective pairing strategy for respective contacts and respective agents; receiving, via a user input into a user interface, a first time period for determining a multitouch data point; performing a respective first validation for each model of the plurality of models based on a first metric, wherein the respective first validation validates an accuracy of each model based on determining metrics for the multitouch data point, and wherein the first metric is based on: a respective first measurement of each contact-agent interaction of the plurality of contact-agent interactions, wherein the respective first measurement is based on measuring a respective outcome associated with each contact-agent interaction at a respective first time point; and a respective second measurement of each contact-agent interaction of the plurality of contact-agent interactions, wherein the respective second measurement is based on measuring, at a respective second time point, a contact center state after each contact-agent interaction, wherein the contact center state after each contact-agent interaction is based on whether the respective outcome associated with each contact-agent interaction is changed; performing a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on a respective third measurement of each contact-agent interaction of the plurality of contact-agent interactions; selecting a model of the plurality of models based on the respective first validation for each model and the respective second validation for each model; and establishing a connection between communication equipment of a contact andcommunication equipment of an agent based on the model.

2. A method for validating contact-agent pairing models prior to deployment in contact centers by verifying outcomes of multitouch data points, the method comprising: obtaining a plurality of models and a plurality of contact-agent interactions; performing a respective first validation for each model of the plurality of models based on a first metric, wherein the first metric is based on a respective first measurement of each contactagent interaction of the plurality of contact-agent interactions, and wherein the respective first measurement is based on measuring a respective outcome associated with each contact-agent interaction; performing a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on measuring a contact center state after each contact-agent interaction; and selecting a model of the plurality of models based on the respective first validation for each model and the respective second validation for each model.

3. The method of claim 2, wherein measuring the respective outcome associated with each contact-agent interaction comprises determining the respective outcome associated with each contact-agent interaction at a respective first time point; and wherein measuring the contact center state after each contact-agent interaction comprises determining to measure the contact center state after each contact-agent interaction at a second time point.

4. The method of claim 2, wherein measuring the respective outcome associated with each contact-agent interaction further comprises: determining a respective contact account corresponding to each contact-agent interaction; determining a respective first contact time point corresponding to each contact-agent interaction; and determining whether a respective contact account state for the respective contact account changes from a first value to a second value within a first threshold proximity of the respective first contact time point.

5. The method of claim 4, wherein measuring the respective outcome associated with each contact-agent interaction further comprises: determining a first model of the plurality of models under which a first contact-agent interaction of the plurality of contact-agent interactions was paired; and assigning a first outcome associated with the first contact-agent interaction to the first model.

6. The method of claim 4, wherein measuring the contact center state after each contact-agent interaction further comprises: retrieving the respective contact account; and determining whether the respective contact account state for the respective contact account changes from the first value to the second value within a second threshold proximity of the respective first contact time point.

7. The method of claim 2, further comprising: aggregating the respective first measurement of each contact-agent interaction; and determining the first metric based on aggregating the respective first measurement of each contact-agent interaction.

8. The method of claim 2, wherein determining the first metric based on aggregating the respective first measurement of each contact-agent interaction further comprises: assigning a first outcome associated with a first contact-agent interaction to a first model of the plurality of models in response to determining that the first contact-agent interaction occurred under the first model; assigning a second outcome associated with a second contact-agent interaction to the first model in response to determining that the second contact-agent interaction occurred under the first model; and aggregating the first outcome and the second outcome to determine the first metric.

9. The method of claim 8, wherein performing the respective second validation for each model of the plurality of models based on the second metric further comprises:assigning a first contact center state associated with a third contact-agent interaction to the first model in response to determining that a first contact corresponds to both the first contactagent interaction and the third contact-agent interaction; and determining the second metric for the first model based on determining that the first contact corresponds to both the first contact-agent interaction and the third contact-agent interaction.

10. The method of claim 2, further comprising: determining a first value of a contact account after a first contact-agent interaction, wherein the first contact-agent interaction occurred under a first model of the plurality of models; determining the first metric for the first model based on the first value; determining a second value of the contact account after a second contact-agent interaction; and determining the second metric for the first model based on the second value.

11. The method of claim 10, wherein determining the second metric for the first model based on the second value further comprises: determining whether the second contact-agent interaction occurred under the first model; and determining a weight of the second value on the second metric based on determining that the second contact-agent interaction occurred under the first model.

12. The method of claim 2, wherein measuring the contact center state after each contact-agent interaction further comprises: determining a respective contact account corresponding to each contact-agent interaction; determining a respective first contact time point corresponding to each contact-agent interaction; determining a first value of a respective contact account state for the respective contact account at the respective first contact time point; determining a second value of the respective contact account state for the respective contact account at a second contact time point; and determining whether the first value equals the second value.

13. The method of claim 2, wherein measuring the contact center state after each contact-agent interaction further comprises: determining whether each contact-agent interaction corresponds to a subsequent contactagent interaction; and determining a subsequent respective outcome associated with the subsequent contact-agent interaction.

14. The method of claim 2, wherein selecting the model of the plurality of models based on the respective first validation for each model and the respective second validation for each model further comprises: ranking the plurality of models based on the respective first validation for each model and the respective second validation for each model; determining a rank for the model based on ranking the plurality of models; and generating for display, on a user interface, a recommendation for the model based on the rank.

15. A non-transitory, computer-readable medium having instructions recorded thereon that, when executed by one or more processors, cause operations comprising: obtaining a plurality of models and a plurality of contact-agent interactions; determining a respective first measurement of each contact-agent interaction of the plurality of contact-agent interactions, wherein the respective first measurement is based on measuring a respective outcome associated with each contact-agent interaction; determining a respective second measurement of each contact-agent interaction, wherein the respective second measurement is based on measuring a contact center state after each contactagent interaction; performing a respective first validation for each model of the plurality of models based on the respective first measurement and the respective second measurement; and selecting a model of the plurality of models based on the respective first validation.

16. The non-transitory, computer-readable medium of claim 15, wherein the instructions further cause operations comprising: receiving a first time period for determining a multitouch data point; determining a first time point based on the first time period; and determining the respective outcome associated with each contact-agent interaction at the first time point.

17. The non-transitory, computer-readable medium of claim 16, wherein the instructions further cause operations comprising: determining a second time point based on the first time period; and determining to measure the contact center state after each contact-agent interaction, at the second time point.

18. The non-transitory, computer-readable medium of claim 16, wherein receiving the first time period for determining the multitouch data point further comprises: receiving a first user input; and determining the first time period based on the first user input.

19. The non-transitory, computer-readable medium of claim 16, wherein receiving the first time period for determining the multitouch data point further comprises: receiving a frequency at which to validate the plurality of models; and determining the first time period based on the frequency.

20. The non-transitory, computer-readable medium of claim 15, wherein measuring the respective outcome associated with each contact-agent interaction further comprises: determining a respective contact account corresponding to each contact-agent interaction; determining a respective first contact time point corresponding to each contact-agent interaction; and determining whether a respective contact account state for the respective contact account changes from a first value to a second value within a first threshold proximity of the respective first contact time point.

Images