Evaluate and Enhance Conversational Bots with Meaningful Auto-Connect Scores

Abstract

The disclosed embodiments assist bot managers and builders in identifying specific friction points between bots and customers to enable real-time identification of bot conversation problems, and provide a framework for training bots to improve conversation flow. Conversation data is processed using machine learning models to detect bot states within conversations and calculate Meaningful Auto-Connection Scores (MACS) for these conversations. The MACS for the conversations are provided to bot builders to enable them to identify friction points and update the bots accordingly.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 128,992, filed December 22, 2020, the disclosure of which is incorporated herein by reference.

[0002]

[0002] The present disclosure relates generally to a system and method for facilitating the evaluation of conversational bots using Meaningful Automated Connection Score (MACS). More particularly, techniques are provided for assisting bot managers and builders in identifying specific friction points between bots and customers to enable real-time identification of bot conversation problems, and deploying a framework for training bots to improve conversation flow. Summary of the Invention

[0003]

[0003] The disclosed embodiments assist bot managers and builders in identifying specific friction points between bots and customers to enable real-time identification of bot conversation problems, and provide a framework for training bots to improve conversation flow. According to some embodiments, a computer-implemented method is provided. The computer-implemented method comprises receiving conversation data corresponding to a conversation between a conversational bot agent and a customer. The conversation data includes messages exchanged between the conversational bot agent and the customer. The computer-implemented method further comprises computing a set of features from the conversation data. The computer-implemented method further comprises detecting a set of bot states corresponding to the performance of the conversational bot agent in the conversation. The set of bot states is detected based on the set of features and the conversation data. The computer-implemented method further comprises training a machine learning model using the sample conversation data and the corresponding bot states. The training includes determining a Meaningful Auto-Connection Score (MACS) for the conversation. The computer-implemented method further comprises providing the MACS for the conversation and the set of bot states. The computer-implemented method further comprises receiving updates to the conversational bot agent and implementing the updates to the conversational bot agent. The updates are used to train the machine learning model.

[0004] In one example, a system includes one or more processors and a memory containing instructions that, when executed by the one or more processors, cause the system to perform a process described herein. In another example, a non-transitory computer-readable storage medium has executable instructions stored thereon that, when executed by one or more processors of a computer system, cause the computer system to perform a process described herein.

[0005]

[0005] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter, which subject matter should be understood by reference to appropriate portions of the entire specification of this patent application, any or all drawings, and each claim.

[0006]

[0006] The above, together with other features and examples, are described in more detail below in the following specification, claims, and accompanying drawings.

[0007]

[0007] Various embodiments of the present disclosure are discussed in detail below. Although specific implementations are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the art will recognize that other components and configurations can be used without departing from the spirit and scope of the present disclosure. Therefore, the following description and drawings are illustrative and should not be construed as limiting. Numerous specific details are described to provide a thorough understanding of the present disclosure. However, in some instances, well-known or conventional details are not described to avoid obscuring the description. Reference to an embodiment or an embodiment in the present disclosure may be a reference to the same embodiment or any embodiment, and such reference means at least one of the embodiments.

[0008]

[0008] Reference to "one embodiment" or "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. The appearances of the phrase "one embodiment" in various places in this specification do not necessarily all refer to the same embodiment, or to separate or alternative embodiments that are mutually exclusive of other embodiments. Moreover, various features are described that may be exhibited by some embodiments but not by other embodiments.

[0009]

[0009] The terms used herein generally have their ordinary meanings in the art, within the context of this disclosure and in the specific context in which each term is used. Alternative wording and synonyms may be used for any one or more of the terms described herein, and no special importance should be placed on whether a term is detailed or explained herein. In some cases, synonyms for some terms are provided. The detailed explanation of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any term described herein, is illustrative only and is not intended to further limit the scope and meaning of the disclosure or any exemplary term. Similarly, the disclosure is not limited to the various embodiments provided herein.

[0010]

[0010] Without intending to limit the scope of the present disclosure, examples of devices, apparatus, methods and their related results according to the embodiments of the present disclosure are given below. Please note that titles or subtitles may be used in the examples for the convenience of the reader, and in no case should this limit the scope of the present disclosure. Unless otherwise defined, technical and scientific terms used herein have the meaning as commonly understood by those skilled in the art to which the present disclosure pertains. In case of conflict, the present disclosure, including definitions, shall control.

[0011]

[0011] Additional features and advantages of the present disclosure will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the principles disclosed herein. The features and advantages of the present disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the present disclosure will become more fully apparent from the following description and the appended claims, or may be learned by practice of the principles described herein.

[0012]

[0012] The present disclosure is described with reference to the accompanying drawings. [Brief description of the drawings]

[0013] [Figure 1]

[0013] A diagram showing an illustrative example of an environment in which MACS for a conversational bot agent is determined and provided to a bot builder to enable the generation of bot updates to improve the performance of the conversational bot agent, according to various embodiments. [Diagram 2]

[0014] FIG. 1 illustrates an illustrative example of an environment in which a bot state machine learning modeling engine performs parallel training of a bot state predictive model and a MACS predictive model using bot state and MACS annotations, according to at least one embodiment. [Diagram 3]

[0015] FIG. 1 illustrates an illustrative example of an environment in which a bot state machine learning modeling engine performs parallel training of a bot state predictive model and a MACS predictive model using bot state and MACS annotations and features, according to at least one embodiment. [Figure 4]

[0016] FIG. 1 illustrates an illustrative example of an environment in which a bot state machine learning modeling engine performs training of a MACS predictive model using predictions generated by a bot state predictive model, in accordance with at least one embodiment. [Diagram 5]

[0017] FIG. 1 illustrates an illustrative example of an environment in which a bot state machine learning modeling engine performs training of a MACS predictive model using predictions and features generated by a bot state predictive model, according to at least one embodiment. [Figure 6]

[0018] FIG. 1 illustrates an illustrative example of an environment in which a bot state machine learning modeling engine uses bot state and MACS annotations and features to train a joint bot state and MACS prediction model, according to at least one embodiment. [Figure 7]

[0019] A diagram illustrating an illustrative example of an environment in which a MACS predictive model provides a MACS for one or more bot agents based on conversation data and identified bot states for the one or more bot agents, according to at least one embodiment. [Figure 8]

[0020] FIG. 1 illustrates an illustrative example of an environment in which a bot state machine learning modeling engine performs real-time analysis of messages of active communication sessions to dynamically determine bot states and MACS for a conversational bot agent, according to at least one embodiment. [Figure 9]

[0021] FIG. 1 illustrates an illustrative example of a process for training a bot state prediction model using conversation data and calculated probabilities for bot states corresponding to previously analyzed bot conversations, according to at least one embodiment. [Figure 10]

[0022] FIG. 1 illustrates an illustrative example of a process for training a MACS predictive model based on annotated conversation data and identified bot states, according to at least one embodiment. [Figure 11]

[0023] FIG. 1 illustrates an illustrative example of a process for generating and providing MACS and any bot state for a conversation between a bot agent and a customer, according to at least one embodiment. [Figure 12]

[0024] FIG. 1 illustrates an illustrative example of an environment in which various embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014]

[0025] In the accompanying figures, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. When only a first reference label is used herein, the description is applicable to any of the similar components that have the same first reference label, regardless of the second reference label.

[0015]

[0026] The following description merely provides preferred examples of embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of preferred examples of embodiments provides those skilled in the art with an enablement description for implementing the preferred examples of embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0016]

[0027] FIG. 1 illustrates an illustrative example of an environment in which a MACS for a conversational bot agent 104 is determined and provided to a bot builder 116 to enable generation of bot updates to improve the performance of the conversational bot agent 104, according to various embodiments. In the environment 100, a customer 112 may be engaged in a communication session with a conversational bot agent 104 of a customer service call center 102 via a computing device 114. The conversational bot agent 104 may be configured to communicate autonomously with network devices, such as the computing device 114. Additionally, the conversational bot agent 104 may be configured for specific capabilities. Example capabilities may include updating database records, providing updates to the customer, providing additional data about the customer 112 to a human agent or other conversational bot agent, determining the intent of the customer and routing the customer 112 to a destination system based on the intent, predicting or suggesting a response to a human agent communicating with the customer, expanding the communication session to include one or more additional bots or human agents, and other suitable capabilities. In some implementations, while the conversational bot agent 104 is communicating with the customer 112 during a communication session (e.g., using a chat-enabled interface), a communication server (not shown) can automatically and dynamically decide to switch the conversational bot agent 104 with a terminal device utilized by a human agent. For example, the conversational bot agent 104 can communicate with the customer for some tasks (e.g., updating database records related to the customer 112), while the human agent can communicate with the customer for more difficult tasks (e.g., communicating using a communication channel to resolve a technical issue).

[0017]

[0028] The conversational bot agent 104 can be code configured, when executed, to autonomously communicate with a customer, such as the customer 112, via the computing device 114. For example, the conversational bot agent 104 can be a bot that automatically generates messages to initiate a conversation with the customer 112 associated with the computing device 114 and / or automatically respond to messages from the computing device 114. In one embodiment, the customer service call center 102 can enable clients (e.g., systems external to the customer service call center 102 platform) to deploy the conversational bot agent 104 within their internal communication systems via the customer service call center 102. In some examples, the clients can use their own bots within the customer service call center's platform, which enables the clients to implement the methods and techniques described herein within their internal communication systems.

[0018] In some implementations, the conversational bot agent 104 may be defined by one or more sources. For example, a data store of the customer service call center 102 may store code representing the conversational bot agent 104 defined (e.g., written or coded) by a client of the customer service call center 102. For example, a client that has defined its own conversational bot agent 104 may load the conversational bot agent 104 into the customer service call center 102. The conversational bot agent 104 defined by the client may be stored in a client bot data store. In some examples, the customer service call center 102 may include a data store that may be used to store code representing the conversational bot agent 104 defined by a third-party system. For example, the third-party system may include an independent software vendor. Another data store may store code representing the conversational bot agent 104 defined by an entity associated with the customer service call center 102. For example, a conversational bot agent 104 coded by an entity can be loaded or made accessible by a customer service call center 102 such that the conversational bot agent 104 can execute and communicate autonomously with a customer. In some implementations, the customer service call center 102 can access the bots stored in these data stores using a cloud network. The cloud network can be any network and can include the Internet, a personal area network, an open network such as a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a wireless local area network (WLAN), a private network such as an intranet, an extranet, or other backbone.

[0019]

[0029] In some embodiments, the customer service call center 102 can recommend automation to dynamically switch the conversation between the conversational bot agent 104 and the customer 112 during a particular communication session between the conversational bot agent 104 and the computing device 114. For example, the customer service call center 102 can facilitate a communication session between the computing device 114 and the conversational bot agent 104. The conversational bot agent 104 can be configured to autonomously communicate with the computing device 114 by exchanging one or more messages with the computing device 114 during the communication session. The customer service call center 102 can dynamically determine whether to switch the conversational bot agent 104 with a human agent (or vice versa, as the case may be) such that the live human agent can communicate with the computing device 114 rather than the conversational bot agent 104. In some implementations, the switch can be performed without prompting from the customer 112 or the human agent. For example, the switching can be based on message parameters (e.g., a score representing the sentiment of a message or series of messages) of messages exchanged between the computing device 114 and the conversational bot agent 104, without prompting the computing device to request a transfer to a human agent.

[0020]

[0030] In one embodiment, the customer service call center 102 records conversations between customers and the conversational bot agent 104 for evaluation to identify any bot states encountered during these conversations. These bot states may include any failure states (e.g., asking for outdated information, any misunderstandings, inability to assist the customer, inability to transfer the customer to a live agent, some re-prompts, etc.), information gathering states (e.g., customer identification, prompting for new information, requesting confirmation, etc.), transfer states (e.g., offering to transfer to a live agent, prompting the customer to transfer to a live agent, successful transfer, etc.), assistance states (e.g., providing useful information to the customer, addressing the customer's issue, providing an accurate response to a query, etc.), etc. For example, when a customer 112 engages in a conversation with a conversational bot agent 104, the customer service call center 102 may record each message along with any associated bot metadata in the conversation data store 106. Within the conversation data store 106, the customer service call center 102 may associate each message with a unique identifier that corresponds to a particular conversation between the customer 112 and the conversational bot agent 104. The customer service call center 102 may monitor a conversation between a customer 112 and a conversational bot agent 104 to determine whether the conversation has ended. For example, the customer service call center 102 may determine that a particular conversation has ended as a result of the conversation being transferred from the conversational bot agent 104 to a human agent. Alternatively, the customer service call center 102 may determine that a conversation has ended as a result of the customer 112 submitting a message indicating that a particular issue has been resolved or other acknowledgment of the end of the conversation. In some instances, the customer service call center 102 may determine that a conversation has ended when a communication session between the conversational bot agent 104 and the computing device 114 utilized by the customer 112 has ended.

[0021]

[0031] In one embodiment, a sample set of conversations stored in the conversation data store 106 is provided to a set of annotators 110 for manual evaluation. For example, in some instances, a particular conversation may be provided to a set of annotators 110 to identify any bot states occurring during the particular conversation. These bot states may include the bot's inability to understand the customer's message (e.g., intent, problem, etc.), the bot ignoring the customer, the bot being stuck in a message loop, the bot's detection of customer frustration, the bot's incorrect transfer to a human agent, etc. Additionally, these bot states may include any actions performed by the bot that result in successful transfer of the conversation to a live agent, successful resolution of the problem expressed by the customer during the conversation, etc. The bot states may further include any information gathering activities, such as prompting the customer to provide identifying information, any prompts for new information that may be used to address the problem, any requests for confirmation regarding the action to be performed by the bot, etc. In one embodiment, each annotator 110 may evaluate each message of a particular conversation and indicate any specific states detected corresponding to the message. For example, annotator 110 may annotate each message of a particular conversation to indicate whether the message indicates a particular fault condition, indicates a particular forwarding condition, indicates a particular information gathering condition, etc.

[0022]

[0032] In one embodiment, the annotated data as well as original conversation data corresponding to a conversation between a customer and the conversational bot agent 104 are provided to a bot state machine learning modeling engine 108 to generate and train a bot state prediction model that can be used to dynamically and in real-time evaluate messages between the customer and the conversational bot agent 104 to predict the bot state at a conversation level. The bot state machine learning modeling engine 108 may be implemented on a computer system or other system (e.g., a server, a virtual machine instance, etc.) of the customer service call center 102. Alternatively, the bot state machine learning modeling engine 108 may be implemented as an application or other process running on a computing system of the customer service call center 102.

[0023]

[0033] In one embodiment, the bot state machine learning modeling engine 108 implements an aggregation strategy to determine the probability that any message in a conversation has a particular state. For example, the bot state machine learning modeling engine 108 may calculate the probability that a particular state exists for each message in a conversation. As an illustrative example, if a particular message is evaluated by three annotators 110 and one of these annotators 110 determines that a particular state exists for a particular message, the bot state machine learning modeling engine 108 may determine that the probability that a particular state exists in the message is the number of annotations indicating the particular state divided by the total number of annotations, i.e., 1 / 3 in this case. The bot state machine learning modeling engine 108 may subtract this probability from 1 to obtain the probability that a particular message of a transformation does not contain this particular state. The bot state machine learning modeling engine 108 may multiply the obtained probabilities for each message in a particular conversation to obtain the probability that a particular state does not exist in a particular conversation. The bot state machine learning modeling engine 108 may subtract this probability from 1 to obtain the probability that a particular state exists in a particular conversation.

[0024]

[0034] The resulting probability of the presence of a particular state in a conversation may be monotonically decreasing, whereby the presence of more messages may not decrease the probability that a particular state exists. For example, if all annotators 110 agree that a particular state exists in a particular message, then the probability that a particular state exists in a particular conversation is 1 regardless of the number of messages in the particular conversation. The resulting annotated conversations and corresponding state probabilities may be used as a "ground truth" or sample dataset for training a classification model that can be used to dynamically detect states from conversations in real time. Thus, each message in the sample dataset may be known to correspond to a particular classification (e.g., customer frustration, the bot ignored the customer, the bot was stuck in a message loop, the bot asked the customer for new information to resolve the issue, the bot successfully transferred the conversation to a live agent, the bot successfully addressed the issue, etc.) according to the previously calculated probabilities corresponding to the presence of each classification. Classification models that may be used to classify messages from a particular conversation for a particular fault condition may include a logistic regression algorithm, a random forest model, a naive Bayes model, a linear regression model, a decision tree model, a k-means clustering model, a k-nearest neighbor (kNN) model, a support vector machine (SVM) model, a gradient boosting machine model, and the like.

[0025]

[0035] In one embodiment, the bot state prediction model provides as an output, for each conversation processed by the bot state prediction model, a confidence score for each possible bot state. The confidence score may correspond to a probability that a particular bot state is present in the conversation. In some cases, the confidence score may be compared to the aforementioned probability that the corresponding state is present in the particular conversation to determine whether the classification model is producing an accurate result. For example, the classification model used to calculate the confidence score for each possible state for a particular conversation may be evaluated to determine whether the classification model correctly identifies any bot states that may be present in the particular conversation based on the bot state probabilities corresponding to the annotator evaluation of the particular conversation. Based on this evaluation, the classification model may be modified to increase the likelihood that the classification model produces a desired result.

[0026]

[0036] In one embodiment, the bot state machine learning modeling engine 108 can convert the confidence score generated by the bot state predictive model (e.g., classification model) into a MACS for a particular conversation. For example, the bot state machine learning modeling engine 108 may use the confidence score for each bot state obtained using the bot state predictive model for a particular conversation as an input to the MACS predictive model to obtain the MACS for the particular conversation. Alternatively, the bot state machine learning modeling engine 108 may determine whether the confidence score for each bot state meets a threshold. If the confidence score for a particular bot state is above a threshold, the bot state machine learning modeling engine 108 may determine that a corresponding bot state exists in the conversation. The presence of the bot state in the conversation may be used as an input to the MACS predictive model according to a threshold to calculate the MACS for the conversation.

[0027]

[0037] The MACS prediction model may be trained through various techniques. In one embodiment, the MACS prediction model may be trained in parallel with the bot state prediction model by using the bot state and MACS annotations generated by the annotator 110 for one or more sample conversations as inputs to generate a MACS for each of the one or more sample conversations. Based on the bot state probabilities corresponding to the annotator evaluation of a particular conversation and the MACS annotations for the conversation, the resulting MACS may be evaluated to determine whether the MACS prediction model correctly calculates the MACS for the particular conversation. Based on this evaluation, the MACS prediction model may be modified to increase the likelihood that the MACS prediction model will generate a desired result.

[0028]

[0038] In one embodiment, the MACS prediction model may be trained using the bot state and MACS annotations generated by the annotator 110 for one or more sample conversations and one or more sets of sample conversation features. For example, the bot state machine learning modeling engine 108 may utilize a feature extractor to extract the conversation length of a particular conversation and determine a binary feature indicating whether the conversation length is at either extreme of conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may increase the correlation between the MACS score determined by the annotator 110 based on a manual evaluation of the conversation and the MACS score predicted by the MACS prediction model. Thus, this binary feature may be used to further improve the training of the MACS prediction model.

[0029]

[0039] In some embodiments, the MACS prediction model may be trained using the predicted bot state confidence scores generated by the bot state prediction model described above. For example, the bot state machine learning modeling engine 108 may use the confidence scores generated by the bot state prediction model and the annotated MACS determined by the annotator 110 for the sample conversation as inputs to the MACS prediction model to generate a predicted MACS for the sample conversation. This predicted MACS may be compared to the annotated MACS determined by the annotator 110 to determine whether the MACS prediction model is generating accurate results. Based on this evaluation, the MACS prediction model may be modified to increase the likelihood that the MACS prediction model will generate a desired MACS for the sample conversation. In some instances, the MACS prediction model may be trained using a combination of the predicted bot state confidence scores and a set of features for one or more sample conversations. The set of features may be similar to those described above, including binary features indicating whether the conversation length is at either extreme of the conversation length.

[0030]

[0040] In one embodiment, an evaluation of a MACS prediction model based on the predicted MACS for the different sample conversations and the annotated MACS for these different sample conversations determined by the annotator 110 may be used to further determine the impact of each bot state on the MACS for the different sample conversations. For example, based on the difference between the predicted MACS and the annotated MACS determined by the annotator 110, a relative impact of the bot state on the MACS may be determined. This determined impact may be used to train the MACS prediction model to better predict the MACS for a given conversation based on the presence (or absence) of different bot states, as determined by the bot state prediction model or based on a bot state confidence score or other indicator of the presence of a particular bot state as determined by the annotator 110.

[0031]

[0041] The resulting MACS may correspond to the quality of the conversation and the performance of the conversational bot agent 104 with respect to the particular conversation. For example, a MACS of "1" may correspond to a bad conversation in which the conversational bot agent 104 was not helpful in addressing the customer's particular problem. Alternatively, a MACS of "5" may indicate that the conversational bot agent 104 was helpful (e.g., providing relevant actions or information related to the customer's intent) and that the conversation was uneventful, resulting in a positive customer experience. It should be noted that a range of possible MACS values ​​may be determined by the customer service call center 102 to indicate possible performance metrics for the conversational bot agent 104 with respect to a particular conversation with a customer. Thus, other MACS ranges may be used in addition to or as an alternative to the aforementioned "1-5" range.

[0032]

[0042] In one embodiment, once the bot state prediction model and the MACS prediction model are trained to process a new conversation between the conversational bot agent 104 and a customer, the customer service call center 102 can implement these prediction models to detect bot states within an active conversation and calculate the MACS for the conversation in real time. For example, when a customer 112 is engaged in a conversation with the conversational bot agent 104 via a communication session between the conversational bot agent 104 and a computing device 114, the bot state machine learning modeling engine 108 may process messages between the customer 112 and the conversational bot agent 104 to detect any bot states for the conversational bot agent 104 and determine the MACS for the conversation based on these bot states. As more bot states are detected, the bot state machine modeling engine 108 may continue to decrement the MACS for the conversation (via the MACS prediction model) to represent a deterioration in the performance of the conversational bot agent 104. When a conversation with the conversational bot agent 104 ends (e.g., the conversation is transferred to a human agent, the customer 112 disconnects from the communication session, etc.), the bot state machine learning modeling engine 108 may calculate a final MACS for the conversation based on the bot state (if any) detected by the bot state predictive model.

[0033]

[0043] In one embodiment, the bot state machine learning modeling engine 108 provides the MACS for each conversation to the bot builders 116 to enable these bot builders 116 to update the conversational bot agents 104. The MACS for the conversations may be provided by the customer service call center 102 and provided via an interface or portal accessible via the computing device 118 utilized by the bot builders 116. In some instances, the bot state machine learning modeling engine 108 may provide the MACS for the conversations in which the conversational bot agents 104 are constructed by the bot builders 116, as well as the detected bot states and other insights that may be useful to the bot builders 116 to update the conversational bot agents 104. For example, if the MACS for a particular class of conversational bot agents 104 is consistently below a threshold as a result of encountering a particular set of less favorable bot states (e.g., the conversational bot agents 104 are performing poorly), the bot builders 116 may generate updates to the conversational bot agents 104 to reduce the likelihood of these less favorable bot states occurring and improve the MACS for these conversational bot agents 104.

[0034]

[0044] As described above, the bot state machine learning modeling engine 108 can detect bot states and calculate the MACS in real time for a particular conversation or for each message in a particular conversation. In one embodiment, a particular conversation can be monitored by a live agent (e.g., a human), whereby the live agent can be provided with information about the real-time MACS for the conversation and any detected bot states. If the MACS for the conversation falls below a threshold or some bot state (e.g., a particular fault state, etc.) is detected, the live agent can transfer the conversation from the conversational bot agent 104 to the live agent. This transfer can act as a fault state in some cases and can be used to calculate a final MACS for the conversation. The final MACS for the conversation can be provided to the bot builder 116 as described above.

[0035]

[0045] FIG. 2 illustrates an illustrative example of an environment 200 in which a bot state machine learning modeling engine 202 performs parallel training of a bot state prediction model 210 and a MACS prediction model 222 using bot state and MACS annotations, according to at least one embodiment. In the environment 200, the bot state machine learning modeling engine 202 performs training 204 of the bot state prediction model 210 using annotated data 214 generated by an annotator based on sample conversations between a conversational bot agent and a customer or other entity, and conversation data corresponding to the sample conversations evaluated by the annotator, to generate annotated data 214. For example, the annotator may evaluate these sample conversations to identify different bot states of the conversational bot agent involved in the sample conversations. Further, based on these bot states, the annotator may calculate a MACS for each sample conversation. The different bot states and corresponding MACS for the sample conversations may be stored as annotated data 214. The annotated data 214 may include bot state annotation data 216 (e.g., data generated by each annotator indicating the bot states detected in the sample conversation) and bot state and MACS annotation data 218 (e.g., data generated by each annotator indicating the bot states detected in the sample conversation and the corresponding MACS for the sample conversation).

[0036]

[0046] In one embodiment, to perform training 204 of the bot state prediction model 210, the bot state machine learning modeling engine 202 processes the bot state annotation data 216 and uses the soft score thresholding engine 212 to determine probabilities corresponding to the likelihood that a bot state exists in a sample conversation. As described above, the bot state machine learning modeling engine 202 may implement an aggregation strategy to determine the probability that any message in a conversation has a particular state. For example, the bot state machine learning modeling engine 202 may calculate the probability that a particular state exists for each message in a conversation via the soft score thresholding engine 212. The soft score thresholding engine 212 may multiply the obtained probabilities for each message in a particular conversation to obtain the probability that a particular state is not in a particular conversation. The soft score thresholding engine 212 may subtract this probability from 1 to obtain the probability that a state exists in a particular conversation. These probabilities may be used as inputs to the bot state prediction model 210 to train the bot state prediction model 210.

[0037]

[0047] In one embodiment, training 204 the bot state prediction model 210 may further include computing features of the sample conversations using the feature extractor 208. As described above, the bot state machine learning modeling engine 202 may utilize the feature extractor 208 to extract the conversation length of a particular conversation from the conversation data store 228 and determine a binary feature indicating whether the conversation length is at either extreme of conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may provide a correlation between the conversation length and the probability that a bot state is detected in the sample conversation. Thus, this binary feature may be used to further train the bot state prediction model 210 to more accurately detect bot states and compute a confidence score corresponding to the likelihood that a particular state is present in a given conversation.

[0038]

[0048] Various features may be extracted using the feature extractor 208. For example, the feature extractor 208 may compute a reprompt feature, which may be computed when the conversational bot agent repeats a previous message as a result of the conversational bot agent being unable to understand the customer's intent. Another feature that may be extracted by the feature extractor 208 may include an escalation count feature, which may correspond to the number of times a customer issue escalates during a conversation between a customer and a conversational bot agent. These escalations may serve as a signal that there has been improper routing by the conversational bot agent. As another example, the feature extractor 208 may compute an unplanned escalation feature, which may correspond to any customer request during a conversation with a conversational bot agent that ends up being transferred to a human agent. This may serve as an indication that the conversational bot agent is unable to address the customer's issue, resulting in a frustrating customer experience. The feature extractor 208 can further calculate a number of fallback features, which may correspond to the number of times the conversational bot agent asked the customer to rephrase the message. This may serve as an indication that the conversational bot agent is experiencing difficulties in understanding the customer message, resulting in a negative customer experience. Other features that may be calculated by the feature extractor 208 may include unique intent detection features (e.g., the number of intents in a conversation, which may correlate to the probability that the conversational bot agent misunderstands one of the intents), repeated intent features (e.g., if an intent is repeated, it may serve as an indication that the customer is stuck in a loop with the conversational bot agent or that the conversational bot agent was unable to understand the intent or provide an appropriate response), etc.

[0039]

[0049] Other features that are not indicative of a bot failure may also be extracted using the feature extractor 208. For example, customer identification features may be computed that correspond to prompts provided by the bot to the customer for information that may be used to identify the customer. As another illustrative example, information gathering features may be computed based on prompts provided by the bot to the customer for any new information that may be used to solve a particular problem or intent. The feature extractor 208 may further compute check comprehension features, which may correspond to prompts provided by the bot to confirm information accessed by the bot related to the customer and / or the problem to be addressed, and any response provided by the customer. For example, if the bot prompts the customer to indicate whether the provided information is correct, the customer may provide a response such as "yes," "actually no," etc. This pairing of the bot prompt and the customer response may be used by the feature extractor 208 to compute check comprehension features. Another example of features that may be computed by the feature extractor 208 may include assistance provision features. The assistance provision feature may be calculated based on the bot providing useful information to the customer or otherwise addressing or answering the customer's problem, intent, or other query. The information provided by the bot may be evaluated to determine whether the information is relevant to the customer and / or whether the information is immediately usable by the customer.

[0040]

[0050] In some instances, some features may be applicable once a conversation between a customer and a conversational bot agent has ended (e.g., the computation is not performed in real-time). For example, the feature extractor 208 may compute a repeat contact feature corresponding to the number of times a customer re-engages with a particular brand associated with a customer service call center after a conversation with a conversational bot agent has ended within a given time period. This feature may serve as an indication of different failure modes present within a conversation. As another example, the feature extractor 208 may compute the aforementioned binary feature indicating whether the conversation length is at either extreme of the conversation length (e.g., short vs. long).

[0041]

[0051] The output (e.g., confidence score) generated by the bot state prediction model 210 may be evaluated by an annotator or other entity associated with a customer service call center to determine whether the bot state prediction model 210 is generating an accurate or correct result. In some instances, the confidence score generated by the bot state prediction model 210 may be compared to the probability that the corresponding bot state exists in a particular conversation to determine whether the bot state prediction model 210 is generating an accurate result. For example, the bot state prediction model 210 used to calculate the confidence score for each possible bot state for a particular conversation may be evaluated to determine whether the bot state prediction model 210 correctly identifies any bot states that may exist in a particular conversation based on the bot state probabilities corresponding to the annotator evaluation of the particular conversation from the bot state annotation data 216. Based on this evaluation, the bot state prediction model 210 may be modified to increase the likelihood that the bot state prediction model 210 will generate a desired result.

[0042]

[0052] In parallel with the training 204 of the bot state prediction model 210, the bot state machine learning model engine 202 may perform training 206 of the MACS prediction model 222 for the calculation of the MACS for each sample conversation. For example, as shown in FIG. 2, the bot state machine learning model engine 202 may use the annotated score extractor 226 to obtain the MACS for each sample conversation from the bot state and MACS annotation data 218 generated by the annotator. The MACS for each sample conversation may be provided to the bot state soft score thresholding engine 224 along with the bot state detected by the annotator, which may process the MACS for each sample conversation and the corresponding bot state detected in the sample conversation to identify any correlation between the bot state and the MACS. These correlations, as well as the annotated MACS provided by the annotator and the detected bot state, may be used as inputs to the MACS prediction model 222 for training the MACS prediction model 222.

[0043]

[0053] In one embodiment, training 206 of the MACS prediction model 222 may further include using each sample conversation length to identify any correlation between the conversation length and the MACS for the particular sample conversation. For example, the MACS prediction model 222 may be trained to use the conversation length for a particular conversation as an input to determine the impact of different bot states on the MACS for the conversation according to the conversation length of the conversation. For example, the impact of the bot state may change as the conversation length increases. Thus, the bot state machine learning modeling engine 202 may utilize the conversation length extractor 220 to obtain the conversation length for each sample conversation from the conversation data store 228 and utilize this conversation length as an input to the MACS prediction model 222 in addition to the annotated MACS and detected bot states for the sample conversation to train the MACS prediction model 222.

[0044]

[0054] The output of the MACS prediction model 222 may include a MACS for each sample conversation. The resulting MACS may be evaluated to determine whether the MACS prediction model 222 correctly calculates the MACS for a particular conversation based on the bot state probabilities corresponding to the annotator ratings of the sample conversation and the MACS annotations for the sample conversation obtained from the annotated data 214. Based on this evaluation, the MACS prediction model 222 may be modified to increase the likelihood that the MACS prediction model 222 will produce a desired outcome.

[0045]

[0055] 3 illustrates an illustrative example of an environment 300 in which a bot state machine learning modeling engine 302 performs parallel training of a bot state prediction model 310 and a MACS prediction model 322 using bot state and MACS annotations and features, according to at least one embodiment. The environment 300 may be similar to the environment 200 illustrated in FIG. 2 and described above. For example, the training 304 of the bot state prediction model 310 may be performed using annotated data 314 generated by annotators based on sample conversations between a conversational bot agent and a customer or other entity, and conversation data corresponding to the sample conversations evaluated by the annotators to generate the annotated data 314. The annotated data 214 may include bot state annotation data 316 (e.g., data generated by each annotator indicating the bot states detected in the sample conversations) and bot state and MACS annotation data 318 (e.g., data generated by each annotator indicating the bot states detected in the sample conversations and the corresponding MACS for the sample conversations), as described above.

[0046]

[0056] As described above, training 304 the bot state prediction model 310 may further include computing features of the sample conversation using the feature extractor 308. The feature extractor 308 may determine a binary feature indicating whether the conversation length is at either extreme of the conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may be used to identify a correlation between the probability of a bot state being detected in the sample conversation and the conversation length. The output (e.g., confidence score) generated by the bot state prediction model 310 may be evaluated by an annotator or other entity associated with the customer service call center to determine whether the bot state prediction model 310 is generating accurate or correct results. For example, the bot state prediction model 310 used to compute the confidence score for each possible bot state for a particular conversation may be evaluated to determine whether the bot state prediction model 310 is correctly identifying any bot states that may be present in the particular conversation based on the bot state probabilities corresponding to the annotator evaluation of the particular conversation from the bot state annotation data 316. Based on this evaluation, the bot state prediction model 310 may be modified.

[0047]

[0057] In one embodiment, rather than relying on a conversation length extractor to determine each sample conversation length from the conversation data store 328 as described above with respect to FIG. 2, the training 306 of the MACS predictive model 322 may be performed using features extracted by the feature extractor 308 from the sample conversations stored in the conversation data store 328. For example, the bot state machine learning modeling engine 302 may utilize the feature extractor 308 to extract the conversation length of a particular conversation and determine a binary feature indicating whether the conversation length is at either extreme of conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may increase the correlation between the MACS score determined by the annotator (provided in the bot state and MACS annotation data 318) based on a manual evaluation of the conversation and the MACS score predicted by the MACS predictive model 322. Thus, the binary feature may be used to improve the training of the MACS predictive model 322.

[0048]

[0058] Similar to the training 206 described above with respect to FIG. 2 for the MACS prediction model 222, the training 306 of the MACS prediction model 322 may be performed using an annotated score extractor 326 and a bot state soft score thresholding engine 324. For example, the bot state machine learning modeling engine 302 may use the annotated score extractor 326 to obtain a MACS for each sample conversation from the bot state and MACS annotation data 318 generated by the annotator. The MACS for each sample conversation may be provided to the bot state soft score thresholding engine 324 along with the bot state detected by the annotator, which may process the MACS for each sample conversation and the corresponding bot state detected in the sample conversation to identify any correlations between the bot state and the MACS. These correlations, as well as the annotated MACS provided by the annotator and the detected bot state, may be used as inputs to the MACS prediction model 322 for training the MACS prediction model 322.

[0049]

[0059] FIG. 4 illustrates an illustrative example of an environment 400 in which a bot state machine learning modeling engine 402 performs training of a MACS prediction model 422 using predictions generated by a bot state prediction model 410, according to at least one embodiment. In the environment 400, training 404 of the bot state prediction model 410 is performed according to a process similar to that described above with respect to FIG. 2 and FIG. 3. For example, the bot state prediction model 410 may be trained using a soft score thresholding engine 412 that may process bot state annotation data 416 from annotated data 414 to determine probabilities that correspond to the likelihood that a bot state exists in a sample conversation. The soft score thresholding engine 412 may calculate the probability that a bot state exists for each message in a conversation and the probability that a bot state exists in a particular conversation. These probabilities may be used as inputs to the bot state prediction model 410 to train the bot state prediction model 410.

[0050]

[0060] Training 404 the bot state prediction model 410 may further include computing features of the sample conversations using the feature extractor 408. As described above, the feature extractor 408 may be used to extract the conversation length of a particular conversation from the conversation data store 428 and determine a binary feature indicating whether the conversation length is at either extreme of the conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may provide a correlation between the conversation length and the probability that a bot state is detected in the sample conversation. Thus, this binary feature may be used to further train the bot state prediction model 410 to more accurately detect a bot state and compute a confidence score corresponding to the likelihood that the bot state is present in a given conversation.

[0051]

[0061] The output of the bot state prediction model 410 may be evaluated to determine whether the bot state prediction model 410 is generating accurate results (e.g., confidence scores for different bot states for a given conversation). Similar to the training 204 and 304 described above, the bot state prediction model 410 used to calculate the confidence scores for each possible bot state for a particular conversation may be evaluated to determine whether the bot state prediction model 410 correctly identifies any bot states that may be present in a particular conversation based on the bot state probabilities corresponding to the annotator evaluation of the particular conversation from the bot state annotation data 416. Based on this evaluation, the bot state prediction model 410 may be modified to increase the likelihood that the bot state prediction model 410 will generate a desired result.

[0052]

[0062] In one embodiment, the output of the bot state prediction model 410 is further used to perform training 406 of the MACS prediction model 422. For example, the resulting confidence scores generated by the bot state prediction model 410 may be used as input to the MACS prediction model 422 to generate a MACS for each sample conversation. This may eliminate the need to utilize the bot state soft score thresholding engine described above with respect to Figures 2 and 3 to calculate the confidence scores in parallel with the training 404 of the bot state prediction model 410. Thus, rather than having the output of the annotated score extractor 426 used by the bot state soft score thresholding engine to generate the confidence scores, the output of the annotated score extractor 426 may be provided directly to the MACS prediction model 422 for training of the MACS prediction model 422.

[0053]

[0063] As described above, the bot state machine learning modeling engine 402 may use the annotated score extractor 426 to obtain a MACS for each sample conversation from the bot state and MACS annotation data 418 generated by the annotator. The MACS for each sample conversation may be provided to the MACS prediction model 422 along with the bot state detected by the annotator and may be used to evaluate the performance of the MACS prediction model 422. For example, the predicted MACS generated by the MACS prediction model 422 may be compared to the annotated MACS determined by the annotator and extracted from the bot state and MACS annotation data 418 to determine whether the MACS prediction model 422 is generating accurate results. Based on this evaluation, the MACS prediction model 422 may be modified to increase the likelihood that the MACS prediction model 422 will generate a desired MACS for the sample conversation.

[0054]

[0064] Similar to the training 206 described above with respect to FIG. 2, the training 406 may also rely on the conversation length extractor 420 to obtain a conversation length for each sample conversation for which the MACS is to be determined using the MACS prediction model 422. As described above, the training 406 of the MACS prediction model 422 may further include using each sample conversation length to identify any correlation between the conversation length and the MACS for the particular sample conversation. For example, the MACS prediction model 422 may be trained to use the conversation length for a particular conversation as an input to determine the impact of different bot states on the MACS for the conversation according to the conversation length of the conversation. Thus, the bot state machine learning modeling engine 402 may utilize the conversation length extractor 420 to obtain the conversation length for each sample conversation from the conversation data store 428 and utilize this conversation length as an input to the MACS prediction model 422 in addition to the annotated MACS for the sample conversation and the bot state confidence score to train the MACS prediction model 422.

[0055]

[0065] FIG. 5 illustrates an illustrative example of an environment 500 in which a bot state machine learning modeling engine 502 performs training 506 of a MACS prediction model 522 using predictions and features generated by a bot state prediction model 510, according to at least one embodiment. In the environment 500, training 504 of the bot state prediction model 510 is performed according to a process similar to that described above with respect to FIGS. 2-4. For example, the bot state prediction model 510 may be trained using a soft score thresholding engine 512, which may process bot state annotation data 516 from annotated data 514 to determine probabilities corresponding to the likelihood that a bot state is present in a sample conversation. The soft score thresholding engine 512 may calculate the probability of a bot state being present for each message in a conversation and the probability that a bot state is present in a particular conversation. These probabilities may be used as inputs to the bot state prediction model 510 to train the bot state prediction model 510.

[0056]

[0066] Similar to the training 306 of the MACS prediction model 322 described above with respect to FIG. 3, the training 506 of the MACS prediction model 522 may be performed using features extracted by the feature extractor 508 from sample conversations stored in the conversation data store 528. For example, the bot state machine learning modeling engine 502 may utilize the feature extractor 508 to extract the conversation length of a particular conversation and determine a binary feature indicating whether the conversation length is at either extreme of conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may increase the correlation between the MACS determined by the annotator (provided in the bot state and MACS annotation data 518) based on a manual evaluation of the conversation and the MACS predicted by the MACS prediction model 522. Thus, this binary feature may be used to improve the training of the MACS prediction model 522.

[0057]

[0067] Additionally, similar to the training 406 described above with respect to FIG. 4, the output of the bot state prediction model 510 may be used to perform training 506 of the MACS prediction model 522. The resulting confidence scores generated by the bot state prediction model 510 may be used as input to the MACS prediction model 522 to generate a MACS for each sample conversation. Additionally, the bot state machine learning modeling engine 502 may use an annotated score extractor 526 to obtain the MACS for each sample conversation from the bot state and MACS annotation data 518 and provide these MACS to the MACS prediction model 522. These MACS may be used to evaluate the performance of the MACS prediction model 522 and, based on this evaluation, modify the MACS prediction model 502 to increase the likelihood that the MACS prediction model 522 will generate the desired MACS for the sample conversation.

[0058]

[0068] FIG. 6 illustrates an illustrative example of an environment 600 in which a bot state machine learning modeling engine 602 uses bot state and MACS annotations and features to train a joint bot state and MACS prediction model 630, according to at least one embodiment. In the environment 600, the bot state machine learning modeling engine 602 trains a joint bot state and MACS prediction model 630 that can be used to both identify a bot state in a conversation and provide MACS for a conversational bot agent engaged in a conversation with a customer (after the conversation is over or in real time). Thus, rather than performing parallel training processes for separate bot state and MACS prediction models as illustrated in FIGS. 2-5, the joint bot state and MACS prediction model 630 can be trained through a single process described herein.

[0059]

[0069] Training of the bot state and MACS prediction model 630 may incorporate features similar to those described above with respect to Figures 2-5. For example, the bot state and MACS prediction model 630 may be trained using a soft score thresholding engine 612, which may process the bot state and MACS annotation data 618 from the MACS annotated data 614 to determine probabilities that correspond to the likelihood that a bot state is present in a sample conversation. The soft score thresholding engine 612 may calculate the probability of a bot state being present for each message in a conversation and the probability of a bot state being present in a particular conversation. These probabilities may be used as inputs to the bot state and MACS prediction model 630 to train the bot state and MACS prediction model 630 to predict the bot state of a conversational bot agent during a conversation with a customer.

[0060]

[0070] In addition to processing the bot state and MACS annotation data 618 from the annotated data 614 to generate inputs for training the bot state and MACS prediction model 630, the bot state machine learning modeling engine 602 may utilize a feature extractor 608 to compute a set of features from the conversation data stored in the conversation data store 628 that may be used to further train the bot state and MACS prediction model 630. As described above, the feature extractor 608 may be used to extract attributes of a particular conversation from the conversation data store 628 and determine features that may provide a correlation between the probability that a bot state is detected in a sample conversation and the attributes of the conversation. Thus, these features may be used to further train the bot state and MACS prediction model 630 to more accurately detect a bot state and compute a confidence score that corresponds to the likelihood that the bot state is present in a given conversation.

[0061]

[0071] Additionally, the features may be used to train the bot state and MACS prediction model 630 to predict the MACS for a conversational bot agent for a particular conversation. For example, the bot state and MACS prediction model 630 may utilize the features calculated by the feature extractor 508 and the bot state confidence scores generated by the bot state and MACS prediction model 630 to predict the MACS for a conversation. The features may increase the correlation between the MACS determined by the annotator (provided in the bot state and MACS annotation data 618) based on a manual evaluation of the conversation and the MACS predicted by the bot state and MACS prediction model 630. Thus, these features may be used to improve the training of the bot state and MACS prediction model 630.

[0062]

[0072] FIG. 7 illustrates an illustrative example of an environment 700 in which a MACS predictive model 710 provides a MACS for one or more bot agents 714 based on conversation data and identified bot states for the one or more bot agents 714, according to at least one embodiment. In the environment 700, conversations between a conversational bot agent 714 and customers and other users of a customer service call center may be recorded in a conversation data store 712. A conversation stored in the conversation data store 712 may include any messages communicated between a particular conversational bot agent 714 and a customer or other entity via a communication session established between the conversational bot agent 714 and the customer or other entity's computing device. For example, a customer or other user may use a computing device to initiate a conversation with a conversational bot agent 714 regarding the resolution of a problem. The conversational bot agent 714 may automatically identify the intent of the customer or other user and important user information (e.g., order number, account number, etc.) that may be used to address the intent. As described above, the conversational bot agent 714 can be configured to communicate autonomously with customers and other users and for specific capabilities.

[0063]

[0073] In one embodiment, conversation data for a particular conversation between the conversational bot agent 714 and a customer or other user over a communication session is used as input to the feature extractor 704 of the bot state machine learning modeling engine 702. For example, conversation data from the conversation data store 712 may be automatically streamed to the feature extractor 704 to compute a set of features corresponding to a conversation between the conversational bot agent 714 and a customer or other user. As described above, the bot state machine learning modeling engine 702 may utilize the feature extractor 704 to compute a conversation length for a particular conversation and determine a binary feature indicating whether the conversation length is at either extreme of conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may increase the correlation between the MACS for the conversation determined by the annotator based on a manual evaluation of the conversation and the MACS predicted by the MACS prediction model 710 during training of the MACS prediction model 710. In some implementations, the feature extractor 704 may also be used to extract features that may be used by the bot state prediction model 706 to more accurately compute confidence scores for different possible bot states that may be detected within the conversation.

[0064]

[0074] In one embodiment, the features calculated by the feature extractor 704 as well as the conversation data corresponding to the particular calculations may be used as input to the bot state prediction model 706 to generate confidence scores for different bot states that may be present in the conversation. The confidence scores may correspond to the probability that a particular bot state is present in the conversation. In some instances, the bot state prediction model 706 applies a threshold to the confidence scores such that if the confidence score for a particular bot state is above the threshold, the particular bot state is indicated as being present in the conversation. Alternatively, if the confidence score for a particular bot state is equal to or less than the threshold, the particular bot state is indicated as not being in the conversation. Thus, the bot state prediction model 706 may provide to the MACS prediction model 710 either confidence scores corresponding to the possible presence of bot states in the conversation or binary indications corresponding to the presence of these bot states.

[0065]

[0075] In one embodiment, the bot state machine learning modeling engine 702 can provide confidence scores corresponding to the possible presence of bot states in a conversation, or binary indications corresponding to the presence of these bot states, to the bot builder 718 without processing these scores or binary indications through the MACS predictive model 710. For example, the bot state machine learning modeling engine 702 can provide confidence scores or binary indications corresponding to the presence of different bot states in a conversation involving a conversational bot agent 714 implemented by the bot builder 718 via an interface or portal accessible via a computing device 716 provided by a customer service call center and utilized by the bot builder 718. Based on the confidence scores or binary indications, the bot builder 718 can update the configuration of the conversational bot agent 714 to potentially improve the performance of the conversational bot agent 714 (e.g., reduce the likelihood that a particular bot state is present, etc.). For example, if certain bot states exist for a particular class of conversational bot agents 714 (e.g., having a confidence score above a threshold, being shown as being presented through a binary display, etc.) and the particular bot state is undesirable (e.g., the bot asks for outdated information, the bot misunderstands the customer, the bot is unable to assist the customer, the bot is unable to transfer the conversation to a live agent, etc.), the bot builder 718 may generate updates to the conversational bot agents 714 to reduce the likelihood of these bot states occurring for these conversational bot agents 714.

[0066]

[0076] In one embodiment, in addition to providing the bot builder 718 with a confidence score or a binary bot state indication for a particular conversation, the MACS prediction model 710 can process the output of the bot state prediction model 706 (e.g., the confidence score or the binary bot state indication) and a metric corresponding to the conversation length to generate a MACS for the conversation. In some cases, if the MACS prediction model 710 is configured to further utilize features to calculate the MACS for the conversation, the MACS prediction model 710 can further obtain these features from the feature extractor 704 or the bot state prediction model 706. These features can correspond to different characteristics of the conversation, such as the conversation length, instances of previous message repetition by the conversational bot agent 714, the number of times the customer re-engaged with the brand after the conversation ended, the number of times the customer's issue was escalated, the number of customer requests to transfer the conversation to a human agent, etc.

[0067]

[0077] The output of the MACS prediction model 710 may include an overall MACS for a particular conversation and a MACS for each message in the conversation. As discussed above, the MACS may correspond to the quality of the conversation and the performance of the conversational bot agent 714 for a particular conversation. For example, a MACS of "1" may correspond to a bad conversation in which the conversational bot agent 714 was not helpful in addressing a particular problem of the customer. Alternatively, a MACS of "5" may indicate that the conversational bot agent 714 was helpful (e.g., providing relevant actions or information related to the customer's intent) and that the conversation was uneventful, resulting in a positive customer experience. It should be noted that the range of possible MACS values ​​may be determined by the customer service call center to indicate possible performance metrics for the conversational bot agent 714 with respect to a particular conversation with a customer. Thus, other MACS ranges may be used in addition to or as an alternative to the aforementioned "1-5" range.

[0068]

[0078] In one embodiment, the MACS for a particular conversation and corresponding messages, as well as any identified bot states and corresponding confidence scores / binary indications, may be provided to the bot builder 718 to provide an indication of the performance of the conversational bot agent 714. For example, the bot state machine learning modeling engine 702 may provide the MACS and bot states (along with corresponding confidence scores and / or binary indications) for different conversations involving the conversational bot agent 714 implemented by the bot builder 718 via an interface or portal accessible via the computing device 716 provided by the customer service call center and utilized by the bot builder 718. Based on the resulting MACS, the identified bot states, and the corresponding confidence scores and / or binary indications for these identified bot states, the bot builder 718 may update the configuration of the conversational bot agent 714 to potentially improve the performance of the conversational bot agent 714 (e.g., MACS, reduction in the likelihood of unfavorable bot states, etc.). For example, if the MACS for a particular class of conversational bot agents 714 is consistently below a threshold (e.g., the conversational bot agents 714 are performing poorly) as a result of encountering a particular set of bot states, the bot builder 718 may generate updates to the conversational bot agents 714 to reduce the likelihood of these bot states occurring and improve the MACS for these conversational bot agents 714.

[0069]

[0079] In one embodiment, the MACS for a particular conversation, as well as the confidence scores and / or binary indications for different bot states, may be dynamically updated in real-time as new messages from a conversation are processed by the bot state prediction model 706 and the MACS prediction model 710. For example, rather than processing conversation data from the conversation data store 712 corresponding to completed or terminated conversations, the bot state machine learning modeling engine 702 may process ongoing conversations. As an illustrative example, the bot state machine learning modeling engine 702 may monitor ongoing conversations from the conversation data store 712 to detect any new messages added to the ongoing conversation. If the bot state machine learning modeling engine 702 detects one or more new messages in the ongoing conversation, the bot state machine learning modeling engine 702 may process the ongoing conversation, including the one or more new messages, using the bot state prediction model 706 and the MACS prediction model 710 to identify any new bot states (if any) and new MACS for the ongoing conversation. The new bot state and updated MACS score can be provided to the Bot Builder 718 in real time to enable the Bot Builder 718 to make modifications or implement updates to the conversational bot agent 714 on the fly, thereby enabling enhancements to the conversational bot agent 714 during a conversation.

[0070]

[0080] In one embodiment, feedback from the bot builder 718 regarding the provided bot states and MACS for a conversation in which the conversational bot agent 714 engaged a customer or other user may be used to further train the bot state prediction model 706 and the MACS prediction model 710. For example, the bot builder 718 may evaluate a particular conversation and the indicated bot state and MACS for that conversation to determine whether the indicated bot state and MACS are accurate. For example, if the bot builder 718 determines that a bot state was misidentified in a conversation, the bot builder 718 may provide an indication to the bot state machine learning modeling engine 702 that the bot state was misidentified by the bot state prediction model 706 for the conversation. In response, the bot state machine learning modeling engine 702 may use this feedback to further train the bot state prediction model 706 and the MACS prediction model 710 to improve the accuracy of these models in identifying bot states and calculating MACS for different conversations, respectively.

[0071]

[0081] 8 illustrates an illustrative example of an environment 800 in which a bot state machine learning modeling engine 802 performs real-time analysis of messages 806-1 through 806-N of an active communication session 804 to dynamically determine a bot state and MACS for a conversational bot agent 814, according to at least one embodiment. In the environment 800, a customer 808 may be engaged in a communication session with a conversational bot agent 814 at a customer service call center via a computing device 810. Throughout the communication session 804, the customer 808 and the conversational bot agent 814 may exchange various messages 806-1 through 806-N over time.

[0072]

[0082] In one embodiment, a customer service call center implementing conversational bot agent 814 and other conversational bot agents to autonomously communicate with customers adds any newly received messages for communication session 804 into a bus or cache. The customer service call center may implement one or more services configured to monitor the bus or cache for any new messages corresponding to communication session 804 and retrieves these new messages for processing. As shown in FIG. 8, messages 806-1 and 806-2 are newly received and retrieved by these one or more services for feature extraction as described above. These messages 806-1 and 806-2 and the extracted features are provided to the bot state machine learning modeling engine 802 for processing. As described above, the bot state machine learning modeling engine 802 may be configured to predict the bot state and MACS for a given conversation through the use of separate models or jointly through the use of a joint bot state and MACS prediction model as shown in FIG. 6.

[0073]

[0083] In one embodiment, in addition to using the newly received messages 806-1 and 806-2 and features extracted from these messages 806-1 and 806-2, the bot state machine learning modeling engine 802 retrieves conversation data corresponding to the communication session 804 from the conversation data store 812. This conversation data may include context regarding the communication session 804 that may be used by the bot state machine learning modeling engine 802 to evaluate the newly received messages 806-1 and 806-2 in their appropriate context. For example, the bot state machine learning modeling engine 802 may retrieve from the conversation data store 812 any messages previously exchanged between the customer 808 and the conversational bot agent 814 over the duration of the communication session 804. Alternatively, the bot state machine learning modeling engine 802 may retrieve metadata about the communication session 804 that may specify attributes of the communication session 804 that may be used to determine the context of the communication session 804. For example, the metadata may specify a title or label for the communication session 804 (e.g., "Customer Billing Issue," "Customer Refund Request," etc.) and any data corresponding to the current performance of the conversational bot agent 814. This additional data may include any previously determined bot state and / or MACS for the communication session 804 based on previously exchanged messages.

[0074]

[0084] The bot state machine learning modeling engine 802 may use the bot state and MACS prediction model to process the newly received messages 806-1 and 806-2 and features extracted from these messages and the context of the communication session 804 from the conversation data store 812. As described above, the bot state and MACS prediction model is trained to detect one or more bot states from a conversation and calculate a MACS for the conversation based on the detected one or more bot states. The bot state and MACS prediction model may be trained using features extracted by a feature extractor from a sample conversation. For example, the bot state machine learning modeling engine 802 may utilize a feature extractor to extract features of a particular conversation. These features may increase the correlation between the MACS determined by the annotator based on a manual evaluation of the conversation (provided in the bot state and MACS annotation data provided by the annotator) and the MACS predicted by the bot state and MACS prediction model. Further, the bot state and MACS prediction models may be trained using a soft score thresholding engine that may process the bot state and MACS annotation data for the sample conversation to determine probabilities that correspond to the likelihood that a bot state is present in the sample conversation. The soft score thresholding engine may calculate the probability that a bot state is present for each message in the conversation and the probability that a bot state is present in a particular conversation. These probabilities may be used as inputs to the models to train the bot state and MACS prediction models.

[0075]

[0085] In one embodiment, the bot state machine learning modeling engine 802 stores the processed messages 806-1 and 806-2 and any previously processed messages of the communication session 804 and the determined bot state and MACS for the communication session 804 in a bot state and MACS data store 816. This data store 816 may be implemented as a message bus, whereby subscribers to the message bus (e.g., bot builders, etc.) may evaluate the performance of the conversational bot agent 814 in real time. For example, as newly received messages 806-1 and 806-2 are processed to identify any new bot state and update the MACS for the communication session 804, subscribers to the bot state and MACS data store 816 may be presented with the updated bot state and MACS for the communication session 804 in real time. This data may also be stored in the conversation data store 812 to provide additional context to the communication session 804 when processing new messages (e.g., messages 806-3 through 806-N, etc., received after messages 806-1 and 806-2). Thus, as new messages are exchanged during the communication session 804, the bot state machine learning modeling engine 802 may dynamically update the MACS for the communication session 804 and identify any new bot states in real time. For example, as new messages (e.g., messages 806-3 through 806-N, etc.) are received, the bot state machine learning modeling 802 may process these new messages dynamically and in real time. The bot state machine learning modeling engine 802 may retrieve previously calculated MACS and previously identified bot states for the communication session 804 from the conversation data store 812.In some instances, the previously calculated MACS and previously identified bot states for the communication session 804 may be kept in memory such that as new messages are exchanged over the communication session 804, the bot state machine learning modeling engine 802 may obtain these new messages in real time and process these new messages to perform a preliminary identification of the bot state and MACS for the communication session 804. The bot state machine learning modeling engine 802 may use this preliminary identification of the bot state and MACS along with the previously obtained bot state and MACS to dynamically update the MACS for the communication session 804 and identify any new bot states.

[0076]

[0086] 9 illustrates an illustrative example of a process 900 for training a bot state prediction model using conversation data and calculated probabilities for bot states corresponding to previously analyzed bot conversations, according to at least one embodiment. The process 900 may be performed by the bot state machine learning modeling engine described above. For example, the bot state machine learning modeling engine may utilize a feature extractor and a soft score thresholding engine to process sample conversations and annotations created for these sample conversations to train a bot state prediction model.

[0077]

[0087] In step 902, the bot state machine learning modeling engine may identify previously analyzed conversations performed by the conversational bot agent (e.g., conversations in which the conversational bot agent engaged with a customer or other entity via a communication session) for training the bot state prediction model. For example, the bot state machine learning modeling engine may identify a set of sample conversations annotated by a set of annotators for training the bot state prediction model and the MACS prediction model. As described above, the particular conversation may be provided to a set of annotators to identify any bot states occurring during the particular conversation. These bot states may include the bot being unable to understand the customer's message (e.g., intent, problem, etc.), the bot ignoring the customer, the bot being stuck in a message loop, the bot detecting customer frustration, the bot incorrectly forwarding to a human agent, the bot performing information gathering to resolve the intent, the bot successfully providing assistance to the customer, the bot forwarding the conversation to a live agent or other bot that is better able to handle the intent, the bot checking understanding of the customer response, etc. In one embodiment, each annotator may evaluate each message of a particular conversation and indicate whether a bot state was detected for that message. For example, an annotator may annotate each message of a particular conversation to indicate whether the message exhibits a particular bot state.

[0078]

[0088] In addition to identifying any previously analyzed conversations for training the bot state prediction model, the bot state machine learning modeling engine may compute a set of features of these previously analyzed conversations in step 904. For example, the bot state machine learning modeling engine may utilize a feature extractor to extract the conversation length of the previously analyzed conversations and determine a binary feature indicating whether the conversation length is at either extreme of the conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may provide a correlation between the conversation length and the probability that a bot state is detected in a sample conversation. Thus, this binary feature may be used to further train the bot state prediction model to more accurately detect a bot state and compute a confidence score corresponding to the likelihood that the bot state is present in a given conversation.

[0079]

[0089] In step 906, the bot state machine learning modeling engine may obtain bot state annotations for previously analyzed conversations performed by the conversational bot agent. As described above, each sample conversation to be used to train the bot state prediction model may be annotated by a set of annotators to indicate the presence (or absence) of a particular bot state in the sample conversation. For example, an annotator may evaluate each message of the conversation and indicate whether a bot state was detected by the annotator as well as specify which bot state was detected. Thus, from a previously analyzed conversation to be used to train the bot state prediction model, the bot state machine learning modeling engine may identify annotations provided by a set of annotators for a particular conversation and, based on these annotations, identify the bot states detected by each of the annotators.

[0080]

[0090] In step 908, based on the annotations provided by the set of annotators for the previously analyzed conversation, the bot state machine learning modeling engine may calculate probabilities corresponding to the presence of different bot states in the previously analyzed conversation. For example, the bot state machine learning modeling engine may implement an aggregation strategy to determine the probability that any message in the conversation has a particular bot state. For example, the bot state machine learning modeling engine may calculate the probability that a bot state exists for each message in the conversation. As an illustrative example, if a particular message is evaluated by three annotators and one of these annotators determines that a particular bot state exists for a particular message, the bot state machine learning modeling engine may determine that the probability that a bot state exists in the message is the number of annotations indicating the bot state divided by the total number of annotations, i.e., 1 / 3 in this case. The bot state machine learning modeling engine may subtract this probability from 1 to obtain the probability that a particular message of the transformation does not include this bot state. The bot state machine learning modeling engine may multiply the obtained probabilities for each message in the particular conversation to obtain the probability that a particular bot state does not exist in the particular conversation. The bot state machine learning modeling engine may subtract this probability from 1 to obtain the probability that the bot state is present in a particular conversation. The resulting annotated conversation and corresponding bot state probabilities may be used as a "ground truth" or sample data set for training a classification model that can be used to dynamically detect bot states from conversations in real time.

[0081]

[0091] In step 910, the bot state machine learning modeling engine may use the aforementioned probabilities and conversation data corresponding to the previously analyzed conversation (e.g., messages, conversation length, features, etc.) to train a bot state prediction model. For example, the bot state machine learning modeling engine may use the aforementioned probabilities and conversation data corresponding to the previously analyzed conversation as inputs to the bot state prediction model. The bot state prediction model may generate as output a confidence score corresponding to the likelihood that different bot states exist in the sample conversation.

[0082]

[0092] The output (e.g., confidence score) generated by the bot state prediction model may be evaluated by an annotator or other entity to determine whether the bot state prediction model is generating an accurate or correct result. In some cases, the confidence score generated by the bot state prediction model may be compared to the probability that the corresponding bot state exists in a particular conversation to determine whether the bot state prediction model is generating an accurate result. For example, the bot state prediction model used to calculate the confidence score for each possible bot state for a particular conversation may be evaluated to determine whether the bot state prediction model correctly identifies any bot states that may exist in a particular conversation based on the bot state probabilities corresponding to the annotator evaluation of the particular conversation from the conversation data. Based on this evaluation, the bot state prediction model may be modified to increase the likelihood that the bot state prediction model will generate a desired result.

[0083]

[0093] Once the bot state prediction model has been trained and evaluated, the bot state machine learning modeling engine may repeat the process 900 using any newly acquired conversations and feedback from the bot builder and other entities based on their evaluation of these conversations. This allows for continuous training of the bot state prediction model to ensure that it is producing accurate results for each conversation in real time.

[0084]

[0094] 10 illustrates an illustrative example of a process 1000 for training a MACS predictive model based on annotated conversation data and identified bot states, according to at least one embodiment. The process 1000 may be performed by a bot state machine learning modeling engine that may use the bot state and MACS annotation data (e.g., data generated by each annotator indicating the bot states detected in the sample conversation and the corresponding MACS for the sample conversation) and attributes of the sample conversation (e.g., conversation length, etc.) for training the MACS predictive model. As described in more detail herein, the bot state machine learning modeling engine may also utilize a set of features calculated from the sample conversation and a bot state confidence score generated by the bot state predictive model for training the MACS predictive model.

[0085]

[0095] In step 1002, the bot state machine learning modeling engine may identify previously analyzed conversations performed by the conversational bot agent (e.g., conversations in which the conversational bot agent engaged with a customer or other entity via a communication session) for training the bot state prediction model. For example, the bot state machine learning modeling engine may identify a set of sample conversations annotated by a set of annotators for training the bot state prediction model and the MACS prediction model. As described above, a particular conversation may be provided to a set of annotators to identify any bot states occurring during the particular conversation and the MACS for the particular conversation. Furthermore, when a bot state is detected, the annotators may decrement the MACS for the particular conversation until a final MACS for the particular conversation is determined. Thus, a particular sample conversation may be annotated with an incremental MACS that is updated when a bot state is detected within the flow of the particular conversation and a final MACS for the complete conversation.

[0086]

[0096] In step 1004, the bot state machine learning modeling engine may determine whether to compute a set of features for the previously analyzed conversation for training the MACS prediction model. As described above, the bot state machine learning modeling engine may utilize a feature extractor to extract the conversation length of a particular conversation and determine a binary feature indicating whether the conversation length is at either extreme of the conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may increase the correlation between the MACS score determined by the annotator based on the manual evaluation of the conversation and the MACS score predicted by the MACS prediction model. Thus, this binary feature may be used to further improve the training of the MACS prediction model. However, in some cases, the MACS prediction model may be trained without features, whereby characteristics of a particular conversation may be extracted and directly used to train the MACS prediction model.

[0087]

[0097] If the bot state machine learning modeling engine is configured to compute features for use in training the MACS prediction, the bot state machine learning modeling engine may utilize a feature extractor to compute features of one or more sample conversations in step 1006. Alternatively, if the bot state machine learning modeling engine is configured to train the MACS prediction engine without computing features of the sample conversations, the bot state machine learning modeling engine may extract conversation lengths of the sample conversations using a conversation length extractor in step 1008. The bot state machine learning modeling engine may use these conversation lengths as inputs to the MACS prediction model, in addition to the annotated MACS and detected bot states for the sample conversations, to train the MACS prediction model, as described herein.

[0088]

[0098] In step 1010, the bot state machine learning modeling engine may extract annotated scores (e.g., MACS) from the annotated conversation data for one or more previously analyzed conversations. As described above, the bot state machine learning modeling engine may obtain bot state and MACS annotation data (e.g., data generated by each annotator indicating the bot state detected in the sample conversation and the corresponding MACS for the sample conversation) from the annotated data generated by one or more annotators for the set of sample conversations. The bot state machine learning modeling engine may utilize an annotated score extractor to obtain the MACS for each sample conversation from the bot state and MACS annotation data generated by the annotators.

[0089]

[0099] In step 1012, the bot state machine learning modeling engine may determine whether to use the bot state confidence score generated by the bot state prediction model as an input to the MACS prediction model for training the MACS prediction model. As described above, the training of the MACS prediction model and the bot state prediction model may be performed in parallel, whereby the MACS prediction model does not rely on the output from the bot state prediction model to train. However, in some cases, rather than performing the training of the MACS prediction model in parallel with the training of the bot state prediction model, the MACS prediction model may be trained using the confidence score (or the bot state detected based on the confidence score) as an input.

[0090]

[0100] If the bot state machine learning modeling engine is configured to perform training of the MACS prediction model in parallel with training of the bot state prediction model, the bot state machine learning modeling engine may perform soft score thresholding on the annotated bot states in the previously analyzed conversations in step 1014. For example, the MACS for each sample conversation may be provided to the bot state soft score thresholding engine along with the bot states detected by the annotator, and the bot state soft score thresholding engine may process the MACS for each sample conversation and the corresponding bot states detected in the sample conversation to identify any correlations between the bot states and the MACS. These correlations, as well as the annotated MACS provided by the annotator and the detected bot states, may be used as inputs to the MACS prediction model for training the MACS prediction model.

[0091]

[0101] If the bot state machine learning modeling engine is configured to utilize the confidence score from the bot state predictive model for training the MACS predictive model, the bot state machine learning modeling engine may obtain the confidence score from the bot state predictive model for the previously analyzed conversation in step 1016. In one embodiment, the bot state machine learning modeling engine may use the confidence score for each bot state obtained using the bot state predictive model for a particular conversation as an input to the MACS predictive model to obtain the MACS for the particular conversation. Alternatively, the bot state machine learning modeling engine may determine whether the confidence score for each bot state meets a threshold. If the confidence score for a particular bot state is above the threshold, the bot state machine learning modeling engine may determine that the corresponding bot state exists in the conversation. The presence of the bot state in the conversation may be used as an input to the MACS predictive model according to the threshold to calculate the MACS for the conversation.

[0092]

[0102] In step 1018, the bot state machine learning modeling engine may use the aforementioned data (e.g., conversation length, feature values, annotated MACS, annotated bot state, confidence scores, correlations generated using soft score thresholding, etc.) as inputs to train a MACS prediction model for evaluation of the performance of the MACS prediction model. As described above, the output of the MACS prediction model may include a MACS for each sample conversation. The resulting MACS may be evaluated to determine whether the MACS prediction model correctly calculates the MACS for a particular conversation based on the bot state probabilities corresponding to the annotator evaluations of the sample conversations and the MACS annotations for the sample conversations obtained from the annotated data. Based on this evaluation, the MACS prediction model may be modified to increase the likelihood that the MACS prediction model will generate a desired outcome.

[0093]

[0103] Similar to the continuous training of the bot state prediction model described above with respect to FIG. 9, the training of the MACS may be performed continuously to improve the accuracy of the MACS prediction model. For example, the bot state machine learning modeling engine may repeat the process 1000 using any newly acquired conversations and feedback from the bot builder and other entities based on their evaluation of these conversations. This allows for the continuous training of the MACS prediction model to ensure that it is producing accurate results for each conversation in real time.

[0094]

[0104] FIG. 11 illustrates an illustrative example of a process 1100 for generating and providing a MACS and any bot state for a conversation between a bot agent and a customer, according to at least one embodiment. The process 1100 may be performed by a customer service call center or other entity implementing a bot state prediction model and a MACS prediction model for detecting a bot state and calculating a MACS for a conversation between a conversational bot agent and a customer, respectively. In step 1102, the customer service call center or other entity may obtain conversation data for a particular conversation between a customer and a conversational bot agent. For example, a conversation between a conversational bot agent and a customer and other users of the customer service call center may be recorded. The recorded conversation may include any messages communicated between a particular conversational bot agent and a customer or other entity via a communication session established between the conversational bot agent and a computing device of the customer or other entity.

[0095]

[0105] In step 1104, the customer service call center or other entity may compute a set of conversation features from the conversation data. For example, conversation data for a particular conversation between a conversational bot agent and a customer or other user over a communication session may be used as input to a feature extractor. As described above, the feature extractor may be used to compute the conversation length of a particular conversation and determine a binary feature indicating whether the conversation length is at either extreme of conversation length (e.g., short vs. long). The binary feature corresponding to the conversation length may increase the correlation between the MACS for the conversation determined by the annotator based on a manual evaluation of the conversation and the MACS predicted by the MACS prediction model during training of the MACS prediction model. In some implementations, the feature extractor may also be used to extract features that may be used by the bot state prediction model to more accurately calculate confidence scores for different possible bot states that may be detected within the conversation.

[0096]

[0106] In step 1106, the customer service call center or other entity may use a bot state prediction model to process the set of features and the conversation data corresponding to the particular conversation. As described above, the features calculated by the feature extractor as well as the conversation data corresponding to the particular calculation may be used as input to the bot state prediction model to generate confidence scores for different bot states that may be present in the conversation. In some cases, the bot state prediction model may apply a threshold to the confidence score such that if the confidence score for a particular bot state is above the threshold, the particular bot state is indicated as being present in the conversation. Alternatively, if the confidence score for a particular bot state is equal to or less than the threshold, the particular bot state is indicated as not being in the conversation. Thus, the bot state prediction model may provide the MACS prediction model with either a confidence score corresponding to the possible presence of the bot state in the conversation or a binary indication corresponding to the presence of these bot states.

[0097]

[0107] In step 1108, the customer service call center or other entity may extract a conversation length from the conversation data. The customer service call center or other entity may use a conversation length extractor to determine the conversation length for which the MACS is being determined. The conversation length may be determined as the number of messages exchanged between the conversational bot agent and the customer or other user during a communication session between the conversational bot agent and the customer or other user. Alternatively, the conversation length may be defined as the number of messages sent by the conversational bot agent during a communication session with the customer or other user.

[0098]

[0108] In step 1110, the customer service call center or other entity may determine whether to use the set of features as inputs to a MACS prediction model to obtain a MACS for the conversation. As described above, the MACS prediction model may be configured to process the output of the bot state prediction model (e.g., a confidence score or a binary bot state indication) and a metric corresponding to the conversation length to generate a MACS for the conversation. In some cases, if the MACS prediction model is configured to further utilize the features to calculate a MACS for the conversation, the customer service call center or other entity may use the MACS prediction model to process these features, the output of the bot state prediction model, the conversation length, and the conversation data itself in step 1114. Alternatively, if the MACS prediction model is configured to utilize the output of the bot state prediction model and the conversation length as inputs, the customer service call center or other entity may use the MACS prediction model to process them in step 1112.

[0099]

[0109] In step 1116, the customer service call center or other entity provides the resulting MACS and the detected bot states for the conversation. For example, the customer service call center or other entity may provide the MACS and bot states for the conversation involving the conversational bot agent implemented by the bot builder via an interface or portal provided by the customer service call center and accessible via a computing device utilized by the bot builder. Based on the resulting MACS and the identified bot states, the bot builder may update the configuration of the conversational bot agent to potentially improve the performance (e.g., MACS, etc.) of the conversational bot agent. For example, if the MACS for a particular class of conversational bot agents is consistently below a threshold (e.g., the conversational bot agent is performing poorly) as a result of encountering a particular set of bot states, the bot builder may generate updates to the conversational bot agent to reduce the likelihood of these bot states occurring and improve the MACS for these conversational bot agents.

[0100]

[0110] FIG. 12 illustrates a computing system architecture 1200 including various components in electrical communication with each other using a connection 1206, such as a bus, according to some implementations. The exemplary system architecture 1200 includes a processing unit (CPU or processor) 1204 and system connections 1206 coupling various system components, including a system memory 1220, such as a ROM 1218 and a RAM 1216, to the processor 1204. The system architecture 1200 may include a cache 1202 of high-speed memory connected directly to the processor 1204, in close proximity to the processor 1204, or integrated as part of the processor 1204. The system architecture 1200 may copy data from the memory 1220 and / or storage device 1208 to the cache 1202 for quick access by the processor 1204. In this manner, the cache may provide performance improvements that avoid delays of the processor 1204 while waiting for data. These and other modules may control or be configured to control the processor 1204 to perform various actions.

[0101]

[0111] Other system memory 1220 may also be available for use. Memory 1220 may include multiple different types of memory having different performance characteristics. Processor 1204 may include any general purpose processor and hardware or software services, such as service 1 1210, service 2 1212, and service 3 1214 stored in storage device 1208, configured to control processor 1204 as well as special purpose processors, where software instructions are incorporated into the actual processor design. Processor 1204 may be a completely self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0102]

[0112] To enable user interaction with the computing system architecture 1200, the input devices 1222 can represent any number of input mechanisms, such as a microphone for audio, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output devices 1224 can also be one or more of several output mechanisms known to those skilled in the art. In some cases, a multimodal system can enable a user to provide multiple types of input to communicate with the computing system architecture 1200. The communication interface 1226 can generally govern and manage user input and system output. There is no restriction to operating on any particular hardware configuration, and thus the basic features herein can be easily substituted with improved hardware or firmware configurations as they are developed.

[0103]

[0113] The storage device 1208 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing data that is accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, RAM 1216, ROM 1218, and hybrids thereof.

[0104]

[0114] The storage device 1208 can include services 1210, 1212, 1214 for controlling the processor 1204. Other hardware or software modules are contemplated. The storage device 1208 can be connected to a system connection 1206. In one aspect, a hardware module that performs a particular function can include software components stored in a computer readable medium in relation to the necessary hardware components, such as the processor 1204, the connection 1206, the output device 1224, etc., to perform that function.

[0105]

[0115] The disclosed methods may be performed using a computing system. An exemplary computing system may include a processor (e.g., a central processing unit), a memory, a non-volatile memory, and an interface device. The memory may store data and / or one or more sets of code, software, scripts, etc. The components of the computer system may be coupled to each other via a bus or through some other known or convenient device. The processor may be configured, for example, to perform all or part of the methods described herein by executing code stored in the memory, for example. One or more of the user device or computer, the provider server or system, or the suspended database update system may include components of a computing system or variations of such systems.

[0106]

[0116] The present disclosure contemplates a computer system taking any suitable physical form, including, but not limited to, a point-of-sale system ("POS"). By way of example and not limitation, the computer system may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, or a combination of two or more of these. Where appropriate, the computer system may include one or more computer systems that are unitary or distributed, across multiple locations, across multiple machines, and / or reside in a cloud that may include one or more cloud components in one or more networks. Where appropriate, the one or more computer systems may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, the one or more computer systems may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. One or more computer systems may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations, where appropriate.

[0107]

[0117] The processor may be, for example, a conventional microprocessor such as an Intel Pentium microprocessor or a Motorola Power PC microprocessor. Those skilled in the art will recognize that the terms "machine-readable (storage) medium" or "computer-readable (storage) medium" include any type of device that is accessible by a processor.

[0108]

[0118] The memory may be coupled to the processor, for example, by a bus. The memory may include random access memory (RAM), such as, by way of example and not limitation, dynamic RAM (DRAM) and static RAM (SRAM). The memory may be local, remote, or distributed.

[0109]

[0119] The bus may also couple the processor to non-volatile memory and drive units. The non-volatile memory is often a magnetic floppy disk or hard disk, a magneto-optical disk, an optical disk, a CD-ROM, a read-only memory (ROM) such as an EPROM or EEPROM, a magnetic or optical card, or another form of storage for large amounts of data. Part of this data is often written to the memory during the execution of software in the computer by a direct memory access process. The non-volatile storage may be local, remote, or distributed. The non-volatile memory is optional, since the system may be created with all applicable data available in the memory. A typical computer system will usually include at least a processor, a memory, and a device (e.g., a bus) that couples the memory to the processor.

[0110]

[0120] The software may be stored in a non-volatile memory and / or a drive unit. In fact, for large programs, it may not even be possible to store the entire program in memory. Nevertheless, it should be understood that in order for the software to run, it is moved, when necessary, to a computer-readable location suitable for processing, which for purposes of explanation will be referred to herein as memory. Even when the software is moved to memory for execution, the processor may utilize hardware registers to store values ​​associated with the software, and ideally a local cache that serves to speed up execution. As used herein, a software program is assumed to be stored in any known or convenient location (from non-volatile storage to hardware registers) when the software program is referred to as being "implemented in a computer-readable medium." A processor is considered to be "configured to execute a program" when at least one value associated with the program is stored in a register readable by the processor.

[0111]

[0121] The bus may also couple the processor to a network interface device. The interface may include one or more of a modem or a network interface. It will be appreciated that the modem or network interface may be considered to be part of the computer system. The interface may include an analog modem, an integrated services digital network (ISDN modem, cable modem, a token ring interface, a satellite transmission interface (e.g., "Direct PC"), or other interface for coupling the computer system to other computer systems. The interface may include one or more input and / or output (I / O) devices. The I / O devices may include, by way of example and not limitation, a keyboard, a mouse or other pointing device, a disk drive, a printer, a scanner, and other input and / or output devices including a display device. The display device may include, by way of example and not limitation, a cathode ray tube (CRT), a liquid crystal display (LCD), or any other applicable known or convenient display device.

[0112]

[0122] In operation, a computer system may be controlled by operating system software, including a file management system, such as a disk operating system. One example of operating system software with its associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Washington, and their associated file management systems. Another example of operating system software with its associated file management system software is the Linux® operating system and its associated file management system. The file management system may be stored in non-volatile memory and / or the drive unit, and may cause the processor to perform various actions required for the operating system to input and output data and store data in memory, including storing files on the non-volatile memory and / or the drive unit.

[0113]

[0123] Some portions of the detailed descriptions may be presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0114]

[0124] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities.Unless otherwise indicated, and as will be apparent from the following description, it will be appreciated that throughout the description, descriptions utilizing terms such as "processing" or "calculating" or "computing" or "determining" or "displaying" or "generating" refer to the acts and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data that are similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission or display device.

[0115]

[0125] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform some example methods. The required structure for a variety of these systems will appear from the description below. In addition, the techniques are not described with reference to any particular programming language, and thus the various examples may be implemented using a variety of programming languages.

[0116]

[0126] In various implementations, the system may operate as a standalone device or may be connected (e.g., networked) to other systems. In a networked deployment, the system may operate in the capacity of a server or a client system in a client-server network environment, or as a peer system in a peer-to-peer (or distributed) network environment.

[0117]

[0127] The system may be a server computer, a client computer, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, an iPhone, a Blackberry, a processor, a telephone, a web appliance, a network router, a switch or bridge, or any system capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the system.

[0118]

[0128] Although a machine-readable medium or machine-readable storage medium is shown as being a single medium by way of example, the terms "machine-readable medium" and "machine-readable storage medium" should be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The terms "machine-readable medium" and "machine-readable storage medium" are also intended to include any medium capable of storing, encoding, or carrying a set of instructions for a system to execute, causing the system to perform any one or more of the methods or modules disclosed herein.

[0119]

[0129] Generally, the routines executed to implement implementations of the present disclosure may be implemented as parts of an operating system or a specific application, component, program, object, module or set of instructions referred to as a "computer program." A computer program generally comprises one or more sets of instructions at different times in various memory and storage devices in the computer that, when read and executed by one or more processing units or processors in a computer, cause the computer to perform operations to implement elements involved in various aspects of the present disclosure.

[0120]

[0130] Furthermore, while the examples have been described in the context of fully functional computers and computer systems, those skilled in the art will appreciate that the various examples may be distributed as program objects in a variety of forms, and that the present disclosure applies equally regardless of the particular type of machine or computer-readable medium used to actually affect the distribution.

[0121]

[0131] Further examples of machine-readable storage media, machine-readable media, or computer-readable (storage) media include, but are not limited to, recordable-type media such as volatile and non-volatile memory devices, floppy disks and other removable disks, hard disk drives, optical disks (e.g., compact disks read-only (CD ROM), digital versatile disks (DVD), etc.), and transmission-type media such as digital and analog communications links, among others.

[0122]

[0132] In some circumstances, an operation of a memory device, such as a change of state from a binary 1 to a binary 0 or vice versa, may comprise a transformation, such as, for example, a physical transformation. In certain types of memory devices, such a physical transformation may comprise a physical transformation of an article to a different state or thing. For example, but not limited to, for some types of memory devices, a change of state may involve the accumulation and storage of an electric charge or the release of a stored electric charge. Similarly, in other memory devices, a change of state may comprise a physical change or transformation of a magnetic orientation or a physical change or transformation of a molecular structure from crystalline to amorphous or vice versa. The above is not intended to be an exhaustive list of all examples in which a change of state of a binary 1 to a binary 0 or vice versa in a memory device may comprise a transformation, such as a physical transformation. Rather, the above is intended as an illustrative example.

[0123]

[0133] Storage media may generally be non-transitory or comprise a non-transitory device. In this context, non-transitory storage media may include a device that is tangible, meaning that the device has a concrete physical form, but the device may change its physical state. Thus, for example, non-transitory refers to a device that remains tangible despite this change in state.

[0124]

[0134] The above description and drawings are illustrative and should not be construed as limiting the subject matter to the precise form disclosed. Those skilled in the art can appreciate that many modifications and variations are possible in light of the above disclosure. Numerous specific details are described in order to provide a thorough understanding of the present disclosure. However, in some instances, well-known or conventional details are not described in order to avoid obscuring the description.

[0125]

[0135] As used herein, the terms "connected," "coupled," or any variation thereof, when applied to modules of a system, refer to any direct or indirect connection or coupling between two or more elements, and the coupling of connections between elements may be physical, logical, or any combination thereof. Additionally, the terms "herein," "above," "below," and terms of similar import, when used herein, refer to this specification as a whole and not to any particular portion of this specification. Where the context permits, terms in the above Detailed Description using the singular or plural may also include the plural or singular, respectively. The term "or" in reference to a list of two or more items covers all interpretations of the term to any of the items in the list, all of the items in the list, or any combination of the items in the list.

[0126]

[0136] Those skilled in the art will appreciate that the disclosed subject matter may be embodied in other forms and manners not set forth below. It should be understood that the use of relationship-indicating terminology, such as first, second, top, and bottom, if any, is used only to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between such entities or actions.

[0127]

[0137] Although processes or blocks are presented in a given order, alternative implementations may perform routines having steps or employ systems having blocks in a different order, and some processes or blocks may be deleted, moved, added, sub-divided, substituted, combined, and / or modified to provide alternative or sub-combinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are sometimes shown as being performed in series, these processes or blocks may instead be performed in parallel or at different times. Additionally, any specific numbers referenced herein are examples only, and alternative implementations may employ different values ​​or ranges.

[0128]

[0138] The teachings of the disclosure provided herein may be applied to other systems, not necessarily limited to the systems described above. Elements and acts of the various examples described above may be combined to provide further examples.

[0129]

[0139] Any patents and applications and other references referred to above, including those that may be listed in accompanying application papers, are incorporated herein by reference. Aspects of the present disclosure may be modified, where necessary, to adopt the systems, functions, and concepts of the various references described above to provide yet further examples of the present disclosure.

[0130]

[0140] These and other changes may be made to the present disclosure in light of the above detailed description. Although the above description describes some examples and describes the best modes contemplated, no matter how detailed it may appear in the text, the teachings may be implemented in many ways. The details of the system are still encompassed by the subject matter disclosed herein, but may vary widely in its implementation details. As stated above, a particular term used when describing some features or aspects of the present disclosure should not be taken to imply that the term is redefined herein to be limited to any particular characteristic, feature, or aspect of the disclosure with which the term is associated. In general, the terms used in the following claims should not be construed to limit the disclosure to the particular implementations disclosed herein, unless the above detailed description section explicitly defines such terms. Thus, the actual scope of the present disclosure encompasses not only the disclosed implementations, but also all equivalent ways of practicing or implementing the present disclosure under the claims.

[0131]

[0141] Although some aspects of the disclosure are presented below in some claim form, the inventors contemplate various aspects of the disclosure in any number of claim forms. Any claim intended to be treated under 35 U.S.C. 112(f) will begin with the phrase "means for." Accordingly, the applicants reserve the right to add additional claims after filing this application to pursue such additional claim forms for other aspects of the disclosure.

[0132]

[0142] The terms used in this specification generally have their ordinary meaning in the art in the specific context in which each term is used within the context of this disclosure. Some terms used to describe this disclosure are explained above or elsewhere in this specification to provide additional guidance to attorneys regarding the description of this disclosure. For convenience, some terms may be highlighted, for example, using capitalization, italics, and / or quotation marks. The use of highlighting has no effect on the scope and meaning of a term, and the scope and meaning of a term is the same in the same context whether it is highlighted or not. It will be appreciated that the same element may be described in more than one way.

[0133]

[0143] Thus, alternative wording and synonyms may be used for any one or more of the terms described herein, and no special importance should be placed on whether a term is recited or explained herein. Synonyms for some terms are provided. The recitation of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any term described herein, is merely illustrative and is not intended to further limit the scope and meaning of the disclosure or any exemplified term. Similarly, the disclosure is not limited to the various examples provided herein.

[0134]

[0144] Without intending to further limit the scope of the present disclosure, examples of instruments, devices, methods and their related results according to the examples of the present disclosure are given below. Please note that titles or subtitles may be used in the examples for the convenience of the reader, and in any case, this should not limit the scope of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this disclosure pertains. In case of conflict, the present disclosure, including definitions, shall prevail.

[0135]

[0145] Some portions of this specification describe examples in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Further, it has proven convenient at times to refer to these configurations of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0136]

[0146] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In some examples, a software module is implemented in a computer program object comprising a computer readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0137]

[0147] Examples may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium that may be coupled to a computer system bus, or any type of medium suitable for storing electronic instructions. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design to increase computing power.

[0138]

[0148] Examples may also relate to objects generated by the computing processes described herein. Such objects may comprise information resulting from the computing processes, which may be stored on a non-transitory tangible computer-readable storage medium and may include any implementation of the computer program objects or other data combinations described herein.

[0139]

[0149] The language used in this specification has been selected primarily for ease of reading and instructional purposes, and may not have been selected to delineate or limit the subject matter. Therefore, it is intended that the scope of the disclosure be limited not by this detailed description, but by the claims published on the application based on this specification. Therefore, the disclosure of the examples is intended to be illustrative, but not limiting, of the scope of the subject matter described in the following claims.

[0140]

[0150] Specific details have been given in the preceding description to provide a thorough understanding of various implementations of systems and components for a context-connected system. However, those skilled in the art will appreciate that the implementations described above may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0141]

[0151] It should also be noted that individual implementations may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process is terminated when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0142]

[0152] The client devices, network devices, and other devices may be computing systems that include, among other things, one or more integrated circuits, input devices, output devices, data storage devices, and / or network interfaces. The integrated circuits may include, among other things, one or more processors, volatile memory, and / or non-volatile memory. The input devices may include, among other things, keyboards, mice, keypads, touch interfaces, microphones, cameras, and / or other types of input devices. The output devices may include, among other things, display screens, speakers, haptic feedback systems, printers, and / or other types of output devices. Data storage devices, such as hard drives or flash memory, may enable the computing device to store data temporarily or permanently. Network interfaces, such as wireless or wired interfaces, may enable the computing device to communicate with a network. Examples of computing devices include desktop computers, laptop computers, server computers, handheld computers, tablets, smartphones, personal digital assistants, digital home assistants, and machines and equipment in which computing devices are embedded.

[0143]

[0153] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instruction(s) and / or data. Computer-readable media may include non-transitory media on which data may be stored, which does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0144]

[0154] The various examples described above may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable storage medium (e.g., a medium for storing program code or code segments). Processor(s) implemented in an integrated circuit may perform the necessary tasks.

[0145]

[0155] When a component is described as being "configured to" perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0146]

[0156] The various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0147]

[0157] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device with multiple uses, including applications in wireless communication device handsets and other devices. Features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. When implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0148]

[0158] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured to implement an interrupted database update system.

[0149]

[0159] The above detailed description of the present technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the present technology to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The described embodiments have been chosen to best explain the principles of the present technology and its practical application, and to enable others skilled in the art to best utilize the present technology in various embodiments with various modifications as may be suitable for the particular applications contemplated. It is intended that the scope of the present technology be defined by the claims. The invention as described in the claims of the original application is set forth below. [C1] 1. A computer-implemented method, comprising: receiving conversation data corresponding to a conversation between a conversational bot agent and a customer, wherein the conversational data includes messages exchanged between the conversational bot agent and the customer; calculating a set of features from the conversation data; detecting a set of bot states corresponding to a performance of the conversational bot agent in the conversation, wherein the set of bot states is detected based on the set of features and the conversation data. training a machine learning model using the sample conversation data and the corresponding bot states, where training includes determining a Meaningful Auto-Connection Score (MACS) for the conversation; providing a MACS for the conversation and the set of bot states; receiving updates to the conversational bot agent; implementing the updates to the conversational bot agent, wherein the updates are used to train the machine learning model. A computer-implemented method comprising: [C2] training a bot state machine learning model using the sample conversation data and annotations created for the sample conversation data, where training includes determining a bot state for the conversation; using the bot state machine learning model to detect the set of bot states; The computer-implemented method of C1, further comprising: [C3] calculating a confidence score corresponding to a set of possible bot states, where the confidence score is calculated based on the set of features and the conversation data; applying a threshold to the confidence scores, where a bot state is detected as a result of a corresponding confidence score greater than the threshold. The computer-implemented method of C1, further comprising: [C4] extracting a conversation length corresponding to the sample conversation data; using the conversation length to further train the machine learning model, wherein the machine learning model is further trained to identify a correlation between conversation length and the MACS for the conversation. The computer-implemented method of C1, further comprising: [C5] receiving new messages related to the conversation; generating a determination as to whether a new bot state is present in the new message; dynamically updating the MACS for the conversation, wherein the MACS is updated based on the determination, the conversation data, and the new messages; and providing the updated MACS. The computer-implemented method of C1, further comprising: [C6] The computer-implemented method of C1, wherein the machine learning model is further trained based on a confidence score corresponding to the presence of a bot state in the sample conversation data, wherein the confidence score is obtained from a bot state machine learning model. [C7] The computer-implemented method of claim 1, wherein the machine learning model is further trained using a set of features extracted from the sample speech data. [C8] 1. A system comprising: one or more processors; A memory for storing instructions; the instructions, when executed by the one or more processors, provide to the system: receiving conversation data corresponding to a conversation between a conversational bot agent and a customer, wherein the conversational data includes messages exchanged between the conversational bot agent and the customer; calculating a set of features from the conversation data; detecting a set of bot states corresponding to a performance of the conversational bot agent in the conversation, wherein the set of bot states is detected based on the set of features and the conversation data. training a machine learning model using the sample conversation data and the corresponding bot states, where training includes determining a Meaningful Auto-Connection Score (MACS) for the conversation; providing a MACS for the conversation and the set of bot states; receiving updates to the conversational bot agent; implementing the updates to the conversational bot agent, wherein the updates are used to train the machine learning model. A system that allows the user to: [C9] The instructions for causing the system to detect the set of bot states include instructions for causing the system to: training a bot state machine learning model using the sample conversation data and annotations created for the sample conversation data, where training includes determining a bot state for the conversation; using the bot state machine learning model to detect the set of bot states; The system according to C8, further comprising: [C10] The instructions for causing the system to detect the set of bot states include instructions for causing the system to: calculating a confidence score corresponding to a set of possible bot states, where the confidence score is calculated based on the set of features and the conversation data; applying a threshold to the confidence scores, where a bot state is detected as a result of a corresponding confidence score greater than the threshold. The system according to C8, further comprising: [C11] The instructions include: extracting a conversation length corresponding to the sample conversation data; using the conversation length to further train the machine learning model, wherein the machine learning model is further trained to identify a correlation between conversation length and the MACS for the conversation. The system according to C8, further comprising: [C12] The instructions include: receiving new messages related to the conversation; generating a determination as to whether a new bot state is present in the new message; dynamically updating the MACS for the conversation, wherein the MACS is updated based on the determination, the conversation data, and the new messages; and providing the updated MACS. The system according to C8, further comprising: [C13] The system of C8, wherein the machine learning model is further trained based on a confidence score corresponding to the presence of a bot state in the sample conversation data, wherein the confidence score is obtained from a bot state machine learning model. [C14] The system of C8, wherein the machine learning model is further trained using a set of features extracted from the sample conversation data. [C15] A non-transitory computer-readable storage medium having executable instructions stored thereon, the executable instructions, when executed by one or more processors of a computer system, causing the computer system to: receiving conversation data corresponding to a conversation between a conversational bot agent and a customer, wherein the conversational data includes messages exchanged between the conversational bot agent and the customer; calculating a set of features from the conversation data; detecting a set of bot states corresponding to a performance of the conversational bot agent in the conversation, wherein the set of bot states is detected based on the set of features and the conversation data. training a machine learning model using the sample conversation data and the corresponding bot states, where training includes determining a Meaningful Auto-Connection Score (MACS) for the conversation; providing a MACS for the conversation and the set of bot states; receiving updates to the conversational bot agent; implementing the updates to the conversational bot agent, wherein the updates are used to train the machine learning model. A non-transitory computer-readable storage medium that causes [C16] The executable instructions for causing the computer system to detect the set of bot states include: training a bot state machine learning model using the sample conversation data and annotations created for the sample conversation data, where training includes determining a bot state for the conversation; using the bot state machine learning model to detect the set of bot states; The non-transitory computer-readable medium of C15, further comprising: [C17] The executable instructions for causing the computer system to detect the set of bot states include instructions for causing the computer system to: calculating a confidence score corresponding to a set of possible bot states, where the confidence score is calculated based on the set of features and the conversation data; applying a threshold to the confidence scores, where a bot state is detected as a result of a corresponding confidence score greater than the threshold. The non-transitory computer-readable medium of C15, further comprising: [C18] The executable instructions may include instructions for causing the computer system to: extracting a conversation length corresponding to the sample conversation data; using the conversation length to further train the machine learning model, wherein the machine learning model is further trained to identify a correlation between conversation length and the MACS for the conversation. The non-transitory computer-readable medium of C15, further comprising: [C19] The executable instructions may include instructions for causing the computer system to: receiving new messages related to the conversation; generating a determination as to whether a new bot state is present in the new message; dynamically updating the MACS for the conversation, wherein the MACS is updated based on the determination, the conversation data, and the new messages; and providing the updated MACS. The non-transitory computer-readable medium of C15, further comprising: [C20] The non-transitory computer-readable medium of C15, wherein the machine learning model is further trained based on a confidence score corresponding to the presence of a bot state in the sample conversation data, wherein the confidence score is obtained from a bot state machine learning model. [C21] The non-transitory computer-readable storage medium of C15, wherein the machine learning model is further trained using a set of features extracted from the sample conversation data.

Claims

1. 1. A computer-implemented method comprising: receiving conversation data corresponding to a conversation between a conversational bot agent and a customer, wherein the conversational data includes messages exchanged between the conversational bot agent and the customer; calculating a set of features from the conversation data; detecting a set of bot states corresponding to a performance of the conversational bot agent in the conversation, wherein the set of bot states is detected based on the set of features and the conversation data. training a machine learning model using the sample conversation data and the corresponding bot states, where training includes determining a Meaningful Auto-Connection Score (MACS) for the conversation; providing a MACS for the conversation and the set of bot states; receiving updates to the conversational bot agent; implementing the updates to the conversational bot agent, wherein the updates are used to train the machine learning model. A computer-implemented method comprising:

2. training a bot state machine learning model using the sample conversation data and annotations created for the sample conversation data, where training includes determining a bot state for the conversation; using the bot state machine learning model to detect the set of bot states; The computer-implemented method of claim 1 , further comprising:

3. calculating a confidence score corresponding to a set of possible bot states, wherein the confidence score is calculated based on the set of features and the conversation data; applying a threshold to the confidence score, wherein a bot state is detected as a result of a corresponding confidence score being greater than the threshold. The computer-implemented method of claim 1 , further comprising:

4. extracting a conversation length corresponding to the sample conversation data; using the conversation length to further train the machine learning model, wherein the machine learning model is further trained to identify a correlation between conversation length and the MACS for the conversation. The computer-implemented method of claim 1 , further comprising:

5. receiving new messages related to the conversation; generating a determination as to whether a new bot state is present in the new message; dynamically updating the MACS for the conversation, wherein the MACS is updated based on the determination, the conversation data, and the new messages. Providing said updated MACS; and The computer-implemented method of claim 1 , further comprising:

6. 2. The computer-implemented method of claim 1, wherein the machine learning model is further trained based on confidence scores corresponding to the presence of a bot state in the sample conversation data, wherein the confidence scores are obtained from a bot state machine learning model.

7. The computer-implemented method of claim 1 , wherein the machine learning model is further trained using a set of features extracted from the sample speech data.

8. 1. A system comprising: one or more processors; A memory for storing instructions; the instructions, when executed by the one or more processors, provide the system with: receiving conversation data corresponding to a conversation between a conversational bot agent and a customer, wherein the conversational data includes messages exchanged between the conversational bot agent and the customer; calculating a set of features from the conversation data; detecting a set of bot states corresponding to a performance of the conversational bot agent in the conversation, wherein the set of bot states is detected based on the set of features and the conversation data. training a machine learning model using the sample conversation data and the corresponding bot states, where the training includes determining a Meaningful Auto-Connection Score (MACS) for the conversation; providing a MACS for the conversation and the set of bot states; receiving updates to the conversational bot agent; implementing the updates to the conversational bot agent, wherein the updates are used to train the machine learning model; A system that allows the user to:

9. The instructions for causing the system to detect the set of bot states include instructions for causing the system to: training a bot state machine learning model using the sample conversation data and annotations created for the sample conversation data, where training includes determining a bot state for the conversation; using the bot state machine learning model to detect the set of bot states; The system of claim 8 , further comprising:

10. The instructions for causing the system to detect the set of bot states include instructions for causing the system to: calculating a confidence score corresponding to a set of possible bot states, wherein the confidence score is calculated based on the set of features and the conversation data; applying a threshold to the confidence score, wherein a bot state is detected as a result of a corresponding confidence score being greater than the threshold. The system of claim 8 , further comprising:

11. The instructions include: extracting a conversation length corresponding to the sample conversation data; using the conversation length to further train the machine learning model, wherein the machine learning model is further trained to identify a correlation between conversation length and the MACS for the conversation. The system of claim 8 , further comprising:

12. The instructions include: receiving new messages related to the conversation; generating a determination as to whether a new bot state is present in the new message; dynamically updating the MACS for the conversation, wherein the MACS is updated based on the determination, the conversation data, and the new messages. Providing said updated MACS; and The system of claim 8 , further comprising:

13. 10. The system of claim 8, wherein the machine learning model is further trained based on a confidence score corresponding to a presence of a bot state in the sample conversation data, wherein the confidence score is obtained from a bot state machine learning model.

14. The system of claim 8 , wherein the machine learning model is further trained using a set of features extracted from the sample speech data.

15. A non-transitory computer-readable storage medium having executable instructions stored thereon, the executable instructions, when executed by one or more processors of a computer system, causing the computer system to: receiving conversation data corresponding to a conversation between a conversational bot agent and a customer, wherein the conversational data includes messages exchanged between the conversational bot agent and the customer; calculating a set of features from the conversation data; detecting a set of bot states corresponding to a performance of the conversational bot agent in the conversation, wherein the set of bot states is detected based on the set of features and the conversation data. training a machine learning model using the sample conversation data and the corresponding bot states, where the training includes determining a Meaningful Auto-Connection Score (MACS) for the conversation; providing a MACS for the conversation and the set of bot states; receiving updates to the conversational bot agent; implementing the updates to the conversational bot agent, wherein the updates are used to train the machine learning model; A non-transitory computer-readable storage medium that causes

16. The executable instructions for causing the computer system to detect the set of bot states include: training a bot state machine learning model using the sample conversation data and annotations created for the sample conversation data, where training includes determining a bot state for the conversation; using the bot state machine learning model to detect the set of bot states; 20. The non-transitory computer-readable storage medium of claim 15, further comprising:

17. The executable instructions for causing the computer system to detect the set of bot states include: calculating a confidence score corresponding to a set of possible bot states, wherein the confidence score is calculated based on the set of features and the conversation data; applying a threshold to the confidence score, wherein a bot state is detected as a result of a corresponding confidence score being greater than the threshold.

20. The non-transitory computer-readable storage medium of claim 15, further comprising:

18. The executable instructions may include instructions for causing the computer system to: extracting a conversation length corresponding to the sample conversation data; using the conversation length to further train the machine learning model, wherein the machine learning model is further trained to identify a correlation between conversation length and the MACS for the conversation.

20. The non-transitory computer-readable storage medium of claim 15, further comprising:

19. The executable instructions may include instructions for causing the computer system to: receiving new messages related to the conversation; generating a determination as to whether a new bot state is present in the new message; dynamically updating the MACS for the conversation, wherein the MACS is updated based on the determination, the conversation data, and the new messages. Providing said updated MACS; and 20. The non-transitory computer-readable storage medium of claim 15, further comprising:

20. 16. The non-transitory computer-readable storage medium of claim 15, wherein the machine learning model is further trained based on confidence scores corresponding to the presence of a bot state in the sample conversation data, wherein the confidence scores are obtained from a bot state machine learning model.

21. 16. The non-transitory computer-readable storage medium of claim 15, wherein the machine learning model is further trained using a set of features extracted from the sample speech data.

Images