Confidence classifiers in the context of intent classification

Abstract

A method of applying a confidence classifier for intent classification associated with an automated chatbot, according to one embodiment, includes processing an utterance with the intent classifier, by a computing system, to determine a probability distribution of possible intents associated with the utterance, generating, by the computing system, a plurality of measures of kurtosis of the probability distribution, and applying, by the computing system, the trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the plurality of measures of kurtosis of the probability distribution.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Non-provisional Patent Application No. 17 / 135,114, filed December 28, 2020, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Intent classification is a key function of a typical natural language understanding (NLU) system. An intent classifier typically receives a user's speech in a transcribed format (or text input) and outputs the probability that the user's utterance is associated with a particular predefined intent. For example, a user interface associated with an airline may allow four options: book a ticket, cancel a ticket, get flight information, and get the airline's baggage policy. An intent classifier for such an interface may receive a user's text / voice input and analyze the input to determine which of the four options the user most likely intended to select. As the number of options increases, verifying true user intent often becomes more complex. Summary of the Invention

[0003] One embodiment is directed to unique systems, components, and methods for applying confidence classifiers for intent classification associated with an automated chatbot. Other embodiments are directed to apparatus, systems, devices, hardware, methods, and combinations thereof for applying confidence classifiers for intent classification associated with an automated chatbot.

[0004] According to one embodiment, a method of applying a confidence classifier for intent classification associated with an automated chatbot may include: processing an utterance with an intent classifier, by a computing system, to determine a probability distribution of possible intents associated with the utterance; generating, by the computing system, a plurality of measures of kurtosis of the probability distribution; and applying, by the computing system, the trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the plurality of measures of kurtosis of the probability distribution.

[0005] In some embodiments, the method may further include comparing, by the computing system, the normalized probability of the most likely intent associated with the utterance to a confidence threshold.

[0006] In some embodiments, the method may further include: in response to determining, by the computing system, that the normalized probability of the most likely intent associated with the utterance satisfies a confidence threshold, selecting the most likely intent as the intent associated with the utterance; and in response to selecting, by the computing system, the most likely intent as the intent associated with the utterance, sending a message to a user device in communication with the automated chatbot, where the message is a response to the utterance.

[0007] In some embodiments, generating the multiple measures of kurtosis of the probability distribution may include sorting the probability scores of the probability distribution in descending order, selecting a subset of the largest probabilities of the sorted probability scores as a probability set, determining multiple ratios of the consecutively ranked probabilities in the probability set, determining a kurtosis score of the probabilities in the probability set, determining an entropy of the probability distribution, and generating a normalized entropy by dividing the entropy of the probability distribution by the maximum possible entropy of the probability distribution to normalize the entropy.

[0008] In some embodiments, generating the plurality of measures of kurtosis of the probability distribution may include applying a sigmoid function to each of the plurality of ratios, the kurtosis score, and the normalized entropy.

[0009] In some embodiments, selecting a subset of the largest probabilities of the sorted probability scores as the probability set may include selecting the five largest probabilities of the sorted probability scores as the probability set.

[0010] In some embodiments, generating the plurality of measures of kurtosis of the probability distribution may include determining a kurtosis score for the probability distribution.

[0011] In some embodiments, generating the plurality of measures of kurtosis of the probability distribution may include determining the entropy of the probability distribution.

[0012] In some embodiments, generating the plurality of measures of kurtosis of the probability distribution may include ranking the probabilities of the probability distribution by maximum probability and determining a plurality of ratios of the successively ranked probabilities in response to ranking the probabilities of the probability distribution.

[0013] In some embodiments, the method may further include training, by the computing system, the confidence classifier by tuning parameters using stochastic gradient descent optimization.

[0014] According to another embodiment, a system for applying a confidence classifier for intent classification associated with an automated chatbot may include at least one processor and at least one memory, the at least one memory including a plurality of instructions stored thereon that, when executed by the at least one processor, cause the system to process an utterance with the intent classifier to determine a probability distribution of possible intents associated with the utterance, generate a plurality of measures of kurtosis of the probability distribution, and apply the trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the plurality of measures of kurtosis of the probability distribution.

[0015] In some embodiments, the instructions may further cause the system to compare the normalized probability of the most likely intent associated with the utterance to a confidence threshold.

[0016] In some embodiments, the instructions may further cause the system to: in response to determining that the normalized probability of the most likely intent associated with the utterance satisfies a confidence threshold, select the most likely intent as the intent associated with the utterance; and in response to selecting the most likely intent as the intent associated with the utterance, send a message to a user device in communication with the automated chatbot, where the message is a response to the utterance.

[0017] In some embodiments, generating the multiple measures of kurtosis of the probability distribution may include sorting the probability scores of the probability distribution in descending order, selecting a subset of the largest probabilities of the sorted probability scores as a probability set, determining multiple ratios of the consecutively ranked probabilities in the probability set, determining kurtosis scores of the probabilities in the probability set, determining the entropy of the probability distribution, and normalizing the entropy by dividing the entropy of the probability distribution by the maximum possible entropy of the probability distribution to generate a normalized entropy.

[0018] In some embodiments, generating the plurality of measures of kurtosis of the probability distribution may include applying a sigmoid function to each of the plurality of ratios, the kurtosis score, and the normalized entropy.

[0019] In some embodiments, selecting a subset of the largest probabilities of the sorted probability scores as the probability set may include selecting the five largest probabilities of the sorted probability scores as the probability set.

[0020] In some embodiments, generating the plurality of measures of kurtosis of the probability distribution may include determining a kurtosis score for the probability distribution.

[0021] In some embodiments, generating the plurality of measures of kurtosis of the probability distribution may include determining the entropy of the probability distribution.

[0022] In some embodiments, generating the plurality of measures of kurtosis of the probability distribution may include ranking the probabilities of the probability distribution by maximum probability and determining a plurality of ratios of the successively ranked probabilities in response to ranking the probabilities of the probability distribution.

[0023] In some embodiments, the instructions may further cause the system to train the confidence classifier by adjusting parameters using stochastic gradient descent optimization.

[0024] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter. Further embodiments, forms, features, and aspects of the present application will become apparent from the description and figures provided herewith. [Brief explanation of the drawings]

[0025] The concepts described herein are illustrated by way of example, and not by way of limitation, in the accompanying drawings. For simplicity and clarity of illustration, elements illustrated in the figures have not necessarily been drawn to scale. Where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements. [Figure 1] FIG. 1 is a simplified block diagram of at least one embodiment of a system for applying a confidence classifier for intent classification. [Figure 2] FIG. 1 is a simplified block diagram of at least one embodiment of a cloud-based system. [Figure 3] FIG. 1 is a simplified block diagram of at least one embodiment of a computing system. [Figure 4] FIG. 1 is a simplified flow diagram of at least one embodiment of a method for training a confidence classifier. [Figure 5] FIG. 1 is a simplified flow diagram of at least one embodiment of a method for generating a measure of kurtosis of a probability distribution. [Figure 6] FIG. 1 is a simplified flow diagram of at least one embodiment of a method for determining intent using a trained confidence classifier. DETAILED DESCRIPTION OF THE INVENTION

[0026] While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that there is no intention to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the scope of the present disclosure and the appended claims.

[0027] References herein to “one embodiment,” “an embodiment,” “exemplary embodiment,” and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but that all embodiments may or may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. It should be further understood that references to “preferred” components or features may indicate the desirability of a particular component or feature with respect to one embodiment, but that the present disclosure is not so limited with respect to other embodiments that may omit such component or feature. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, its implementation with respect to other embodiments, whether explicitly described or not, is within the knowledge of one of ordinary skill in the art. Furthermore, it should be understood that an item in a list of the form “at least one of A, B, and C” can mean (A), (B), (C), (A and B), (B and C), (A and C), or (A, B, and C). Similarly, it should be understood that an item listed in the form "at least one of A, B, or C" can mean (A), (B), (C), (A and B), (B and C), (A and C), or (A, B, and C). Further, with respect to the claims, the use of words and phrases such as "(a)," "(an)," "at least one," and / or "at least one portion," unless specifically stated to the contrary, should be construed as limiting to only one of such elements, and the use of phrases such as "at least a portion" and / or "a portion," unless specifically stated to the contrary, should be construed to encompass both embodiments including only a portion of such elements and embodiments including the whole of such elements.

[0028] The disclosed embodiments may, in some cases, be implemented in hardware, firmware, software, or a combination thereof. The disclosed embodiments may also be implemented as instructions stored on or executed by one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., volatile or non-volatile memory, a media disk, or other media device).

[0029] In the drawings, some structural or method features may be shown in a specific arrangement and / or ordering. However, it should be understood that such specific arrangement and / or ordering may not be required. Rather, in some embodiments, such features may be arranged in a different manner and / or order than that shown in the illustrative drawings, unless indicated to the contrary. Furthermore, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, it may not be included or may be combined with other features.

[0030] 1 , in an exemplary embodiment, a system 100 for applying a confidence classifier for intent classification includes a cloud-based system 102, a network 104, a contact center system 106, and a user device 108. Additionally, the exemplary cloud-based system 102 includes an intent classifier 110 and a confidence classifier 112. Although the exemplary embodiment of FIG. 1 shows only one cloud-based system 102, one network 104, one contact center system 106, one user device 108, one intent classifier 110, and one confidence classifier 112, in other embodiments, the system 100 may include multiple cloud-based systems 102, networks 104, contact center systems 106, user devices 108, intent classifiers 110, and / or confidence classifiers 112. For example, in some embodiments, multiple cloud-based systems 102 (e.g., related or unrelated systems) may be used to perform various functions described herein. Furthermore, in some embodiments, one or more of the systems described herein may be excluded from system 100, one or more of the systems described as being independent may form part of another system, and / or one or more of the systems described as forming part of another system may be independent.

[0031] In some systems, an NLU creator / manager can train an intent classifier using supervised learning techniques by supplying it with a training set of utterances mapped to intent labels. These utterances can be converted (e.g., via a training process) into dense vectors (e.g., arrays of floating-point numbers) that can be processed before optimizing for the intent classifier's parameters. Once completed, the trained model can be utilized when a user provides an input text utterance, which is processed using the same feature generator used during the classifier training process to enable the intent classifier to predict a set of probabilities for possible intents. The input may come from, for example, a dialogue management system, a chatbot, a personal bot, or other system, and the classifier may provide the probabilities to the system so that it can respond and determine the best course of action.

[0032] A range of output probabilities between 0.0 and 1.0 can cause considerable ambiguity for systems that rely on the intent classifier's output. For example, assume an intent classifier is trained to recognize three intents, and a downstream consuming system calibrates a particular response when the highest probability is 0.7 (e.g., a confidence threshold). If each possible intent is equally probable, the probability of each intent should be approximately 0.33. However, if the number of possible intents increases to four, the probability of each equally probable case should be 0.25, not 0.33. As the number of possible intents increases, the probability of each intent continues to decrease. This significantly attenuates the individual probabilities, and therefore the highest probability is typically lower. Furthermore, different numbers of intents can make setting / defining the confidence threshold a moving target.

[0033] The system 100 and techniques described herein enable improved confidence classification in the context of intent classification (e.g., based on user communication with a chatbot or personal bot), which addresses the problem of lower probabilities associated with a larger number of possible intents. As described herein, the output probability distribution can be used to measure the confidence of an intent classifier in predicting the mapping from input utterances to intents, essentially based on a measure of the kurtosis of the probability distribution. The confidence classifier is trained and then used in real time to convert the output probabilities from the intent classifier into normalized output values ​​between 0.0 and 1.0, which serve as a proxy for the intent classifier's best possible intent probability. Thus, the confidence threshold may also be set and maintained within the range of 0.0 to 1.0, such that it is no longer necessary to change the threshold based on the number of possible intent options.

[0034] It should be understood that each of the cloud-based system 102, the network 104, the contact center system 106, the user device 108, the intent classifier 110, and the confidence classifier 112 may be embodied as any type of device / system, collection of devices / systems, or portion thereof suitable for performing the functions described herein.

[0035] Cloud-based system 102 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in an exemplary embodiment, cloud-based system 102 is configured to receive user input data indicative of a user's utterance and process the user input data (e.g., using intent classifier 110 and confidence classifier 112) to determine a most likely user intent associated with the utterance and a corresponding confidence in the system's prediction, which may be used to make a decision (e.g., by another system). As described herein, intent classifier 110 of cloud-based system 102 is configured to process the user input data associated with the user's utterance and output a corresponding probability that the user's utterance corresponds to each of the possible predefined intents analyzed by intent classifier 110. The confidence classifier 112 further analyzes the probability distribution output by the intent classifier 110 to generate multiple measures of kurtosis of the probability distribution (e.g., ratio, kurtosis score, entropy, etc.) and determines a single normalized probability of the most likely intent associated with the utterance based on the kurtosis measures.

[0036] Although cloud-based system 102 is described herein in the singular, it should be understood that cloud-based system 102 may, in some embodiments, be embodied as or include multiple servers / systems. Additionally, while cloud-based system 102 is described herein as a cloud-based system, it should be understood that system 102 may, in other embodiments, be embodied as one or more servers / systems that exist outside of a cloud computing environment. In some embodiments, cloud-based system 102 may be embodied as or similar to cloud-based system 200 described with reference to FIG. 2.

[0037] In a cloud-based embodiment, the cloud-based system 102 may be embodied as a server-ambiguous computing solution that, for example, executes instructions on demand, executes instructions only when prompted by a specific activity / trigger, and does not consume computing resources when not in use. That is, the system 102 may be embodied as a virtual computing environment residing “on” a computing system (e.g., a distributed network of devices) in which various virtual functions (e.g., Lambda functions, Azure functions, Google Cloud Functions, and / or other suitable virtual functions) may be executed corresponding to the functionality of the system 102 described herein. For example, when an event occurs (e.g., data is transferred to the system 102 for processing), the virtual computing environment may be communicated (e.g., via a request to the virtual computing environment's API), which may then route the request to the correct virtual function (e.g., a particular server-ambiguous computing resource) based on a set of rules. Thus, when a request for transmission of data is made by a user (e.g., via an appropriate user interface to the system 102), the appropriate virtual function may be executed to perform the action before deleting the instance of the virtual function.

[0038] Network 104 may be embodied as any one or more types of communications network capable of facilitating communication between various devices communicatively connected via network 104. Accordingly, network 104 may include one or more networks, routers, switches, access points, hubs, computers, and / or other intervening network devices. For example, network 104 may be embodied as or include one or more cellular networks, telephone networks, local or wide area networks, publicly available global networks (e.g., the Internet), ad hoc networks, short-range communications links, or combinations thereof. In some embodiments, network 104 may include circuit-switched voice or data networks, packet-switched voice or data networks, and / or any other network capable of transmitting voice and / or data. In particular, in some embodiments, network 104 may include Internet Protocol (IP)-based and / or asynchronous transfer mode (ATM)-based networks. In some embodiments, network 104 may handle voice traffic (e.g., via a Voice over IP (VOIP) network), web traffic (e.g., hypertext transfer protocol (HTTP) traffic and hypertext markup language (HTML) traffic), and / or other network traffic, depending on the particular embodiment and / or the devices of system 100 that communicate with each other.In various embodiments, network 104 may include analog or digital wired and wireless networks (e.g., IEEE 802.11 networks, Public Switched Telephone Networks (PSTN), Integrated Services Digital Networks (ISDN), and Digital Subscriber Lines (xDSL)), third generation (3G) mobile networks, fourth generation (4G) mobile networks, fifth generation (5G) mobile networks, wired Ethernet networks, private networks (e.g., intranets, etc.), radio, television, cable, satellite, and / or any other delivery or tunneling mechanism for carrying data, or any suitable combination of such networks. Network 104 may enable connectivity between the various devices / systems 102, 106, 108, 110, 112 of system 100. It should be understood that the various devices / systems 102, 106, 108, 110, 112 may communicate with each other over different networks 104 depending on the source and / or destination devices / systems 102, 106, 108, 110, 112.

[0039] It should be understood that in some embodiments, the cloud-based system 102 may be communicatively coupled to, form a part of, and / or otherwise be used in conjunction with the contact center system 106. For example, the contact center system 106 may include a chatbot (e.g., similar to the chatbot 218 of FIG. 2 ) configured to communicate with a user (e.g., via a user device 108). Furthermore, in some embodiments, the user device 108 may communicate directly with the cloud-based system 102.

[0040] The contact center system 106 may be embodied as any system capable of providing contact center services (e.g., call center services) to end users and otherwise performing the functions described herein. It should be understood that, depending on the particular embodiment, the contact center system 106 may be located on the premises / campus of the organization utilizing the contact center system 106 and / or may be located remotely relative to the organization (e.g., in a cloud-based computing environment). In some embodiments, a portion of the contact center system 106 may be located on the premises / campus of the organization, while other portions of the contact center system 106 are located remotely relative to the organization's premises / campus. As such, it should be understood that the contact center system 106 may be deployed within equipment dedicated to the organization or its third-party service provider and / or may be deployed in a remote computing environment, such as, for example, a private or public cloud environment with infrastructure to support multiple contact centers for multiple enterprises. In some embodiments, the contact center system 106 includes resources (e.g., workers, computers, and telecommunications equipment) to enable the delivery of services via telephone and / or other communication mechanisms. Such services may include, for example, technical support, help desk support, emergency response, and / or other contact center services, depending on the particular type of contact center.

[0041] The user device 108 may be embodied as any type of device capable of running applications and otherwise performing the functions described herein. For example, in some embodiments, the user device 108 is configured to run an application for participating in a conversation with a personal bot, an automated agent, a chatbot, or other automated system. Accordingly, the user device 108 may have various input / output devices with which a user can interact and provide and receive audio, text, video, and / or other forms of data. It should be understood that the application may be embodied as any type of application suitable for performing the functions described herein. In particular, in some embodiments, the application may be embodied as a mobile application (e.g., a smartphone application), a cloud-based application, a web application, a thin-client application, and / or another type of application. For example, in some embodiments, the application may function as a client-side interface (e.g., via a web browser) for a web-based application or service.

[0042] It should be understood that each of the cloud-based system 102, the network 104, the contact center system 106, and / or the user devices 108 may be embodied as (and / or include) one or more computing devices similar to the computing device 300 described below with reference to Figure 3. For example, in an exemplary embodiment, each of the cloud-based system 102, the network 104, the contact center system 106, and / or the user devices 108 may include a processing device 302 and a memory 306 that stores operational logic 308 (e.g., instructions) that the processing device 302 executes for operation of the corresponding device.

[0043] 2, there is shown a simplified block diagram of at least one embodiment cloud-based system 200. The exemplary cloud-based system 200 includes a border communication device 202, a SIP server 204, a resource manager 206, a media control platform 208, a speech / text analysis system 210, a voice generator 212, an audio gateway 214, a media augmentation system 216, a chatbot 218, and voice data storage 220. 2 shows only one border communication device 202, one SIP server 204, one resource manager 206, one media control platform 208, one speech / text analysis system 210, one voice generator 212, one audio gateway 214, one media augmentation system 216, one chatbot 218, and one voice data storage 220, in other embodiments, cloud-based system 200 may include multiple border communication devices 202, SIP servers 204, resource managers 206, media control platforms 208, speech / text analysis systems 210, voice generators 212, audio gateways 214, media augmentation systems 216, chatbots 218, and / or voice data storage 220. For example, in some embodiments, multiple chatbots 218 may be used to communicate about different subject matter handled by the same cloud-based system 200. Furthermore, in some embodiments, one or more of the components described herein may be excluded from system 200, one or more of the components described as being independent may form part of another component, and / or one or more of the components described as forming part of another component may be independent.

[0044] The border communication device 202 may be embodied as any one or more types of device / system capable of performing the functions described herein. For example, in some embodiments, the border communication device 202 may be configured to control signaling and media streams involved in the setup, execution, and teardown of voice conversations and other media communications between, for example, an end user and a contact center system. In some embodiments, the border communication device 202 may be a session border controller (SBC) that controls signaling and media exchanged during a media session (also referred to as a “call,” “telephone call,” or “communication session”) between an end user and a contact center system. In some embodiments, the signaling exchanged during a media session may include SIP, H.323, Media Gateway Control Protocol (MGCP), and / or any other voice-over-IP (VoIP) call signaling protocol. The media exchanged during a media session may include media streams carrying audio, video, or other data for the call along with call statistics and quality information.

[0045] In some embodiments, the border communication device 202 may operate according to a standard SIP back-to-back user agent (B2BUA) configuration. In this regard, the border communication device 202 may be inserted into the signaling and media paths established between the calling and called parties in a VoIP call. It should be understood that in some embodiments, other intermediate software and / or hardware devices may be involved in establishing the signaling and / or media paths between the calling and called parties.

[0046] In some embodiments, the border communication device 202 may exercise control over signaling (e.g., SIP messages) and media streams (e.g., RTP data) routed between end user devices (e.g., user devices 108) and contact center systems (e.g., contact center system 106) across a network (e.g., network 104). In this regard, the border communication device 202 may be coupled to trunks that carry signals and media for calls to and from user devices over the network, and trunks that carry signals and media to and from the contact center systems over the network.

[0047] The SIP server 204 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in some embodiments, the SIP server 204 may operate as a SIP B2UBA and control the flow of SIP requests and responses between SIP endpoints. In other embodiments, any other controller configured to set up and tear down VoIP communication sessions may be contemplated in addition to or instead of the SIP server 204. The SIP server 204 may be a separate logical component or may be combined with the resource manager 206. In some embodiments, the SIP server 204 may be hosted in a contact center system (e.g., the contact center system 106). While the exemplary embodiment uses a SIP server 204, in other embodiments, another call server configured with another VoIP protocol, such as the H.232 protocol, the Media Gateway Control Protocol, the Skype protocol, and / or other suitable technologies, may be used in addition to or instead of SIP.

[0048] Resource manager 206 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. In an example embodiment, resource manager 206 may be configured to allocate and monitor a pool of media control platforms to provide load balancing and high availability for each resource type. In some embodiments, resource manager 206 may monitor and select a media control platform 208 from a cluster of available platforms. The selection of a media control platform 208 may be dynamic, for example, based on identification of the calling end user's location, the type of media service to be rendered, the detected quality of the current media service, and / or other factors.

[0049] In some embodiments, resource manager 206 may be configured to process requests for media services and, for example, interact with a configuration server having a configuration database to determine service profiles that can deliver interactive voice response (IVR) profiles, voice applications (e.g., Voice Extensible Markup Language (Voice XML) applications), announcement and conferencing applications, resources, and services such as, for example, a media control platform. According to some embodiments, the resource manager may offer service providers a hierarchical multi-tenant configuration, allowing the service provider to allocate a selected number of resources to each tenant.

[0050] In some embodiments, resource manager 206 may be configured to operate as a SIP proxy, a SIP registrar, and / or a SIP notifier. In this regard, resource manager 206 may act as a proxy for SIP traffic between two SIP components. As a SIP registrar, resource manager 206 may accept registrations of various resources, for example, via SIP REGISTER messages. In this manner, cloud-based system 200 may support transparent relocation of call processing components. In some embodiments, components such as media control platform 208 do not register with resource manager 206 upon startup. Resource manager 206 may discover instances of media control platform 208 through configuration information retrieved from a configuration database. If media control platform 208 is configured for monitoring, resource manager 206 may monitor resource health, for example, by using SIP OPTIONS messages. In some embodiments, resource manager 206 may periodically send a SIP OPTIONS message to each media control platform 208 resource in the group to determine whether the resources in the group are alive. If the resource manager 206 receives an OK response, the resource is considered alive. It should be understood that the resource manager 206 may be configured to perform various other functions that are omitted for brevity. The resource manager 206 and the media control platform 208 may be collectively referred to as a media controller.

[0051] In some embodiments, resource manager 206 may act as a SIP notifier, for example, by accepting SIP SUBSCRIBE requests from SIP server 204 and maintaining multiple independent subscriptions for the same or different SIP devices. Subscription notifications are targeted to tenants managed by resource manager 206. In this role, resource manager 206 may periodically generate SIP NOTIFY requests to subscribers (or tenants) about port usage and the number of available ports. Resource manager 206 can support multi-tenancy by sending notifications that include the tenant name and the current status (in or out of service) of the media control platform 208 associated with the tenant, as well as the tenant's current capacity.

[0052] Media control platform 208 may be embodied as any service or system capable of providing media services and otherwise performing the functions described herein. For example, in some embodiments, media control platform 208 may be configured to provide call and media services in response to requests from service users. Such services may include, but are not limited to, initiating outbound calls, playing music or providing other media while a call is on hold, call recording, conferencing, call progress detection, playing audio / video prompts during customer self-service sessions, and / or other call and media services. One or more of the services may be defined by a voice application (e.g., a VoiceXML application) that executes as part of the process of establishing a media session between media control platform 208 and an end user.

[0053] The speech / text analytics system (STAS) 210 may be embodied as any service or system capable of providing various speech analysis and text processing functions (e.g., text-to-speech) and otherwise performing the functions described herein, as will be appreciated by those skilled in the art. The speech / text analytics system 210 may perform automatic speech and / or text recognition and grammar matching for end-user communication sessions processed by the cloud-based system 200. The speech / text analytics system 210 may include one or more processors and instructions stored on a machine-readable medium that are executed by the processors to perform various operations. In some embodiments, the machine-readable medium may include a non-transitory storage medium, such as a hard disk and a hardware memory system.

[0054] The audio generator 212 may be embodied as any service or system capable of generating audio communications and otherwise performing the functions described herein. In some embodiments, the audio generator 212 may generate audio communications based on a particular audio signature.

[0055] The Audio Gateway 214 may be embodied as any service or system capable of performing the functions described herein. In an exemplary embodiment, the Audio Gateway 214 receives end-user calls from or places calls to voice communication devices, such as end-user devices, and answers the calls according to a voice program corresponding to the communication routing configuration of the contact center system. In some embodiments, the voice program may include a voice avatar. The voice program may be accessed from local memory within the Audio Gateway 214 or from other storage media within the cloud-based system 200. In some embodiments, the Audio Gateway 214 can process voice programs that are script-based voice applications. Thus, the voice program may be a script written in a scripting language such as Voice Extensible Markup Language (VoiceXML) or speech application language tag (SALT). The cloud-based system 200 can also communicate with the voice data storage 220 to read and / or write user interaction data (e.g., state variables for data communication sessions) in a shared memory space.

[0056] Media augmentation system 216 specifies how portions of cloud-based system 200 (e.g., one or more of border communication device 202, SIP server 204, resource manager 206, media control platform 208, speech / text analysis system 210, voice generator 212, audio gateway 214, media augmentation system 216, chatbot 218, voice data storage 220, and / or one or more portions thereof) interact with each other and may be embodied as any service or system capable of otherwise performing the functions described herein. In some embodiments, media augmentation system 216 may be embodied as or include an application program interface (API). In some embodiments, media augmentation system 216 enables integration of different parameters and / or protocols used with various planned applications and media types utilized within cloud-based system 200.

[0057] The chatbot 218 may be embodied as any automated service or system capable of using automation to engage with end users and otherwise perform the functions described herein. For example, in some embodiments, the chatbot 218 may operate as an executable program that may be launched, for example, pursuant to a request for a particular chatbot. In some embodiments, the chatbot 218 simulates and processes human conversations (written or spoken), allowing humans to interact with digital devices as if they were communicating with another human. In some embodiments, the chatbot 218 may be as simple as a rudimentary program that answers simple queries with a single-line response, or as sophisticated as a digital assistant that learns and evolves to deliver increasing levels of personalization as it gathers and processes information. In some embodiments, the chatbot 218 includes and / or leverages artificial intelligence, adaptive learning, bots, cognitive computing, and / or other automation technologies. The chatbot 218 may also be referred to herein as one or more of a chat robot, an AI chatbot, an automated chat robot, a chatterbot, a dialog system, a conversational agent, an automated chat resource, and / or a bot.

[0058] An advantage of utilizing automated chat robots to engage in chat conversations with end users can be that it helps contact centers more efficiently use valuable and expensive resources, such as human resources, while maintaining end-user satisfaction. For example, a chat robot can be initially invoked to handle a chat conversation without the human end user knowing that they are conversing with a robot. The chat conversation can be escalated to human resources, if appropriate. Thus, human resources need not be unnecessarily tied up in handling simple requests, but instead can be more effectively used to handle more complex requests or to monitor the progress of many different automated communications simultaneously.

[0059] Audio data storage 220 may be embodied as one or more databases, data structures, and / or data storage devices capable of storing data in or otherwise facilitating the storage of such data for cloud-based system 200. For example, in some embodiments, audio data storage 220 may include one or more cloud storage buckets. It should be understood that in other embodiments, audio data storage 220 may additionally or alternatively include other types of audio data storage mechanisms that enable dynamic scaling of the amount of data storage available to cloud-based system 200. In some embodiments, audio data storage 220 may store scripts (e.g., pre-programmed scripts, etc.). While audio data storage 220 is described herein as data storage and a database, it should be understood that audio data storage 220 may include both a database (or other type of organized collection of data and structures) and data storage for the actual storage of the underlying data. Audio data storage 220 may store a variety of data useful for performing the functions described herein.

[0060] 3, illustrated is a simplified block diagram of at least one embodiment of a computing device 300. The exemplary computing device 300 illustrates at least one embodiment of a cloud-based system, a contact center system, and / or a user device that may be utilized in connection with the cloud-based system 102, the contact center system 106, and / or the user device 108 (and / or portions thereof) illustrated in FIG. 1. Additionally, in some embodiments, one or more of the border communication device 202, the SIP server 204, the resource manager 206, the media control platform 208, the speech / text analysis system 210, the voice generator 212, the audio gateway 214, the media augmentation system 216, the chatbot 218, and / or the voice data storage 220 (and / or portions thereof) may be embodied as or executed by a computing device similar to the computing device 300. Depending on the particular embodiment, computing device 300 may be embodied as a server, a desktop computer, a laptop computer, a tablet computer, a notebook, a netbook, an Ultrabook™, a mobile phone, a mobile computing device, a smartphone, a wearable computing device, a personal digital assistant, an Internet of Things (IoT) device, a processing system, a wireless access point, a router, a gateway, and / or any other computing, processing, and / or communication device capable of performing the functions described herein.

[0061] The computing device 300 includes a processing device 302 that executes algorithms and / or processes data according to operational logic 308, an input / output device 304 that enables communication between the computing device 300 and one or more external devices 310, and a memory 306 that stores data received from the external device 310 via the input / output device 304, for example.

[0062] The input / output devices 304 enable the computing device 300 to communicate with external devices 310. For example, the input / output devices 304 may include a transceiver, a network adapter, a network card, an interface, one or more communication ports (e.g., a USB port, a serial port, a parallel port, an analog port, a digital port, VGA, DVI, HDMI, FireWire, CAT5, or any other type of communication port or interface), and / or other communication circuitry. The communication circuitry of the computing device 300 may be configured to perform such communication using any one or more communication technologies (e.g., wireless or wired communication) and associated protocols (e.g., Ethernet, Bluetooth, Wi-Fi, WiMAX, etc.) depending on the particular computing device 300. The input / output devices 304 may include hardware, software, and / or firmware suitable for performing the techniques described herein.

[0063] External device 310 may be any type of device that allows data to be input or output from computing device 300. For example, in various embodiments, external device 310 may be embodied as cloud-based system 102, contact center system 106, user device 108, and / or portions thereof. Furthermore, in some embodiments, external device 310 may be embodied as another computing device, a switch, a diagnostic tool, a controller, a printer, a display, an alarm, a peripheral device (e.g., a keyboard, a mouse, a touchscreen display, etc.), and / or any other computing, processing, and / or communication device capable of performing the functions described herein. Furthermore, it should be understood that in some embodiments, external device 310 may be integrated into computing device 300.

[0064] The processing device 302 may be embodied as any type of processor capable of performing the functions described herein. In particular, the processing device 302 may be embodied as one or more single-core or multi-core processors, microcontrollers, or other processors or processing / control circuitry. For example, in some embodiments, the processing device 302 may include or be embodied as an arithmetic logic unit (ALU), a central processing unit (CPU), a digital signal processor (DSP), and / or another suitable processor. The processing device 302 may be of a programmable type, a dedicated hardwired state machine, or a combination thereof. A processing device 302 having multiple processing units may, in various embodiments, utilize distributed processing, pipelined processing, and / or parallel processing. Furthermore, the processing device 302 may be dedicated to performing only the operations described herein or may be utilized in one or more additional applications. In the exemplary embodiment, processing device 302 is programmable and executes algorithms and / or processes data according to operating logic 308 defined by programming instructions (e.g., software or firmware) stored in memory 306. Additionally or alternatively, operating logic 308 of processing device 302 may be defined at least in part by hardwired logic or other hardware. Further, processing device 302 may include one or more components of any type suitable for processing signals received from input / output device 304 or from other components or devices and providing a desired output signal. Such components may include digital circuits, analog circuits, or a combination thereof.

[0065] Memory 306 may be one or more types of non-transitory computer-readable media, such as solid-state memory, electromagnetic memory, optical memory, or a combination thereof. Furthermore, memory 306 may be volatile and / or non-volatile, and in some embodiments, some or all of memory 306 may be of a portable type, such as a disk, tape, memory stick, cartridge, and / or other suitable portable memory. During operation, memory 306 may store various data and software used during operation of computing device 300, such as an operating system, applications, programs, libraries, and drivers. It should be understood that memory 306 may store data manipulated by operating logic 308 of processing device 302, such as, for example, data representing signals received from and / or transmitted to input / output devices 304, in addition to or instead of storing programming instructions defining operating logic 308. As shown in FIG. 3 , memory 306 may be included in and / or coupled to processing device 302, depending on the particular embodiment. For example, in some embodiments, the processing device 302, memory 306, and / or other components of the computing device 300 may form part of a system-on-a-chip (SoC) and be integrated into a single integrated circuit chip.

[0066] In some embodiments, various components of computing device 300 (e.g., processing device 302 and memory 306) may be communicatively coupled via an input / output subsystem, which may be embodied as circuits and / or components for facilitating input / output operations with processing device 302, memory 306, and other components of computing device 300. For example, the input / output subsystem may be embodied as or otherwise include a memory controller hub, an input / output control hub, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems for facilitating input / output operations.

[0067] Computing device 300 may, in other embodiments, include other or additional components, such as those commonly found in a typical computing device (e.g., various input / output devices and / or other components). It should further be understood that one or more of the components of computing device 300 described herein may be distributed across multiple computing devices. In other words, the techniques described herein may be employed by a computing system including one or more computing devices. In addition, while only a single processing device 302, I / O device 304, and memory 306 are illustratively shown in FIG. 3 , it should be understood that a particular computing device 300 may, in other embodiments, include multiple processing devices 302, I / O devices 304, and / or memory 306. Furthermore, in some embodiments, two or more external devices 310 may communicate with computing device 300.

[0068] 4, in use, system 100 (e.g., cloud-based system 102) may perform method 400 for training a confidence classifier. It should be understood that certain blocks of method 400 are shown by way of example, and that such blocks may be combined, divided, added, removed, and / or reordered in whole or in part, depending on the particular embodiment, unless stated to the contrary. In some embodiments, method 400 is associated with a user conversation with an automated chatbot, a personal bot, and / or other type of automated conversation system.

[0069] The example method 400 begins at block 402, where the system 100 (or, more specifically, the confidence classifier 112) receives a probability distribution of possible intentions associated with an utterance. In other words, the probability distribution includes corresponding probabilities that the user's utterance corresponds to each of the possible intentions processed by the intent classifier 110 as described herein. It should be understood that in some embodiments, the system 100 may process the user's input corresponding to the utterance as described above (e.g., via the intent classifier 110) to arrive at the probability distribution, while in other embodiments, the system 100 receives the probability distribution (e.g., as the corresponding probabilities) from another system.

[0070] At block 404, system 100 generates multiple measures of kurtosis of the probability distribution. To do so, in some embodiments, system 100 may perform method 500 of FIG. 5, described in detail below, which utilizes three different measures of kurtosis of the probability distribution. However, it should be understood that in other embodiments, a different number of measures of kurtosis of the probability distribution and / or different specific measures of kurtosis may be used.

[0071] In block 406, the system 100 trains the confidence classifier 112 using the multiple measures of kurtosis as features for classification. In an exemplary embodiment, the confidence classifier 112 outputs a single normalized probability of the most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution. To train the confidence classifier 112, the intention classifier 110 may be provided with positive examples representing correct pairings of an utterance Ui with an intent label L i , and a set of features (e.g., multiple measures of kurtosis) may be extracted and provided to the confidence classifier 112 with a target of 1.0 for the confidence classifier 112 to train. Similarly, the intention classifier 110 may be provided with negative examples representing incorrect pairings of an utterance with a randomly selected utterance from another class / label, and a set of features (e.g., multiple measures of kurtosis) may be extracted and provided to the confidence classifier 112 with a target of 0.0 for the confidence classifier 112 to train. This process may be repeated over multiple iterations (eg, until all or multiple possibilities are exhausted) to train the confidence classifier 112.

[0072] In some embodiments, at block 408, the system 100 may use stochastic gradient descent optimization to tune the parameters of the confidence classifier 112. In other embodiments, the system 100 may utilize one or more other algorithms for training the confidence classifier 112 using multiple measures of kurtosis as features for classification.

[0073] It should be appreciated that once trained, the single number (e.g., in the range of 0.0 to 1.0) output by confidence classifier 112 can be used by system 100 (or another system) to classify real-time intent data (associated with a user's utterance) using a single confidence threshold (e.g., in the range of 0.0 to 1.0). Thus, in an exemplary embodiment, the confidence threshold is not a moving target, as in many other classification systems.

[0074] Although blocks 402-408 are described relatively serially, it should be understood that various blocks of method 400 may be performed in parallel in some embodiments.

[0075] 5, in use, system 100 may perform a method 500 for generating a measure of kurtosis of a probability distribution. It will be understood that certain blocks of method 500 are shown by way of example, and that such blocks may be combined or divided, added or removed, and / or reordered in whole or in part, depending on the particular embodiment, unless stated to the contrary.

[0076] The exemplary method 500 begins at block 502, where the system 100 sorts the probability scores of a probability distribution in descending order. At block 504, the system 100 selects the five largest probabilities (e.g., the first five probabilities in the sorted list of probability scores) as the probability set. It should be understood that in various other embodiments (with corresponding modifications to the methods described herein), a different number of probabilities may be selected. If there are fewer than five probability scores in the probability distribution (or any other defined number of elements in the probability set), the system 100 pads the probability set with one or more small numbers (e.g., epsilon equal to 10 to the power of −6) to eliminate each missing score.

[0077] In block 506, the system 100 determines the ratio of consecutively ranked probabilities (including padding values, if applicable) in the probability set. For example, for a set of five probability scores (x i ), the system 100

[0078]

number

[0079]

number

[0080] where i represents the index and n=5 (any other defined number of elements in the probability set).

[0081] At block 510, the system 100 determines the entropy (e.g., Shannon's entropy) of the probability distribution. In particular, the entropy is given by

[0082]

number

[0083] where i represents the index, n represents the number of probabilities / elements in the probability distribution, and p i denotes the corresponding probability.

[0084] At block 512, the system 100 normalizes the entropy of the probability distribution (i.e., generates a normalized entropy) by dividing the entropy of the probability distribution by the maximum possible entropy of the probability distribution. It should be understood that the maximum possible entropy of a probability distribution is based on the number of elements in the probability distribution, not the values ​​of the elements themselves.

[0085] In block 514, the system 100 applies a sigmoid function to the ratio (from block 506), the kurtosis score (from block 508), and the normalized entropy (from block 512). The sigmoid function may be calculated according to:

[0086]

number

[0087] Thus, the system 100 obtains six output values ​​(e.g., floating-point numbers) after applying the sigmoid function. In block 516, the system 100 uses the six output values ​​as features (e.g., by the confidence classifier 112).

[0088] Although blocks 502-516 are described relatively serially, it should be understood that various blocks of method 500 may be performed in parallel in some embodiments.

[0089] 6, in use, system 100 (e.g., cloud-based system 102) may perform method 600 for determining intent (e.g., in real time) using a trained confidence classifier. It should be understood that certain blocks of method 600 are shown by way of example, and that such blocks may be combined, divided, added, removed, and / or reordered in whole or in part, unless stated to the contrary, depending on the particular embodiment. In some embodiments, method 600 is associated with a user conversation with an automated chatbot, a personal bot, and / or other type of automated conversation system.

[0090] The example method 600 begins at block 602, where the system 100 processes an utterance using the intent classifier 110 to determine a probability distribution of possible intents, and at block 604, the system 100 generates multiple measures of kurtosis of the probability distribution (e.g., using the same measure of kurtosis used to train the confidence classifier 112).

[0091] In block 606, the system 100 applies the trained confidence classifier 112 to determine a single normalized probability of the most likely intent associated with the utterance. For example, the confidence classifier 112 may use multiple measures of kurtosis of the probability distribution as features, in a manner similar to that described above. In block 608, the system 100 compares the normalized probability of the most likely intent to a confidence threshold. In an exemplary embodiment, the confidence threshold is predefined by the system. In some embodiments, the confidence threshold may be modified by a system administrator or other authorized party. As described above, in an exemplary embodiment, by using the confidence classifier 112 and the single normalized probability output by the classifier 112, the system 100 can rely on a single confidence threshold (e.g., 50%, 60%, etc.) rather than a moving target inherent in many other systems.

[0092] At block 610, system 100 determines whether a confidence threshold is met. If so, at block 612, system 100 selects the most likely intent identified by system 100 as the correct intent associated with the utterance. If not, at block 614, system 100 determines that the most likely intent identified by system 100 is an incorrect intent or determines that there is insufficient information to be acceptably reliable in the prediction. Thus, in some embodiments, system 100 may request more information, prompt the user for additional clarification information, and / or otherwise address system 100's inability to properly classify the user's utterance.

[0093] It should be further appreciated that if system 100 determines that the most likely intent identified by system 100 is the correct intent (e.g., in response to a confidence threshold being met), system 100 may perform one or more actions in response to that determination. For example, in embodiments in which system 100 is used with an automated chatbot, system 100 may execute one or more processes associated with the inferred intent, and the automated chatbot may transmit the results of those processes and / or other information associated therewith to user device 108 (e.g., as an automated “response” to a prompt utterance). In other words, in some embodiments, each of the possible intents processed by classifiers 110, 112 may be mapped to or otherwise correspond to one or more processes to be executed in response to a determination that the user's utterance corresponds to that intent.

[0094] Although blocks 602-612 are described relatively serially, it should be understood that various blocks of method 600 may be performed in parallel in some embodiments.

Claims

1. 1. A method of applying a confidence classifier for intent classification associated with an automated chatbot, the method comprising: a computing system processing an utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; the computing system applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; generating a plurality of measures of the kurtosis of the probability distribution, sorting the probability scores of the probability distribution in descending order; selecting a maximum probability subset of the sorted probability scores as a probability set; determining a plurality of ratios of consecutively ranked probabilities in the set of probabilities; determining a kurtosis score for the probabilities in the set of probabilities; determining the entropy of the probability distribution; normalizing the entropy of the probability distribution by dividing it by the maximum possible entropy of the probability distribution to generate a normalized entropy.

2. The method of claim 1 , further comprising the computing system comparing the normalized probability of the most likely intent associated with the utterance to a confidence threshold.

3. In response to the computing system determining that the normalized probability of the most likely intent associated with the utterance satisfies the confidence threshold, selecting the most likely intent as the intent associated with the utterance; 3. The method of claim 2, further comprising: in response to the computing system selecting the most likely intent as the intent associated with the utterance, sending a message to a user device in communication with the automated chatbot, the message being a response to the utterance.

4. 2. The method of claim 1 , wherein generating the plurality of measures of kurtosis of the probability distribution comprises applying a sigmoid function to each of the plurality of ratios, the kurtosis score, and the normalized entropy.

5. 2. The method of claim 1 , wherein selecting the subset of the largest probabilities of the sorted probability scores as the probability set comprises selecting five largest probabilities of the sorted probability scores as the probability set.

6. A method of applying a confidence classifier for intent classification associated with an automated chatbot, the method comprising: a computing system processing an utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; the computing system applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; The method, wherein generating the plurality of measures of kurtosis of the probability distribution comprises determining a kurtosis score of the probability distribution.

7. A method of applying a confidence classifier for intent classification associated with an automated chatbot, the method comprising: a computing system processing an utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; the computing system applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; The method, wherein generating the plurality of measures of kurtosis of the probability distribution comprises determining an entropy of the probability distribution.

8. A method of applying a confidence classifier for intent classification associated with an automated chatbot, the method comprising: a computing system processing an utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; the computing system applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; generating a plurality of measures of the kurtosis of the probability distribution, ranking the probabilities of said probability distributions by maximum probability; and determining a plurality of ratios of successively ranked probabilities in response to ranking the probabilities of the probability distribution.

9. A method of applying a confidence classifier for intent classification associated with an automated chatbot, the method comprising: a computing system processing an utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; the computing system applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; The method further includes the computing system training the confidence classifier by tuning parameters using stochastic gradient descent optimization.

10. 1. A system for applying a confidence classifier for intent classification associated with an automated chatbot, the system comprising: at least one processor; at least one memory, the at least one memory including a plurality of instructions stored therein, the instructions being responsive to execution by the at least one processor to cause the system to: processing the utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; generating a plurality of measures of the kurtosis of the probability distribution, sorting the probability scores of the probability distribution in descending order; selecting a maximum probability subset of the sorted probability scores as a probability set; determining a plurality of ratios of consecutively ranked probabilities in the set of probabilities; determining a kurtosis score for the probabilities in the set of probabilities; determining the entropy of the probability distribution; normalizing the entropy of the probability distribution by dividing it by a maximum possible entropy of the probability distribution to generate a normalized entropy.

11. 11. The system of claim 10, wherein the instructions further cause the system to compare the normalized probability of the most likely intent associated with the utterance to a confidence threshold.

12. The instructions may include: In response to determining that the normalized probability of the most likely intent associated with the utterance satisfies the confidence threshold, selecting the most likely intent as the intent associated with the utterance; 12. The system of claim 11, further comprising: in response to selecting the most likely intent as the intent associated with the utterance, sending a message to a user device in communication with the automated chatbot, the message being a response to the utterance.

13. 11. The system of claim 10, wherein generating the plurality of measures of kurtosis of the probability distribution comprises applying a sigmoid function to each of the plurality of ratios, the kurtosis score, and the normalized entropy.

14. 11. The system of claim 10, wherein selecting the subset of the largest probabilities of the sorted probability scores as the probability set comprises selecting five largest probabilities of the sorted probability scores as the probability set.

15. A system for applying a confidence classifier for intent classification associated with an automated chatbot, the system comprising: at least one processor; at least one memory, the at least one memory including a plurality of instructions stored therein, the instructions being responsive to execution by the at least one processor to cause the system to: processing the utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; The system, wherein generating the plurality of measures of kurtosis of the probability distribution includes determining a kurtosis score of the probability distribution.

16. A system for applying a confidence classifier for intent classification associated with an automated chatbot, the system comprising: at least one processor; at least one memory, the at least one memory including a plurality of instructions stored therein, the instructions being responsive to execution by the at least one processor to cause the system to: processing the utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; The system, wherein generating the plurality of measures of kurtosis of the probability distribution includes determining an entropy of the probability distribution.

17. A system for applying a confidence classifier for intent classification associated with an automated chatbot, the system comprising: at least one processor; at least one memory, the at least one memory including a plurality of instructions stored therein, the instructions being responsive to execution by the at least one processor to cause the system to: processing the utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; generating a plurality of measures of the kurtosis of the probability distribution, ranking the probabilities of said probability distributions by maximum probability; and determining a plurality of ratios of successively ranked probabilities in response to ranking the probabilities of the probability distribution.

18. A system for applying a confidence classifier for intent classification associated with an automated chatbot, the system comprising: at least one processor; at least one memory, the at least one memory including a plurality of instructions stored therein, the instructions being responsive to execution by the at least one processor to cause the system to: processing the utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of measures of kurtosis of the probability distribution; applying a trained confidence classifier to determine a single normalized probability of a most likely intent associated with the utterance based on the multiple measures of kurtosis of the probability distribution; The system, wherein the instructions further cause the system to train the confidence classifier by adjusting parameters using stochastic gradient descent optimization.

Images