Confidence classifier in context of intent classification

Abstract

According to one embodiment, a method of applying a confidence classifier to perform intent classification associated with an automated chatbot includes processing, by a computing system, an utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance, generating, by the computing system, a plurality of kurtosis measurements of the probability distribution, and applying, by the computing system, a trained confidence classifier to determine a single normalized probability of a most probable intent associated with the utterance based on the plurality of kurtosis measurements of the probability distribution.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. non-provisional patent application serial number 17 / 135,114, filed on December 28, 2020, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Intent classification is a key feature of typical Natural Language Understanding (NLU) systems. An intent classifier typically receives transcribed user speech (or text input) and outputs the probability that the user's utterance is associated with a specific predefined intent. For example, a user interface associated with an airline agent might allow four options: book a ticket, cancel a ticket, get flight information, and get the airline's baggage policy. An intent classifier for such an interface would receive the user's text / speech input and analyze it to determine which of the four options the user is most likely to choose. As the number of options increases, determining the true user intent typically becomes more complex. Summary of the Invention

[0004] One implementation relates to unique systems, components, and methods for applying a confidence classifier to classify intents associated with automated chatbots. Other implementations relate to apparatus, systems, devices, hardware, methods, and combinations thereof for applying a confidence classifier to classify intents associated with automated chatbots.

[0005] According to one embodiment, a method for classifying intents associated with an automated chatbot using a confidence classifier may include: a computing system processing a utterance using the intent classifier to determine a probability distribution of possible intents associated with the utterance; the computing system generating multiple kurtosis measurements of the probability distribution; and the computing system applying a trained confidence classifier to determine a single normalized probability of the most likely intent associated with the utterance based on the multiple kurtosis measurements of the probability distribution.

[0006] In some implementations, the method may further include having the computing system compare the normalized probability of the most likely intent associated with the utterance with a confidence threshold.

[0007] In some implementations, the method may further include: the computing system selecting the most probable intent as the intent associated with the utterance in response to determining that the normalized probability of the most probable intent associated with the utterance satisfies the confidence threshold; and the computing system transmitting a message to a user device communicating with the automated chatbot in response to selecting the most probable intent as the intent associated with the utterance, wherein the message is a response to the utterance.

[0008] In some embodiments, generating the plurality of kurtosis measures for the probability distribution can include: ordering probability scores of the probability distribution in descending order; selecting a maximum probability subset of the ordered probability scores as a probability set; determining a plurality of ratios of consecutive ordered probabilities in the probability set; determining a kurtosis score for the probability in the probability set; determining an entropy of the probability distribution; and normalizing the entropy of the probability distribution by dividing the entropy by a maximum possible entropy of the probability distribution to generate a normalized entropy.

[0009] In some embodiments, generating the plurality of kurtosis measures for the probability distribution can include applying a sigmoid function to each of the plurality of ratios, the kurtosis score, and the normalized entropy.

[0010] In some embodiments, selecting the maximum probability subset of the ordered probability scores as the probability set can include selecting five maximum probabilities of the ordered probability scores as the probability set.

[0011] In some embodiments, generating the plurality of kurtosis measures for the probability distribution can include determining a kurtosis score for the probability distribution.

[0012] In some embodiments, generating the plurality of kurtosis measures for the probability distribution can include determining an entropy of the probability distribution.

[0013] In some embodiments, generating the plurality of kurtosis measures for the probability distribution can include ordering probabilities of the probability distribution by maximum probability, and determining a plurality of ratios of consecutive ordered probabilities in response to ordering the probabilities of the probability distribution.

[0014] In some embodiments, the method can further include training, by the computing system, the confidence classifier by adjusting parameters using stochastic gradient descent optimization.

[0015] According to another embodiment, a system for applying a confidence classifier for intent classification associated with an automated chatbot can include at least one processor and at least one memory including a plurality of instructions stored thereon that, in response to execution by the at least one processor, cause the system to: process, with an intent classifier, an utterance to determine a probability distribution of possible intents associated with the utterance; generate a plurality of kurtosis measures for the probability distribution; and apply a trained confidence classifier to determine, based on the plurality of kurtosis measures for the probability distribution, a single normalized probability of a most likely intent associated with the utterance.

[0016] In some embodiments, the plurality of instructions can further cause the system to compare the normalized probability of the most likely intent associated with the utterance to a confidence threshold.

[0017] In some embodiments, the plurality of instructions can further cause the system to select the most likely intent as an intent associated with the utterance in response to determining that the normalized probability of the most likely intent associated with the utterance satisfies the confidence threshold, and transmit a message to a user device in communication with the automated chat bot in response to selecting the most likely intent as the intent associated with the utterance, wherein the message is a response to the utterance.

[0018] In some embodiments, generating the plurality of kurtosis measures of the probability distribution can include sorting probability scores of the probability distribution in descending order, selecting a maximum probability subset of the sorted probability scores as a probability set, determining a plurality of ratios of consecutive sorted probabilities in the probability set, determining a kurtosis score of the probability, determining an entropy of the probability distribution, and normalizing the entropy of the probability distribution by dividing the entropy by a maximum possible entropy of the probability distribution to generate a normalized entropy.

[0019] In some embodiments, generating the plurality of kurtosis measures of the probability distribution can include applying a sigmoid function to each of the plurality of ratios, the kurtosis score, and the normalized entropy.

[0020] In some embodiments, selecting the maximum probability subset of the sorted probability scores as the probability set can include selecting five maximum probabilities of the sorted probability scores as the probability set.

[0021] In some embodiments, generating the plurality of kurtosis measures of the probability distribution can include determining a kurtosis score of the probability distribution.

[0022] In some embodiments, generating the plurality of kurtosis measures of the probability distribution can include determining an entropy of the probability distribution.

[0023] In some embodiments, generating the plurality of kurtosis measures of the probability distribution can include sorting probabilities of the probability distribution by maximum probability, and determining a plurality of ratios of consecutive sorted probabilities in response to sorting the probabilities of the probability distribution.

[0024] In some embodiments, the plurality of instructions can further cause the system to train the confidence classifier by adjusting parameters using stochastic gradient descent optimization.

[0025] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter. Other embodiments, forms, features, and aspects of the application will become apparent to those skilled in the art upon consideration of the description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] The concepts described herein are illustrated by way of example in the accompanying drawings and are not limited to the specific arrangements and instrumentalities shown. The elements illustrated in the figures are not necessarily to scale, for the sake of simplicity and clarity. Where considered appropriate, reference numerals have been repeated among the figures to indicate corresponding or analogous elements.

[0027] Figure 1 is a simplified block diagram of at least one embodiment of a system for applying a confidence classifier for intent classification;

[0028] Figure 2 is a simplified block diagram of at least one embodiment of a cloud-based system;

[0029] Figure 3 is a simplified block diagram of at least one embodiment of a computing system;

[0030] Figure 4 is a simplified flow diagram of at least one embodiment of a method for training a confidence classifier;

[0031] Figure 5 is a simplified flow diagram of at least one embodiment of a method for generating a kurtosis measure of a probability distribution; and

[0032] Figure 6 is a simplified flow diagram of at least one embodiment of a method for using a trained confidence classifier to determine intent. DETAILED DESCRIPTION

[0033] While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, it should be understood that the concepts of the present disclosure are not intended to be limited to the particular arrangements and instrumentalities shown, but are to be accorded the widest scope consistent with the present disclosure and the appended claims.

[0034] References in the specification to “one embodiment,” “an embodiment,” “exemplary embodiment,” etc., indicate that the embodiment described can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily indicative of a reference to the same embodiment. It will also be appreciated that, where the description refers to a “preferred” element or embodiment, no preference is made between this element or embodiment and other elements or embodiments. Furthermore, where a particular feature, structure, or characteristic is described in connection with an embodiment, it will be understood that every embodiment can include this particular feature, structure, or characteristic, whether or not it is described in every embodiment. Additionally, it will be understood that the items recited in the list of items that are “at least one of’ a list of items refers to any single one of the listed items individually, as well as to any combination of two or more of the listed items. Similarly, the list of items recited in the list of items that are “at least one of’ a list of items refers to any single one of the listed items individually, as well as to any combination of two or more of the listed items. Furthermore, the use of the terms “a,” “an,” “the,” etc. to refer to a feature or structure does not foreclose the possibility that other feature(s) or structure(s) can be utilized in addition to, or instead of, the feature or structure that is described. Also, as used herein to describe various embodiments, the following terms have the following meanings unless indicated otherwise.

[0035] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software, or combinations thereof. The disclosed embodiments can also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine- readable (e.g., computer-readable) storage media that can be read and executed by one or more processors. Machine-readable storage media can be embodied as any storage device, mechanism, or physical structure for

[0036] In the drawings, some of the structural or methodological acts can be shown in a particular arrangement and / or order. It should be understood, however, that such is not intended to imply that there is only one manner or order in which the acts can be performed. On the contrary, it is contemplated that the acts can be performed in a different manner or order, unless specified explicitly otherwise. Additionally, the inclusion of structural or methodological acts in a particular figure does not imply that each of the acts is needed to practice the embodiments, nor does the exclusion of an act imply that it is unnecessary, unless explicitly stated. Furthermore, it should be understood that the structural or methodological acts can be combined, separated, or reordered, unless explicitly stated otherwise.

[0037] Referring nowFigure 1 In an illustrative embodiment, a system 100 for applying a confidence classifier for intent classification includes a cloud-based system 102, a network 104, a contact center system 106, and a user device 108. Additionally, the illustrative cloud-based system 102 includes an intent classifier 110 and a confidence classifier 112. Although only one cloud-based system 102, one network 104, one contact center system 106, one user device 108, one intent classifier 110, and one confidence classifier 112 are shown in the illustrative embodiment, in other embodiments, the system 100 can include multiple cloud-based systems 102, networks 104, contact center systems 106, user devices 108, intent classifiers 110, and / or confidence classifiers 112. For example, in some embodiments, multiple cloud-based systems 102 (e.g., related or unrelated systems) can be used to perform the various functions described herein. Moreover, in some embodiments, one or more of the systems described herein can be excluded from the system 100, one or more systems described as being independent can form part of another system, and / or one or more systems described as forming part of another system can be independent. Figure 1

[0038] In some systems, an NLU author / administrator can train an intent classifier using a supervised learning approach by providing a set of utterances mapped to intent labels to the intent classifier. These utterances can be converted into dense vectors (e.g., arrays of floating point numbers) that can be processed prior to optimization of the parameters of the intent classifier (e.g., via a training process). Once complete, when a user provides an input text utterance, the trained model can be utilized and the input text utterance is then processed using the same feature generator used during the classifier training process to enable the intent classifier to predict a set of probabilities for possible intents. The input can come from, for example, a dialog management system, a chatbot, a personal robot, or other system, and the classifier can provide the probabilities to the system so that it can determine the best course of action in response.

[0039] ​For systems that depend on the output of an intent classifier, the range of output probabilities in the range of 0.0 to 1.0 can result in considerable ambiguity. For example, assume that an intent classifier has been trained to recognize three intents, and if the highest probability is 0.7 (e.g., a confidence threshold), then a downstream consuming system has calibrated a certain response. If each of the possible intents is equally probable, then the probability of each intent would be approximately 0.33. However, if the number of possible intents increases to four, then the probability of each equally probable case would be 0.25 instead of 0.33. As the number of possible intents increases, the probability of each intent continues to decrease. This significantly reduces the individual probabilities, and thus, the highest probability is typically lower. Moreover, different numbers of intents can make setting / defining a confidence threshold a moving target.

[0040] The system 100 and techniques described herein allow for improved confidence classification in the context of intent classification (e.g., based on a user’s communication with a chatbot or personal bot), which addresses the problem of lower probabilities associated with a large number of possible intents. As described herein, this output probability distribution can be used to essentially measure the confidence of an intent classifier in predicting the mapping of an input utterance to an intent based on a kurtosis measure of the probability distribution. A confidence classifier is trained, and then used in real-time to convert the output probability from the intent classifier to a normalized output value in the range of 0.0 to 1.0 that acts as a proxy for the output probability of the best possible intent probability of the intent classifier. Thus, a confidence threshold can also be set and maintained in the range of 0.0 to 1.0, such that there is no longer a need to change the threshold based on the number of possible intent options.

[0041] It should be appreciated that each of the cloud-based system 102, the network 104, the contact center system 106, the user device 108, the intent classifier 110, and the confidence classifier 112 can be embodied as any type of device / system, collection of devices / systems, or portions thereof, suitable to perform the functions described herein.

[0042] The cloud-based system 102 can be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in illustrative embodiments, the cloud-based system 102 is configured to receive user input data indicative of a user utterance and process the user input data (e.g., using the intent classifier 110 and the confidence classifier 112) to determine a most likely user intent associated with the utterance and a corresponding confidence in the system prediction that can be used (e.g., by other systems) to make a decision. As described herein, the intent classifier 110 of the cloud-based system 102 is configured to process user input data associated with a user utterance and output a corresponding probability that the user utterance corresponds to each of the possible predefined intents analyzed by the intent classifier 110. The confidence classifier 112 further analyzes the probability distribution output by the intent classifier 110 to generate a plurality of kurtosis measures (e.g., ratios, kurtosis scores, entropies, etc.) of the probability distribution and determine a single normalized probability of the most likely intent associated with the utterance based on those kurtosis measures.

[0043] Although the cloud-based system 102 is described herein in the singular, it should be understood that, in some embodiments, the cloud-based system 102 can be embodied as or include multiple servers / systems. Moreover, although the cloud-based system 102 is described herein as a cloud-based system, it should be understood that, in other embodiments, the system 102 can be embodied as one or more servers / systems located outside of a cloud computing environment. In some embodiments, the cloud-based system 102 can be embodied as or similar to the cloud-based system 200 described with reference to Figure 2 the cloud-based system 200 described with reference to

[0044] In cloud-based embodiments, the cloud-based system 102 can be embodied as a serverless computing solution that, for example, executes a plurality of instructions on-demand, contains logic to execute instructions only when prompted by a particular activity / trigger, and does not consume computing resources when not in use. That is, the system 102 can be embodied as a virtual computing environment (e.g., a distributed network of devices) that resides “on” a computing system, where various virtual functions (e.g., Lambda functions, Azure functions, Google Cloud functions, and / or other suitable virtual functions) can correspond to functions of the system 102 described herein to execute. For example, when an event occurs (e.g., data is transmitted to the system 102 for processing), a communication can be made to the virtual computing environment (e.g., via a request to an API of the virtual computing environment), whereby the API can route the request to the correct virtual function (e.g., a particular serverless computing resource) based on a set of rules. Thus, when a user makes a request for data transmission (e.g., via a suitable user interface to the system 102), the appropriate virtual function can be executed to perform an action before an instance of the virtual function is eliminated.

[0045] The network 104 can be embodied as any one or more types of communication networks capable of facilitating communication between various devices communicatively connected via the network 104. Thus, the network 104 can include one or more networks, routers, switches, access points, hubs, computers, and / or other intermediary network devices. For example, the network 104 can be embodied as, or otherwise include, one or more cellular networks, telephone networks, local or wide area networks, publicly available global networks (e.g., the Internet), ad hoc networks, short-range communication links, or combinations thereof. In some embodiments, the network 104 can include circuit-switched voice or data networks, packet-switched voice or data networks, and / or any other networks capable of carrying voice and / or data. In particular, in some embodiments, the network 104 can include Internet Protocol (IP)-based and / or Asynchronous Transfer Mode (ATM)-based networks. In some embodiments, the network 104 can handle voice traffic (e.g., via Voice over IP (VOIP) networks), web traffic (e.g., such as Hypertext Transfer Protocol (HTTP) traffic and Hypertext Markup Language (HTML) traffic), and / or other network traffic depending on the particular embodiments and / or devices of the system 100 in communication with one another. In various embodiments, the network 104 can include analog or digital wired and wireless networks (e.g., IEEE 802.11 networks, Public Switched Telephone Networks (PSTNs), Integrated Services Digital Networks (ISDNs), and digital subscriber lines (xDSL)), third generation (3G) mobile telecommunications networks, fourth generation (4G) mobile telecommunications networks, fifth generation (5G) mobile telecommunications networks, wired Ethernet networks, private networks (e.g., such as intranets), radio, television, cable, satellite, and / or any other delivery or tunneling mechanism for carrying data, or any suitable combination of such networks. The network 104 can enable connectivity between the various devices / systems 102, 106, 108, 110, and 112 of the system 100. It should be understood that the various devices / systems 102, 106, 108, 110, and 112 can communicate with one another via different networks 104 depending on the source and / or destination devices / systems 102, 106, 108, 110, and 112.

[0046] In some embodiments, it should be understood that the cloud-based system 102 can be communicatively coupled to, form a part of, and / or otherwise used with the contact center system 106. For example, the contact center system 106 can include a chatbot (e.g., similar to the chatbot 218 of Figure 2 In some embodiments, it should be understood that the cloud-based system 102 can be communicatively coupled to, form a part of, and / or otherwise used with the contact center system 106. For example, the contact center system 106 can include a chatbot (e.g., similar to the chatbot 218 of

[0047] The contact center system 106 can be embodied as any system capable of providing contact center services (e.g., call center services) to end users and otherwise performing the functionality described herein. Depending on the particular implementation, it should be appreciated that the contact center system 106 can be located at the premises / campus of an organization that utilizes the contact center system 106 and / or remotely located with respect to the organization (e.g., in a cloud-based computing environment). In some embodiments, a portion of the contact center system 106 can be located at the premises / campus of an organization while other portions of the contact center system 106 are remotely located with respect to the premises / campus of the organization. Thus, it should be appreciated that the contact center system 106 can be deployed in equipment dedicated to an organization or a third-party service provider thereof and / or in a remote computing environment such as, for example, a private or public cloud environment having infrastructure for supporting multiple contact centers for multiple enterprises. In some embodiments, the contact center system 106 includes resources (e.g., personnel, computers, and telecommunication equipment) to enable delivery of services via telephone and / or other communication mechanisms. Depending on the particular type of contact center, such services can include, for example, technical support, help desk support, emergency response, and / or other contact center services.

[0048] The user device 108 can be embodied as any type of device capable of executing an application and otherwise performing the functionality described herein. For example, in some embodiments, the user device 108 is configured to execute an application to engage in a conversation with a personal robot, automated agent, chatbot, or other automated system. As such, the user device 108 can have various input / output devices with which a user can interact to provide and receive audio, text, video, and / or other forms of data. It should be appreciated that the application can be embodied as any type of application suitable to perform the functionality described herein. In particular, in some embodiments, the application can be embodied as a mobile application (e.g., a smartphone application), a cloud-based application, a web application, a thin-client application, and / or another type of application. For example, in some embodiments, the application can be used as a client-side interface to a web-based application or service (e.g., via a web browser).

[0049] It should be appreciated that each of the cloud-based system 102, the network 104, the contact center system 106, and / or the user device 108 can be embodied as (and / or include) a similar system as described below with reference to FIG. 2. Figure 3The computing device 300 is described as one or more computing devices. For example, in illustrative embodiments, each of the cloud-based system 102, the network 104, the contact center system 106, and / or the user device 108 can include a processing device 302 and a memory 306 having stored thereon operational logic 308 (e.g., a plurality of instructions) for execution by the processing device 302 to operate the corresponding device.

[0050] Referring now to Figure 2 , a simplified block diagram of at least one embodiment of a cloud-based system 200 is shown. The illustrative cloud-based system 200 includes a border communication device 202, a SIP server 204, a resource manager 206, a media control platform 208, a speech / text analysis system 210, a speech generator 212, a speech gateway 214, a media enhancement system 216, a chatbot 218, and a speech data store 220. While only one border communication device 202, one SIP server 204, one resource manager 206, one media control platform 208, one speech / text analysis system 210, one speech generator 212, one speech gateway 214, one media enhancement system 216, one chatbot 218, and one speech data store 220 are shown in the illustrative embodiment of Figure 2 , in other embodiments, the cloud-based system 200 can include multiple border communication devices 202, SIP servers 204, resource managers 206, media control platforms 208, speech / text analysis systems 210, speech generators 212, speech gateways 214, media enhancement systems 216, chatbots 218, and / or speech data stores 220. For example, in some embodiments, multiple chatbots 218 can be used to communicate on different topics handled by the same cloud-based system 200. Further, in some embodiments, one or more components described herein can be excluded from the system 200, one or more components described as being independent can form part of another component, and / or one or more components described as forming part of another component can be independent.

[0051] The border communication device 202 can be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in some embodiments, the border communication device 202 can be configured to control signaling and media streams that relate to establishing, conducting, and discontinuing voice sessions and other media communications between, for example, end users and contact center systems. In some embodiments, the border communication device 202 can be a session border controller (SBC) that controls signaling and media exchanged during media sessions (also referred to as “calls,” “telephone calls,” or “communication sessions”) between end users and contact center systems. In some embodiments, the signaling exchanged during media sessions can include SIP, H.323, Media Gateway Control Protocol (MGCP), and / or any other Voice over IP (VoIP) call signaling protocol. The media exchanged during media sessions can include media streams that carry audio, video, or other data for a call, as well as information for call statistics and quality.

[0052] In some embodiments, the border communication device 202 can operate in accordance with a standard SIP Back-to-Back User Agent (B2BUA) configuration. In this regard, the border communication device 202 can be inserted in the signaling and media path established between a calling party and a called party in a VoIP call. In some embodiments, it should be understood that other intermediary software and / or hardware devices can be invoked in establishing the signaling and / or media path between the calling party and the called party.

[0053] In some embodiments, the border communication device 202 can exert control over signaling (e.g., SIP messages) and media streams (e.g., RTP data) that traverse a network (e.g., network 104) to and from end user devices (e.g., user devices 108) and contact center systems (e.g., contact center system 106). In this regard, the border communication device 202 can be coupled to a trunk that carries signals and media for calls to and from user devices on a network, as well as a trunk that carries signals and media to and from contact center systems on a network.

[0054] The SIP server 204 can be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in some embodiments, the SIP server 204 can act as a SIP B2UBA and can control the flow of SIP requests and responses between SIP endpoints. In other embodiments, in addition to or instead of the SIP server 204, any other controller configured to establish and terminate VoIP communication sessions can be contemplated. The SIP server 204 can be a separate logical component or can be combined with the resource manager 206. In some embodiments, the SIP server 204 can be hosted at a contact center system (e.g., the contact center system 106). Although a SIP server 204 is used in the illustrative embodiments, in addition to or instead of SIP, another call server configured with another VoIP protocol such as, for example, H.232 protocol, Media Gateway Control Protocol, Skype protocol, and / or other suitable technology can be used.

[0055] The resource manager 206 can be embodied as any one or more types of devices / systems capable of performing the functions described herein. In illustrative embodiments, the resource manager 206 can be configured to allocate and monitor a pool of media control platforms for providing load balancing and high availability for each resource type. In some embodiments, the resource manager 206 can monitor and can select a media control platform 208 from a cluster of available platforms. The selection of the media control platform 208 can be dynamic, for example, based on an identification of a location of a calling end user, a type of media service to be rendered, a detected quality of a current media service, and / or other factors.

[0056] In some embodiments, the resource manager 206 can be configured to process requests for media services and to interact with, for example, a configuration server having a configuration database to determine interactive voice response (IVR) profiles, voice applications (e.g., voice extensible markup language (VoiceXML) applications), announcement and conference applications, resources, and service profiles that can deliver services such as, for example, media control platforms. According to some embodiments, the resource manager can provide a tiered multi-tenant configuration for service providers such that the service providers can allocate a selected number of resources for each tenant.

[0057] In some embodiments, the resource manager 206 can be configured to act as a SIP proxy, a SIP registrar, and / or a SIP notifier. In this regard, the resource manager 206 can act as a proxy for SIP traffic between two SIP components. As a SIP registrar, the resource manager 206 can accept registration of various resources via, for example, SIP REGISTER messages. In this way, the cloud-based system 200 can support transparent relocation of call processing components. In some embodiments, components such as the media control platform 208 do not register with the resource manager 206 upon startup. The resource manager 206 can detect instances of the media control platform 208 through configuration information retrieved from a configuration database. If the media control platform 208 has been configured for monitoring, the resource manager 206 can monitor resource health by using, for example, SIP OPTIONS messages. In some embodiments, to determine whether resources in a group are valid, the resource manager 206 can periodically send SIP OPTIONS messages to each media control platform 208 resource in a group. If the resource manager 206 receives an OK response, the resource is considered valid. It should be appreciated that the resource manager 206 can be configured to perform other various functions, which have been omitted for the sake of brevity of the description. The resource manager 206 and the media control platform 208 can be collectively referred to as a media controller.

[0058] In some embodiments, the resource manager 206 can act as a SIP notifier by accepting SIP SUBSCRIBE requests from, for example, the SIP server 204 and maintaining multiple independent subscriptions to the same or different SIP devices. The subscription notifications are for tenants managed by the resource manager 206. In this role, the resource manager 206 can periodically generate SIP NOTIFY requests to subscribers (or tenants) regarding port usage and the number of available ports. The resource manager 206 can support multi-tenancy by sending notifications containing the tenant name, the current status (in-service or out-of-service) of the media control platform 208 associated with the tenant, and the current capabilities of the tenant.

[0059] The media control platform 208 can be embodied as any service or system capable of providing media services and otherwise performing the functions described herein. For example, in some embodiments, the media control platform 208 can be configured to provide call and media services upon request by a service user. Such services can include, but are not limited to, initiating outbound calls, playing music or providing other media while a call is on hold, call recording, conferencing, call progress detection, playing audio / video prompts during a customer self-service session, and / or other call and media services. One or more services can be defined by a voice application (e.g., a VoiceXML application) that is executed as part of the process of establishing a media session between the media control platform 208 and an end user.

[0060] The speech / text analysis system (STAS) 210 can be embodied as any service or system capable of providing various speech analysis and text processing functions (e.g., text-to-speech) as will be appreciated by those skilled in the art and otherwise performing the functions described herein. The speech / text analysis system 210 can perform automatic speech and / or text recognition and grammar matching for end user communication sessions processed by the cloud-based system 200. The speech / text analysis system 210 can include one or more processors and instructions stored in a machine-readable medium that are executed by the processors to perform various operations. In some embodiments, the machine-readable medium can include non-transitory storage media such as hard disks and hardware memory systems.

[0061] The speech generator 212 can be embodied as any service or system capable of generating speech communications and otherwise performing the functions described herein. In some embodiments, the speech generator 212 can generate speech communications based on a particular speech signature.

[0062] The speech gateway 214 can be embodied as any service or system capable of performing the functions described herein. In an illustrative embodiment, the speech gateway 214 receives end user calls from or to speech communication devices (such as end user devices) and responds to the calls according to a speech program corresponding to a communication routing configuration of a contact center system. In some embodiments, the speech program can include a speech avatar. The speech program can be accessed from local memory within the speech gateway 214 or from other storage media within the cloud-based system 200. In some embodiments, the speech gateway 214 can process the speech program as a scripted speech application. Thus, the speech program can be a script written in a scripting language such as Voice Extensible Markup Language (VoiceXML) or Speech Application Language Tags (SALT). The cloud-based system 200 can also communicate with a speech data store 220 to read and / or write user interaction data (e.g., state variables of a data communication session) in a shared memory space.

[0063] The media augmentation system 216 can be embodied as any service or system capable of specifying how the various parts of the cloud-based system 200 (e.g., the border communication device 202, the SIP server 204, the resource manager 206, the media control platform 208, the speech / text analysis system 210, the speech generator 212, the speech gateway 214, the media augmentation system 216, the chatbot 218, the speech data store 220, and / or one or more of portions thereof) interact with one another and otherwise perform the functions described herein. In some embodiments, the media augmentation system 216 can be embodied as or include an application program interface (API). In some embodiments, the media augmentation system 216 can be capable of integrating different parameters and / or protocols for use with various scheduled applications and media types utilized within the cloud-based system 200.

[0064] The chatbot 218 can be embodied as any automated service or system capable of using automation in conjunction with and otherwise performing the functions described herein. For example, in some embodiments, the chatbot 218 can operate, for example, as an executable program that can be launched as needed by a particular chatbot. In some embodiments, the chatbot 218 simulates and processes human conversations (written or spoken), allowing humans to interact with digital devices as if a person is communicating with another person. In some embodiments, the chatbot 218 can be as simple as a basic program that answers simple queries through single-line responses, or as complex as a digital assistant that learns and evolves to deliver an ever-improving level of personalization as it gathers and processes information. In some embodiments, the chatbot 218 includes and / or utilizes artificial intelligence, adaptive learning, robotics, cognitive computing, and / or other automation technologies. The chatbot 218 can also be referred to herein as one or more chatbots, AI chatbots, automated chatbots, chatterbots, dialog systems, conversational agents, automated chat resources, and / or bots.

[0065] A benefit of utilizing an automated chatbot for chat sessions with end users can be that it helps make contact centers more efficient in their use of valuable and expensive resources, such as human resources, while maintaining end user satisfaction. For example, a chatbot can be invoked to initially handle a chat session with a human end user without the human end user knowing that they are in a session with a bot. If and when appropriate, the chat session can be escalated to a human resource. Thus, human resources need not be unnecessarily used to handle simple requests, but can be more efficiently used to handle more complex requests or to monitor the progress of many different automated communications at the same time.

[0066] The voice data storage 220 can be embodied as one or more databases, data structures, and / or data storage devices that are capable of storing data in the cloud-based system 200 or otherwise facilitating the storage of such data by the cloud-based system 200. For example, in some embodiments, the voice data storage 220 can comprise one or more cloud storage buckets. In other embodiments, it will be appreciated that the voice data storage 220 can additionally or alternatively comprise other types of voice data storage mechanisms that allow for dynamic scaling of the amount of data storage available for use by the cloud-based system 200. In some embodiments, the voice data storage 220 can store scripts (e.g., pre-scripted scripts or otherwise). Although the voice data storage 220 is described herein as a data storage and a database, it will be appreciated that the voice data storage 220 can comprise a database (or other type of organized collection of data and structures) and data storage devices for the actual storage of underlying data. The voice data storage 220 can store various data useful for performing the functions described herein.

[0067] Referring now to Figure 3 , a simplified block diagram of at least one embodiment of a computing device 300 is shown. The illustrative computing device 300 depicts at least one embodiment of a computing device that can be utilized in conjunction with the cloud-based system 102, contact center system 106, and / or user device 108 (and / or portions thereof) shown in Figure 1 FIG. 1. Moreover, in some embodiments, one or more of the border communication device 202, SIP server 204, resource manager 206, media control platform 208, voice / text analysis system 210, voice generator 212, voice gateway 214, media enrichment system 216, chatbot 218, and / or voice data storage 220 (and / or portions thereof) can be embodied as or executed by a computing device similar to the computing device 300. Depending on the particular embodiment, the computing device 300 can be embodied as a server, a desktop computer, a laptop computer, a tablet computer, a notebook, a netbook, an Ultrabook™ computer, a cellular telephone, a mobile computing device, a smartphone, a wearable computing device, a personal digital assistant, an Internet of Things (IoT) device, a processing system, a wireless access point, a router, a gateway, and / or any other computing device, processing device, and / or communication device capable of performing the functions described herein. TM

[0068] ​The computing device 300 includes a processing device 302 that executes algorithms and / or processes data according to operating logic 308, an input / output device 304 that enables communication between the computing device 300 and one or more external devices 310, and a memory 306 that stores, for example, data received from the external devices 310 via the input / output device 304.

[0069] Input / output device 304 allows computing device 300 to communicate with external device 310. For example, input / output device 304 may include a transceiver, network adapter, network interface card, interface, one or more communication ports (e.g., USB port, serial port, parallel port, analog port, digital port, VGA, DVI, HDMI, FireWire, CAT 5, or any other type of communication port or interface) and / or other communication circuitry. Depending on the specific computing device 300, the communication circuitry of computing device 300 may be configured to use any one or more communication technologies (e.g., wireless or wired communication) and associated protocols (e.g., Ethernet, etc.). Wi- This type of communication can be achieved using technologies such as WiMAX. Input / output device 304 may include hardware, software, and / or firmware suitable for performing the technologies described herein.

[0070] External device 310 can be any type of device that allows data to be input or output from computing device 300. For example, in various embodiments, external device 310 can be embodied as cloud-based system 102, contact center system 106, user equipment 108, and / or a portion thereof. Furthermore, in some embodiments, external device 310 can be embodied as another computing device, switch, diagnostic tool, controller, printer, monitor, alarm, peripheral device (e.g., keyboard, mouse, touchscreen display, etc.), and / or any other computing device, processing device, and / or communication device capable of performing the functions described herein. Moreover, in some embodiments, it should be understood that external device 310 can be integrated into computing device 300.

[0071] The processing device 302 can be embodied as any type of processor capable of performing the functions described herein. In particular, the processing device 302 can be embodied as one or more single core or multi-core processors, microcontrollers, or other processors or processing / control circuitry. For example, in some embodiments, the processing device 302 can include or be embodied as an arithmetic logic unit (ALU), central processing unit (CPU), digital signal processor (DSP), and / or another suitable processor. The processing device 302 can be of a programmable type, a dedicated hardwired state machine, or a combination thereof. In various embodiments, the processing device 302 having multiple processing units can utilize distributed, pipelined, and / or parallel processing. Further, the processing device 302 can be dedicated to performing only the operations described herein, or can be utilized in one or more additional applications. In an illustrative embodiment, the processing device 302 is programmable and executes algorithms and / or processes data in accordance with operational logic 308, as defined by programming instructions (such as software or firmware) stored in memory 306. Additionally or alternatively, the operational logic 308 for the processing device 302 can be defined at least in part by hardwired logic components or other hardware. Further, the processing device 302 can include any type of component or components suitable for processing signals received from the input / output devices 304 or from other components or devices, and providing desired output signals. Such components can include digital circuitry, analog circuitry, or a combination thereof.

[0072] The memory 306 can be one or more types of non-transitory computer-readable media, such as solid state memory, electromagnetic memory, optical memory, or a combination thereof. Further, the memory 306 can be volatile and / or non-volatile, and in some embodiments, some or all of the memory 306 can be of a portable type, such as a disk, tape, memory stick, cartridge, and / or other suitable portable memory. In operation, the memory 306 can store various data and software used during operation of the computing device 300, such as operating systems, applications, programs, libraries, and drivers. It will be appreciated that, in addition to or in lieu of storing programming instructions defining the operational logic 308, the memory 306 can store data manipulated by the operational logic 308 of the processing device 302, such as, for example, data representing signals received from and / or transmitted to the input / output devices 304. As Figure 3 As shown, the memory 306 can be included with and / or coupled to the processing device 302, depending on the particular embodiment. For example, in some embodiments, the processing device 302, memory 306, and / or other components of the computing device 300 can form part of a system on a chip (SoC) and be incorporated on a single integrated circuit chip.

[0073] In some embodiments, the various components of computing device 300 (e.g., processing device 302 and memory 306) can be communicatively coupled via an input / output subsystem, which can be embodied as circuitry and / or components to facilitate input / output operations with the processing device 302, memory 306, and other components of the computing device 300. For example, the input / output subsystem can be embodied as, or otherwise include, memory controller hubs, input / output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems, to facilitate the input / output operations.

[0074] In other embodiments, the computing device 300 can include other or additional components, such as those commonly found in a typical computing device (e.g., various input / output devices and / or other components). It should be understood that the computing device 300 described herein can be a special purpose computing device that has been configured to perform one or more specific Figure 3 In other embodiments, the computing device 300 can include other or additional components, such as those commonly found in a typical computing device (e.g., various input / output devices and / or other components). It should be understood that the computing device 300 described herein can be a special purpose computing device that has been configured to perform one or more specific

[0075] Referring now to Figure 4 In use, the system 100 (e.g., the cloud-based system 102) can perform a method 400 for training a confidence classifier. It should be understood that the particular blocks of the method 400 are shown by way of example, and such blocks can be combined or divided, added or removed, and / or reordered in whole or in part, according to particular embodiments. In some embodiments, the method 400 is associated with a user’s conversation with an automated chatbot, personal robot, and / or other type of automated conversational system.

[0076] The illustrative method 400 begins at block 402, where the system 100 (or more specifically, the confidence classifier 112) receives a probability distribution of possible intents associated with an utterance. In other words, the probability distribution includes a corresponding probability that the user’s utterance corresponds to each of the possible intents processed by the intent classifier 110 described herein. In some embodiments, it should be understood that the system 100 can also process the user input corresponding to the utterance as described above (e.g., via the intent classifier 110) in order to arrive at the probability distribution, while in other embodiments, the system 100 receives the probability distribution from another system (e.g., as the corresponding probabilities).

[0077] In block 404, the system 100 generates a plurality of kurtosis measures of the probability distribution. In some embodiments, to do so, the system 100 can perform the method 500 described in detail below, which utilizes three different kurtosis measures of the probability distribution. However, it should be appreciated that a different number of kurtosis measures and / or different specific kurtosis measures of the probability distribution can be used in other embodiments. Figure 5

[0078] In block 406, the system 100 trains the confidence classifier 112 using the plurality of kurtosis measures as classification features. In an illustrative embodiment, the confidence classifier 112 outputs a single normalized probability of the most likely intent associated with an utterance based on the plurality of kurtosis measures of the probability distribution. To train the confidence classifier 112, positive examples representing correct pairings of utterances Ui and intent labels Li can be provided to the intent classifier 110, and a feature set (e.g., the plurality of kurtosis measures) can be extracted and fed into the confidence classifier 112, with the goal of the confidence classifier 112 learning being 1.0. Likewise, negative examples representing incorrect pairings of utterances with utterances randomly selected from another class / label can be provided to the intent classifier 110, and a feature set (e.g., the plurality of kurtosis measures) can be extracted and fed into the confidence classifier 112, with the goal of the confidence classifier 112 learning being 0.0. This process can be repeated for multiple iterations (e.g., until all or a plurality of possibilities are exhausted) in order to train the confidence classifier 112.

[0079] In some embodiments, in block 408, the system 100 can adjust the parameters of the confidence classifier 112 using stochastic gradient descent optimization. In other embodiments, the system 100 can train the confidence classifier 112 using the plurality of kurtosis measures as classification features with one or more other algorithms.

[0080] After having been trained, it should be appreciated that the system 100 (or another system) can use the single number output by the confidence classifier 112 (e.g., in the range of 0.0 to 1.0) to classify real-time intent data (associated with user utterances) using a single confidence threshold (e.g., in the range of 0.0 to 1.0). Thus, in an illustrative embodiment, the confidence threshold is not a moving target as in many other classification systems.

[0081] Although blocks 402-408 are described in a relatively serial fashion, it should be appreciated that in some embodiments, the various blocks of the method 400 can be performed in parallel.

[0082] Referring now to Figure 5 ​In use, the system 100 can perform a method 500 for generating a kurtosis measure of a probability distribution. It should be appreciated that particular blocks of the method 500 are shown by way of example, and such blocks can be combined or divided, added or removed, and / or reordered in whole or in part, according to particular embodiments, unless otherwise specified.

[0083] The illustrative method 500 begins at block 502, where the system 100 sorts the probability scores of the probability distribution in descending order. In block 504, the system 100 selects the five largest probabilities as a probability set (e.g., the top five probabilities in the sorted list of probability scores). It should be appreciated that a different number of probabilities can be selected in various other embodiments (with corresponding modifications to the methods described herein). If there are fewer than five probability scores in the probability distribution (or other defined number of elements in the probability set), the system 100 pads the probability set with one or more small numbers (e.g., epsilon equal to 10 to the -6 power) to make up for each missing score.

[0084] In block 506, the system 100 determines the ratio of the consecutively ordered probabilities in the probability set (including the padded values, if applicable). For example, for a set of five probability scores (x i ), the system 100 calculates:

[0085]

[0086] for all i from 1 to 4, where i represents the index. In block 508, the system 100 determines a kurtosis score for the probabilities in the probability set. In particular, the kurtosis score can be calculated according to the following equation:

[0087]

[0088] where i represents the index and n = 5 (or other defined number of elements in the probability set).

[0089] In block 510, the system 100 determines the entropy (e.g., Shannon entropy) of the probability distribution. In particular, the entropy can be calculated according to the following equation:

[0090]

[0091] where i represents the index, n represents the number of probabilities / elements in the probability distribution, and p i represents the corresponding probability.

[0092] In block 512, the system 100 normalizes the entropy of the probability distribution by dividing the entropy by the maximum possible entropy of the probability distribution (i.e., generating a normalized entropy). It should be appreciated that the maximum possible entropy of the probability distribution is based on the number of elements in the probability distribution, not the values of the elements themselves.

[0093] In block 514, the system 100 applies a sigmoid function to the ratio (from block 506), the kurtosis score (from block 508), and the normalized entropy (from block 512). The sigmoid function can be calculated according to the following equation:

[0094]

[0095] Accordingly, the system 100 obtains six output values (e.g., floating point numbers) after applying the sigmoid function. In block 516, the system 100 uses the six output values as features (e.g., by the confidence classifier 112).

[0096] Although blocks 502-516 are described in a relatively serial fashion, it will be appreciated that, in some embodiments, various blocks of the method 500 can be performed in parallel.

[0097] Referring now to Figure 6 In use, the system 100 (e.g., the cloud-based system 102) can perform a method 600 for determining intent using a trained confidence classifier (e.g., in real-time). It will be appreciated that particular blocks of the method 600 are shown by way of example, and such blocks can be combined or divided, added or removed, and / or reordered in whole or in part, according to particular embodiments. In some embodiments, the method 600 is associated with a conversation of a user with an automated chatbot, a personal robot, and / or other type of automated conversational system.

[0098] The illustrative method 600 begins in block 602, where the system 100 processes an utterance with the intent classifier 110 to determine a probability distribution of possible intents, and in block 604, the system 100 generates a plurality of kurtosis measures of the probability distribution (e.g., using the same kurtosis measures used to train the confidence classifier 112).

[0099] In block 606, the system 100 applies the trained confidence classifier 112 to determine a single normalized probability of the most likely intent associated with the utterance. For example, the confidence classifier 112 can use the plurality of kurtosis measures of the probability distribution as features in a similar manner as described above. In block 608, the system 100 compares the normalized probability of the most likely intent to a confidence threshold. In illustrative embodiments, the confidence threshold is predefined by the system. In some embodiments, the confidence threshold is modifiable by a system administrator or other authorized party. As described above, in illustrative embodiments, the use of the confidence classifier 112 and the single normalized probability output by the classifier 112 allows the system 100 to rely on a single confidence threshold (e.g., 50%, 60%, etc.) rather than the moving target inherent in many other systems.

[0100] In block 610, the system 100 determines whether the confidence threshold is satisfied. If so, in block 612, the system 100 selects the most likely intent identified by the system 100 as the correct intent associated with the utterance; otherwise, in block 614, the system 100 determines that the most likely intent identified by the system 100 is an incorrect intent, or determines that there is not enough information to be reasonably confident in the prediction. Accordingly, in some embodiments, the system 100 can request additional information, prompt the user to provide additional clarifying information, and / or otherwise handle situations in which the system 100 is unable to correctly classify the user utterance.

[0101] It will also be appreciated that if the system 100 determines that the most likely intent identified by the system 100 is the correct intent (e.g., in response to the confidence threshold being satisfied), the system 100 can perform one or more actions in response to the determination. For example, in embodiments in which the system 100 is used with an automated chatbot, the system 100 can perform one or more processes associated with the inferred intent, and the automated chatbot can transmit the results of those processes and / or other information associated therewith to the user device 108 (e.g., as an automated “response” to the prompting utterance). In other words, in some embodiments, each of the possible intents processed by the classifiers 110, 112 can be mapped to or can otherwise correspond to one or more processes that are performed in response to a determination that the user utterance corresponds to that intent.

[0102] Although blocks 602-612 are described in a relatively serial fashion, it will be appreciated that in some embodiments, various blocks of the method 600 can be performed in parallel.

Claims

1. A method of applying a confidence classifier to intent classification associated with an automated chatbot, the method comprising: processing, by a computing system, an utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating, by the computing system, a plurality of kurtosis measurements of the probability distribution; and applying, by the computing system, a trained confidence classifier to determine a single normalized probability of a most probable intent associated with the utterance based on the plurality of kurtosis measurements of the probability distribution, wherein generating the plurality of kurtosis measurements of the probability distribution comprises: ordering probability scores of the probability distribution in descending order; selecting a maximum probability subset of the ordered probability scores as a set of probabilities; determining a plurality of ratios of consecutively ordered probabilities in the set of probabilities; determining a kurtosis score of the probabilities in the set of probabilities; determining an entropy of the probability distribution; and normalizing the entropy of the probability distribution by dividing the entropy by a maximum possible entropy of the probability distribution to generate a normalized entropy.

2. The method of claim 1, further comprising comparing, by the computing system, the normalized probability of the most probable intent associated with the utterance to a confidence threshold.

3. The method of claim 2, further comprising: selecting, by the computing system, the most probable intent as an intent associated with the utterance in response to determining that the normalized probability of the most probable intent associated with the utterance satisfies the confidence threshold; and transmitting, by the computing system, a message to a user device in communication with the automated chatbot in response to selecting the most probable intent as the intent associated with the utterance, wherein the message is a response to the utterance.

4. The method of claim 1, wherein generating the plurality of kurtosis measurements of the probability distribution comprises applying a sigmoid function to each of the plurality of ratios, the kurtosis score, and the normalized entropy.

5. The method of claim 1, wherein selecting the maximum probability subset of the ordered probability scores as the set of probabilities comprises selecting five maximum probabilities of the ordered probability scores as the set of probabilities.

6. The method of claim 1, wherein generating the plurality of kurtosis measurements of the probability distribution comprises determining a kurtosis score of the probability distribution.

7. The method of claim 1, wherein generating the plurality of kurtosis measurements of the probability distribution comprises determining an entropy of the probability distribution.

8. The method of claim 1, wherein generating the plurality of kurtosis measurements of the probability distribution comprises: ordering probabilities of the probability distribution by maximum probability; and determining a plurality of ratios of consecutively ordered probabilities in response to ordering the probabilities of the probability distribution.

9. The method of claim 1, further comprising training, by the computing system, the confidence classifier by adjusting parameters using stochastic gradient descent optimization. ​ ​ 10. A system for applying a confidence classifier for intent classification associated with an automated chatbot, the system comprising: at least one processor; and at least one memory including a plurality of instructions stored therein that, in response to execution by the at least one processor, cause the system to: process an utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generate a plurality of kurtosis measures of the probability distribution; and apply a trained confidence classifier to determine a single normalized probability of a most probable intent associated with the utterance based on the plurality of kurtosis measures of the probability distribution wherein generating the plurality of kurtosis measures of the probability distribution comprises: ordering probability scores of the probability distribution in descending order; selecting a maximum probability subset of the ordered probability scores as a set of probabilities; determining a plurality of ratios of consecutively ordered probabilities in the set of probabilities; determining a kurtosis score of the probabilities in the set of probabilities; determining an entropy of the probability distribution; and normalizing the entropy of the probability distribution by dividing the entropy by a maximum possible entropy of the probability distribution to generate a normalized entropy.

11. The system of claim 10, wherein the plurality of instructions further cause the system to compare the normalized probability of the most probable intent associated with the utterance to a confidence threshold.

12. The system of claim 11, wherein the plurality of instructions further cause the system to: in response to determining that the normalized probability of the most probable intent associated with the utterance satisfies the confidence threshold, select the most probable intent as an intent associated with the utterance; and in response to selecting the most probable intent as the intent associated with the utterance, transmit a message to a user device in communication with the automated chatbot, wherein the message is a response to the utterance.

13. The system of claim 10, wherein generating the plurality of kurtosis measures of the probability distribution comprises applying a sigmoid function to each of the plurality of ratios, the kurtosis score, and the normalized entropy.

14. The system of claim 10, wherein selecting the maximum probability subset of the ordered probability scores as the set of probabilities comprises selecting five maximum probabilities of the ordered probability scores as the set of probabilities.

15. The system of claim 10, wherein generating the plurality of kurtosis measures of the probability distribution comprises determining a kurtosis score of the probability distribution.

16. The system of claim 10, wherein generating the plurality of kurtosis measures of the probability distribution comprises determining an entropy of the probability distribution.

17. The system of claim 10, wherein generating the plurality of kurtosis measures of the probability distribution comprises: ordering probabilities of the probability distribution by maximum probability; and in response to ordering the probabilities of the probability distribution, determining a plurality of ratios of consecutively ordered probabilities.

18. A method for applying a confidence classifier for intent classification associated with an automated chatbot, the method comprising: processing an utterance with an intent classifier to determine a probability distribution of possible intents associated with the utterance; generating a plurality of kurtosis measures of the probability distribution; and applying a trained confidence classifier to determine a single normalized probability of a most probable intent associated with the utterance based on the plurality of kurtosis measures of the probability distribution wherein generating the plurality of kurtosis measures of the probability distribution comprises: ordering probability scores of the probability distribution in descending order; selecting a maximum probability subset of the ordered probability scores as a set of probabilities; determining a plurality of ratios of consecutively ordered probabilities in the set of probabilities; determining a kurtosis score of the probabilities in the set of probabilities; determining an entropy of the probability distribution; and normalizing the entropy of the probability distribution by dividing the entropy by a maximum possible entropy of the probability distribution to generate a normalized entropy.

19. The method of claim 18, further comprising: comparing the normalized probability of the most probable intent associated with the utterance to a confidence threshold.

20. The method of claim 19, further comprising: in response to determining that the normalized probability of the most probable intent associated with the utterance satisfies the confidence threshold, selecting the most probable intent as an intent associated with the utterance; and in response to selecting the most probable intent as the intent associated with the utterance, transmitting a message to a user device in communication with the automated chatbot, wherein the message is a response to the utterance.

21. The method of claim 18, wherein generating the plurality of kurtosis measures of the probability distribution comprises applying a sigmoid function to each of the plurality of ratios, the kurtosis score, and the normalized entropy.

22. The method of claim 18, wherein selecting the maximum probability subset of the ordered probability scores as the set of probabilities comprises selecting five maximum probabilities of the ordered probability scores as the set of probabilities.

23. The method of claim 18, wherein generating the plurality of kurtosis measures of the probability distribution comprises determining a kurtosis score of the probability distribution.

24. The method of claim 18, wherein generating the plurality of kurtosis measures of the probability distribution comprises determining an entropy of the probability distribution.

25. The method of claim 18, wherein generating the plurality of kurtosis measures of the probability distribution comprises: ordering probabilities of the probability distribution by maximum probability; and in response to ordering the probabilities of the probability distribution, determining a plurality of ratios of consecutively ordered probabilities.

18. The system of claim 10, wherein the plurality of instructions further cause the system to train the confidence classifier by adjusting parameters using stochastic gradient descent optimization.

Images