Global Confidence Classifier for Information Retrieval in Contact Centers

Abstract

A method of leveraging a global confidence classifier for information retrieval in a contact center, according to one embodiment, includes receiving, by a computing system, a user query from a contact center client communication with a knowledge-only bot of the computing system; performing, by the computing system, feature extraction on the user query by converting query words of the user query into a numeric vector representation of the user query; identifying, by the computing system, a subset of documents that are most likely to be responsive to the user query; and re-ranking, by the computing system, the subset of documents that are most likely to be responsive to the user query based on the global confidence classifier model.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Non-provisional Patent Application No. 17 / 564,783, filed December 29, 2021, which is incorporated by reference in its entirety. [Background technology]

[0002] Chatbots have become a ubiquitous tool for businesses and contact centers to deliver improved customer experience and responsiveness to their clients. Given the advent of deep learning techniques, improved hardware, and artificial intelligence platforms, there has been a surge in chatbot development. There are multiple styles of chatbots, including knowledge-only bots, self-service bots, and hybrid bots. Knowledge-only bots rely on a knowledge base that may be created using organizational FAQs, product documentation, user manuals, and / or other relevant documentation. Self-service bots are dynamic in that they understand the user's intent, gather the required information from the user, and provide suitable answers based on available resources. Hybrid bots combine approaches from both knowledge-only and self-service bots. Summary of the Invention

[0003] One embodiment is directed to unique systems, components, and methods for leveraging global confidence classifiers for information retrieval in contact centers. Other embodiments are directed to apparatus, systems, devices, hardware, methods, and combinations thereof for leveraging global confidence classifiers for information retrieval in contact centers.

[0004] According to one embodiment, a method of leveraging a global confidence classifier for information retrieval in a contact center may include receiving, by a computing system, a user query from a contact center client communication with a knowledge-only bot of the computing system; performing, by the computing system, feature extraction on the user query by converting query words of the user query into a numeric vector representation of the user query; identifying, by the computing system, a subset of documents that are most likely to be responsive to the user query; and re-ranking, by the computing system, the subset of documents that are most likely to be responsive to the user query based on the global confidence classifier model.

[0005] In some embodiments, the method may further include, in response to re-ranking, by the computing system, the subset of documents that are most likely to respond to the user query, providing a response to the contact center client based on the subset of documents re-ranked via the knowledge-only bot.

[0006] In some embodiments, a global confidence classifier model may be trained based on document sets for multiple organizations.

[0007] In some embodiments, the global confidence classifier model may be trained based on document sets in multiple human languages.

[0008] In some embodiments, the global confidence classifier model may be trained based on a confidence classifier that uses at least five input features.

[0009] In some embodiments, the at least five input features may include a first alignment score between the user query and the corresponding document, a second alignment score between the corresponding document and the user query, a cosine similarity score between the user query and the corresponding document, a document score from a document classifier, and a word overlap score that calculates the number of similar n-grams between the user query and the corresponding document.

[0010] In some embodiments, a global confidence classifier model may be trained based on a confidence classifier having multiple classes.

[0011] In some embodiments, the global confidence classifier model may be a machine learning model.

[0012] In some embodiments, performing feature extraction on the user query may include performing feature extraction on the user query using the custom embedding.

[0013] In some embodiments, the method may further include natural language processing, by the computing system, of the user query to generate processed data, the natural language processing including at least one of cleaning or segmenting the data of the user query.

[0014] In some embodiments, performing feature extraction on the user query may include performing feature extraction on the processed data.

[0015] In some embodiments, identifying the subset of documents that are most likely to respond to a user query may include identifying the subset of documents based on the processed data.

[0016] According to another embodiment, a system for leveraging a global confidence classifier for information retrieval in a contact center may include at least one processor and at least one memory storing a plurality of instructions that, upon execution by the at least one processor, cause the system to receive a user query from a contact center client communicating with a knowledge-only bot of the system, perform feature extraction on the user query by converting query words of the user query into a numeric vector representation of the user query, identify a subset of documents that are most likely to respond to the user query, and re-rank the subset of documents that are most likely to respond to the user query based on the feature extraction and the global confidence classifier model.

[0017] In some embodiments, the instructions may further cause the system, in response to re-ranking the subset of documents that are most likely to respond to a user query, to provide a response to the contact center client via the knowledge-only bot based on the re-ranked subset of documents.

[0018] In some embodiments, a global confidence classifier model may be trained based on document sets for multiple organizations in multiple human languages.

[0019] In some embodiments, the global confidence classifier model may be trained based on a confidence classifier that uses at least five input features.

[0020] In some embodiments, the at least five input features may include a first alignment score between the user query and the corresponding document, a second alignment score between the corresponding document and the user query, a cosine similarity score between the user query and the corresponding document, a document score from a document classifier, and a word overlap score that calculates the number of similar n-grams between the user query and the corresponding document.

[0021] In some embodiments, the global confidence classifier model may be a machine learning model.

[0022] In some embodiments, performing feature extraction on the user query may include performing feature extraction on the user query using the custom embedding.

[0023] In some embodiments, the instructions may further cause the system to apply natural language processing to the user query to generate processed data, where the natural language processing may include at least one of cleaning or segmenting the user query data, where performing feature extraction on the user query may include performing feature extraction on the processed data, and where identifying a subset of documents most likely to be responsive to the user query may include identifying the subset of documents based on the processed data.

[0024] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter. Further embodiments, forms, features, and aspects of the present application will become apparent from the description and figures provided herein. [Brief description of the drawings]

[0025] The concepts described herein are illustrated by way of example, and not by way of limitation, in the accompanying drawings. For simplicity and clarity of illustration, elements illustrated in the figures have not necessarily been drawn to scale. Where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements. [Figure 1] FIG. 1 is a simplified block diagram of at least one embodiment of a system for using and training a global confidence classifier for information retrieval in a contact center. [Diagram 2] FIG. 1 is a simplified block diagram of at least one embodiment of a cloud-based system. [Diagram 3] 1 is a simplified block diagram of at least one embodiment of a computing device. [Figure 4] FIG. 1 is a simplified block diagram of at least one embodiment of a system flow for leveraging a global confidence classifier for information retrieval in a contact center. [Diagram 5] FIG. 1 is a simplified block diagram of at least one embodiment of a system flow for leveraging a global confidence classifier for information retrieval in a contact center. [Figure 6] FIG. 1 is a simplified block diagram of at least one embodiment of a system flow for leveraging a global confidence classifier for information retrieval in a contact center. [Figure 7] 11 is a table detailing statistics comparing conventional classification approaches with the global confidence classification approach of the present disclosure. [Figure 8] 11 is a table detailing statistics comparing conventional classification approaches with the global confidence classification approach of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are herein described in detail, It should be understood, however, that there is no intention to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the scope of the present disclosure and the appended claims.

[0027] References herein such as "one embodiment," "one embodiment," "exemplary embodiment," and the like indicate that the embodiment being described may include a particular feature, structure, or characteristic, but that all embodiments may or may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. It should be further understood that references to "preferred" components or features may indicate the desirability of a particular component or feature with respect to one embodiment, but that the present disclosure is not so limited with respect to other embodiments that may omit such component or feature. Moreover, when a particular feature, structure, or characteristic is described in connection with an embodiment, the implementation of such feature, structure, or characteristic in connection with other embodiments, whether or not expressly described, is within the knowledge of one of ordinary skill in the art. Additionally, it should be understood that an item in a list of the form "at least one of A, B, and C" can mean (A), (B), (C), (A and B), (B and C), (A and C), or (A, B, and C). Similarly, it should be understood that an item recited in the form "at least one of A, B, or C" can mean (A), (B), (C), (A and B), (B and C), (A and C), or (A, B, and C). Additionally, with respect to the claims, use of words and phrases such as "a," "an," "at least one," and / or "at least one portion" should not be construed as limited to only one of such elements, unless specifically stated to the contrary, and use of phrases such as "at least a portion" and / or "a portion" should be construed to encompass both embodiments including only a portion of such an element and embodiments including the entirety of such an element, unless specifically stated to the contrary.

[0028] The disclosed embodiments may, in some cases, be implemented in hardware, firmware, software, or a combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media that may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, a media disk, or other media device).

[0029] In the drawings, some structural or method features may be shown in a specific arrangement and / or ordering. However, it should be understood that such specific arrangement and / or ordering may not be required. Rather, in some embodiments, such features may be arranged in a different manner and / or order than that shown in the illustrative drawings, unless indicated to the contrary. Additionally, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, it may not be included or may be combined with other features.

[0030] Referring now to FIG. 1, in an exemplary embodiment, a system 100 for leveraging a global confidence classifier for information retrieval in a contact center includes a cloud-based system 102, a network 104, a contact center system 106, and a user device 108. Although only one cloud-based system 102, one network 104, one contact center system 106, and one user device 108 are shown in the exemplary embodiment of FIG. 1, the system 100 may include multiple cloud-based systems 102, networks 104, contact center systems 106, and / or user devices 108 in other embodiments. For example, in some embodiments, multiple cloud-based systems 102 (e.g., related or unrelated systems) may be used to perform various functions described herein. Furthermore, in some embodiments, one or more of the systems described herein may be excluded from the system 100, and one or more of the systems described as being independent may form part of another system and / or one or more of the systems described as forming part of another system may be independent.

[0031] It should be appreciated that the techniques described herein may improve confidence classification of knowledge-only bots. Classification challenges for such bots may result from the simplicity of the corresponding framework, poor performance of machine learning models due to lack of data, lack of information (e.g., metadata) about documents, and / or other factors. The global confidence classification techniques described herein include collecting data from various sources and using the same model for multiple bots, which provides higher accuracy and lower latency than traditional approaches (see, e.g., tables in Figs. 7-8). As described below, in an exemplary embodiment, the global confidence classifier trains a model to predict five-class probabilities based on five numeric input features. It should be appreciated that the global confidence classifier may be trained using multiple datasets using a larger number of documents, categories, and alternative phrases due to the fact that text, such as words or bigrams, is not directly used as a feature portion. However, it should be appreciated that in other embodiments, the global confidence classifier may be structured differently.

[0032] It should be understood that each of the cloud-based system 102, the network 104, the contact center system 106, and / or the user devices 108 may be embodied as any type of device / system, collection of devices / systems, or portions thereof suitable for performing the functions described herein.

[0033] The cloud-based system 102 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in an exemplary embodiment, the cloud-based system 102 is configured to leverage a global confidence classifier for information retrieval in a contact center as described herein. In particular, the cloud-based system 102 may include one or more data stores or databases configured to store various documents, models, embeddings, classifiers, and / or other data related to the features described herein. Additionally, the cloud-based system 102 may be configured to perform natural language processing (NLP) of various documents (e.g., document cleaning, data segmentation, etc.), document classification (e.g., using machine learning), feature extraction (e.g., using custom word embeddings, generic embeddings, etc.), confidence classification, document ranking / re-ranking, and / or other features related to the global confidence classification described herein.

[0034] Although cloud-based system 102 is described herein in the singular, it should be understood that cloud-based system 102 may be embodied as or include multiple servers / systems in some embodiments. Additionally, while cloud-based system 102 is described herein as a cloud-based system, it should be understood that system 102 may be embodied in other embodiments as one or more servers / systems that exist outside of a cloud computing environment. In some embodiments, cloud-based system 102 may be embodied as or similar to cloud-based system 200 described with reference to FIG. 2.

[0035] In a cloud-based embodiment, the cloud-based system 102 may be embodied as a server-ambiguous computing solution that, for example, executes instructions on demand, executes instructions only when prompted by a particular activity / trigger, and does not consume computing resources when not in use. That is, the system 102 may be embodied as a virtual computing environment that exists "on" a computing system (e.g., a distributed network of devices) where various virtual functions (e.g., Lambda functions, Azure functions, Google Cloud functions, and / or other suitable virtual functions) may be executed corresponding to the functionality of the system 102 described herein. For example, when an event occurs (e.g., data is transferred to the system 102 for processing), the virtual computing environment may be communicated (e.g., via a request to the virtual computing environment's API), which may then route the request to the correct virtual function (e.g., a particular server-ambiguous computing resource) based on a set of rules. Thus, when a request for transmission of data is made by a user (e.g., via an appropriate user interface to the system 102), the appropriate virtual function may be executed to perform the action before deleting the instance of the virtual function.

[0036] Network 104 may be embodied as any one or more types of communication networks capable of facilitating communication between various devices communicatively connected via network 104. Thus, network 104 may include one or more networks, routers, switches, access points, hubs, computers, and / or other intervening network devices. For example, network 104 may be embodied as or may otherwise include one or more cellular networks, telephone networks, local or wide area networks, generally available global networks (e.g., the Internet), ad-hoc networks, short range communication links, or combinations thereof. In some embodiments, network 104 may include circuit-switched voice or data networks, packet-switched voice or data networks, and / or any other network capable of carrying voice and / or data. Specifically, in some embodiments, network 104 may include Internet Protocol (IP)-based and / or asynchronous transfer mode (ATM)-based networks. In some embodiments, network 104 may handle voice traffic (e.g., via a Voice over IP (VOIP) network), web traffic (e.g., hypertext transfer protocol (HTTP) traffic and hypertext markup language (HTML) traffic), and / or other network traffic, depending on the particular embodiment and / or the devices of system 100 that are in communication with each other.In various embodiments, the network 104 may include analog or digital wired and wireless networks (e.g., IEEE 802.11 networks, Public Switched Telephone Networks (PSTN), Integrated Services Digital Networks (ISDN), and Digital Subscriber Lines (xDSL)), third generation (3G) mobile networks, fourth generation (4G) mobile networks, fifth generation (5G) mobile networks, wired Ethernet networks, private networks (e.g., intranets, etc.), radio, television, cable, satellite, and / or any other delivery or tunneling mechanism for carrying data, or any suitable combination of such networks. The network 104 may enable connectivity between the various devices / systems 102, 106, 108 of the system 100. It should be appreciated that the various devices / systems 102 , 106 , 108 may communicate with one another over different networks 104 depending on the source and / or destination device / system 102 , 106 , 108 .

[0037] It should be appreciated that in some embodiments, the cloud-based system 102 may be communicatively coupled to, form a part of, and / or otherwise be used in conjunction with the contact center system 106. For example, the contact center system 106 may include a chatbot (e.g., similar to chatbot 218 of FIG. 2) configured to communicate with a user (e.g., via a user device 108). Further, in some embodiments, the user device 108 may communicate directly with the cloud-based system 102.

[0038] The contact center system 106 may be embodied as any system capable of providing contact center services (e.g., call center services) to end users (e.g., contact center clients) and otherwise performing the functions described herein. It should be understood that depending on the particular embodiment, the contact center system 106 may be located on the premises / campus of the organization utilizing the contact center system 106 and / or may be located remotely relative to the organization (e.g., in a cloud-based computing environment). In some embodiments, a portion of the contact center system 106 may be located on the premises / campus of the organization while other portions of the contact center system 106 are located remotely relative to the premises / campus of the organization. As such, it should be understood that the contact center system 106 may be deployed within equipment dedicated to the organization or its third party service provider and / or may be deployed in a remote computing environment, such as, for example, a private or public cloud environment with infrastructure to support multiple contact centers for multiple enterprises. In some embodiments, the contact center system 106 includes resources (e.g., workers, computers, and telecommunications equipment) to enable the delivery of services via telephone and / or other communication mechanisms. Such services may include, for example, technical support, help desk support, emergency response, and / or other contact center services, depending on the particular type of contact center.

[0039] The user device 108 may be embodied as any type of device capable of executing applications and otherwise performing the functions described herein. For example, in some embodiments, the user device 108 is configured to execute applications for participating in conversations with a personal bot, an automated agent, a chatbot, or other automated system. Thus, the user device 108 may have a variety of input / output devices with which a user may interact to provide and receive audio, text, video, and / or other forms of data. It should be understood that the application may be embodied as any type of application suitable for performing the functions described herein. Specifically, in some embodiments, the application may be embodied as a mobile application (e.g., a smartphone application), a cloud-based application, a web application, a thin-client application, and / or another type of application. For example, in some embodiments, the application may function as a client-side interface (e.g., via a web browser) for a web-based application or service.

[0040] It should be understood that each of the cloud-based system 102, the network 104, the contact center system 106, and / or the user devices 108 may be embodied as (and / or include) one or more computing devices similar to the computing device 300 described below with reference to Figure 3. For example, in an example embodiment, each of the cloud-based system 102, the network 104, the contact center system 106, and / or the user devices 108 may include a processing device 302 and a memory 306 that stores operational logic 308 (e.g., instructions) to cause the processing device 302 to execute for operation of the corresponding device.

[0041] 2, there is shown a simplified block diagram of at least one embodiment cloud-based system 200. The exemplary cloud-based system 200 includes a border communication device 202, a SIP server 204, a resource manager 206, a media control platform 208, a speech / text analysis system 210, a voice generator 212, an audio gateway 214, a media augmentation system 216, a chatbot 218, and voice data storage 220. 2 shows only one border communication device 202, one SIP server 204, one resource manager 206, one media control platform 208, one speech / text analysis system 210, one voice generator 212, one audio gateway 214, one media augmentation system 216, one chatbot 218, and one voice data storage 220, in other embodiments, the cloud-based system 200 may include multiple border communication devices 202, SIP servers 204, resource managers 206, media control platforms 208, speech / text analysis systems 210, voice generators 212, audio gateways 214, media augmentation systems 216, chatbots 218, and / or voice data storage 220. For example, in some embodiments, multiple chatbots 218 may be used to communicate regarding different subject matter handled by the same cloud-based system 200. Further, in some embodiments, one or more of the components described herein may be excluded from system 200, one or more of the components described as being separate may form part of another component, and / or one or more of the components described as forming part of another component may be separate.

[0042] The border communication device 202 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in some embodiments, the border communication device 202 may be configured to control signaling and media streams involved in, for example, the setup, execution, and teardown of voice conversations and other media communications between an end user and a contact center system. In some embodiments, the border communication device 202 may be a session border controller (SBC) that controls signaling and media exchanged during a media session (also referred to as a "call," "telephone call," or "communication session") between an end user and a contact center system. In some embodiments, the signaling exchanged during a media session may include SIP, H.323, Media Gateway Control Protocol (MGCP), and / or any other voice-over IP (VoIP) call signaling protocol. The media exchanged during a media session may include media streams carrying audio, video, or other data of the call along with call statistics and quality information.

[0043] In some embodiments, the border communication device 202 may operate according to a standard SIP back-to-back user agent (B2BUA) configuration. In this regard, the border communication device 202 may be inserted into the signaling and media paths established between the calling and called parties in a VoIP call. It should be understood that in some embodiments, other intermediate software and / or hardware devices may be involved in establishing the signaling and / or media paths between the calling and called parties.

[0044] In some embodiments, the border communication device 202 may exercise control over signaling (e.g., SIP messages) and media streams (e.g., RTP data) routed between end user devices (e.g., user devices 108) and contact center systems (e.g., contact center system 106) that traverse a network (e.g., network 104). In this regard, the border communication device 202 may be coupled to trunks that carry signals and media for calls to and from the user devices through the network, and trunks that carry signals and media to and from the contact center systems through the network.

[0045] The SIP server 204 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in some embodiments, the SIP server 204 may act as a SIP B2UBA and control the flow of SIP requests and responses between SIP endpoints. In other embodiments, any other controller configured to set up and tear down VoIP communication sessions may be contemplated in addition to or in place of the SIP server 204. The SIP server 204 may be a separate logical component or may be combined with the resource manager 206. In some embodiments, the SIP server 204 may be hosted in a contact center system (e.g., the contact center system 106). While the exemplary embodiment uses a SIP server 204, in other embodiments, another call server configured with another VoIP protocol, such as, for example, the H.232 protocol, the Media Gateway Control Protocol, the Skype protocol, and / or other suitable technologies, may be used in addition to or instead of SIP.

[0046] The resource manager 206 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. In an example embodiment, the resource manager 206 may be configured to allocate and monitor a pool of media control platforms to provide load balancing and high availability for each resource type. In some embodiments, the resource manager 206 may monitor and select a media control platform 208 from a cluster of available platforms. The selection of the media control platform 208 may be dynamic, for example, based on an identification of the location of the calling end user, the type of media service to be rendered, the detected quality of the current media service, and / or other factors.

[0047] In some embodiments, the resource manager 206 may be configured to process requests for media services and, for example, interact with a configuration server having a configuration database to determine service profiles that can deliver interactive voice response (IVR) profiles, voice applications (e.g., Voice Extensible Markup Language (Voice XML) applications), announcement and conferencing applications, resources, and services such as, for example, a media control platform. According to some embodiments, the resource manager may offer a hierarchical multi-tenant configuration to the service provider, allowing the service provider to allocate a selected number of resources to each tenant.

[0048] In some embodiments, the resource manager 206 may be configured to operate as a SIP proxy, a SIP registrar, and / or a SIP notifier. In this regard, the resource manager 206 may act as a proxy for SIP traffic between two SIP components. As a SIP registrar, the resource manager 206 may accept registrations of various resources, for example, via SIP REGISTER messages. In this manner, the cloud-based system 200 may support transparent relocation of call processing components. In some embodiments, a component, such as the media control platform 208, does not register with the resource manager 206 upon startup. The resource manager 206 may discover an instance of the media control platform 208 through configuration information retrieved from a configuration database. If the media control platform 208 is configured for monitoring, the resource manager 206 may monitor resource health, for example, by using a SIP OPTIONS message. In some embodiments, to determine whether the resources in the group are alive, the resource manager 206 may periodically send a SIP OPTIONS message to each media control platform 208 resource in the group. If the resource manager 206 receives an OK response, the resource is considered alive. It should be understood that the resource manager 206 may be configured to perform various other functions that are omitted for brevity of the description. The resource manager 206 and the media control platform 208 may be collectively referred to as a media controller.

[0049] In some embodiments, the resource manager 206 may act as a SIP notifier, for example, by accepting SIP SUBSCRIBE requests from the SIP server 204 and maintaining multiple independent subscriptions for the same or different SIP devices. Subscription notifications are targeted to the tenants managed by the resource manager 206. In this role, the resource manager 206 may periodically generate SIP NOTIFY requests to subscribers (or tenants) about port usage and the number of available ports. The resource manager 206 may support multi-tenancy by sending notifications that include the tenant name and the current status (in or out of service) of the media control platform 208 associated with the tenant, as well as the tenant's current capacity.

[0050] Media control platform 208 may be embodied as any service or system capable of providing media services and otherwise performing the functions described herein. For example, in some embodiments, media control platform 208 may be configured to provide call and media services in response to requests from service users. Such services may include, but are not limited to, initiating outbound calls, playing music or providing other media while a call is on hold, call recording, conferencing, call progress detection, playing audio / video prompts during customer self-service sessions, and / or other call and media services. One or more of the services may be defined by a voice application (e.g., a VoiceXML application) that is executed as part of the process of establishing a media session between media control platform 208 and an end user.

[0051] The speech / text analytics system (STAS) 210 may be embodied as any service or system capable of providing various speech analysis and text processing functions (e.g., text-to-speech) and performing functions otherwise described herein, as will be appreciated by one of ordinary skill in the art. The speech / text analytics system 210 may perform automatic speech and / or text recognition and grammar matching for end-user communication sessions processed by the cloud-based system 200. The speech / text analytics system 210 may include one or more processors and instructions stored on a machine-readable medium that are executed by the processors to perform various operations. In some embodiments, the machine-readable medium may include a non-transitory storage medium, such as a hard disk and a hardware memory system.

[0052] The audio generator 212 may be embodied as any service or system capable of generating audio communications and otherwise performing the functions described herein. In some embodiments, the audio generator 212 may generate audio communications based on a particular audio signature.

[0053] The Audio Gateway 214 may be embodied as any service or system capable of performing the functions described herein. In an exemplary embodiment, the Audio Gateway 214 receives end-user calls from or places calls to voice communication devices, such as end-user devices, and answers the calls according to a voice program corresponding to the communication routing configuration of the contact center system. In some embodiments, the voice program may include a voice avatar. The voice program may be accessed from a local memory in the Audio Gateway 214 or from other storage media in the cloud-based system 200. In some embodiments, the Audio Gateway 214 may process voice programs that are script-based voice applications. Thus, the voice program may be a script written in a scripting language, such as Voice Extensible Markup Language (VoiceXML) or speech application language tag (SALT). The cloud-based system 200 may also communicate with the voice data storage 220 to read and / or write user interaction data (e.g., state variables for a data communication session) in a shared memory space.

[0054] The media augmentation system 216 may be embodied as any service or system capable of specifying how portions of the cloud-based system 200 (e.g., one or more of the border communication device 202, the SIP server 204, the resource manager 206, the media control platform 208, the speech / text analysis system 210, the voice generator 212, the audio gateway 214, the media augmentation system 216, the chatbot 218, the voice data storage 220, and / or one or more portions thereof) interact with each other and perform the functions otherwise described herein. In some embodiments, the media augmentation system 216 may be embodied as or include an application program interface (API). In some embodiments, the media augmentation system 216 enables integration of different parameters and / or protocols used with various planned applications and media types utilized within the cloud-based system 200.

[0055] The chatbot 218 may be embodied as any automated service or system capable of using automation to engage with end users and perform the functions otherwise described herein. For example, in some embodiments, the chatbot 218 may operate as an executable program that may be launched, for example, pursuant to a request for a particular chatbot. In some embodiments, the chatbot 218 simulates and processes human (written or spoken) conversation, allowing a human to interact with a digital device as if the human were communicating with another human. In some embodiments, the chatbot 218 may be as simple as a rudimentary program that answers simple queries with a single-line response, or may be sophisticated as a digital assistant that learns and evolves to deliver increasing levels of personalization as it collects and processes information. In some embodiments, the chatbot 218 includes and / or leverages artificial intelligence, adaptive learning, bots, cognitive computing, and / or other automation techniques. The chatbot 218 may also be referred to herein as one or more of a chat robot, an AI chatbot, an automated chat robot, a chatterbot, a dialogue system, a conversational agent, an automated chat resource, and / or a bot.

[0056] An advantage of utilizing automated chat robots to engage in chat conversations with end users may be that it helps contact centers use valuable and costly resources, such as human resources, more efficiently while maintaining end user satisfaction. For example, a chat robot may be initially invoked to handle a chat conversation without the human end user knowing that they are conversing with a robot. The chat conversation may be escalated to human resources, if appropriate. Thus, human resources need not be unnecessarily tied up in handling simple requests, but instead may be used more effectively to handle more complex requests or to monitor the progress of many different automated communications simultaneously.

[0057] As described herein, in an example embodiment, the chatbot 218 may be embodied as a knowledge-only bot that relies on a knowledge base created using, for example, organizational FAQs, product documentation, user manuals, and / or other relevant documentation.

[0058] The voice data storage 220 may be embodied as one or more databases, data structures, and / or data storage devices capable of storing data in or otherwise facilitating the storage of such data for the cloud-based system 200. For example, in some embodiments, the voice data storage 220 may include one or more cloud storage buckets. It should be understood that in other embodiments, the voice data storage 220 may additionally or alternatively include other types of voice data storage mechanisms that allow for dynamic scaling of the amount of data storage available to the cloud-based system 200. In some embodiments, the voice data storage 220 may store scripts (e.g., pre-programmed scripts, etc.). Although the voice data storage 220 is described herein as a data storage and a database, it should be understood that the voice data storage 220 may include both a database (or other type of organized collection of data and structures) and a data storage for the actual storage of the underlying data. The voice data storage 220 may store various data useful for performing the functions described herein.

[0059] 3, a simplified block diagram of at least one embodiment of a computing device 300 is shown. The exemplary computing device 300 illustrates at least one embodiment of a cloud-based system, a contact center system, and / or a user device that may be utilized in connection with the cloud-based system 102, the contact center system 106, and / or the user device 108 (and / or portions thereof) illustrated in FIG. 1. Additionally, in some embodiments, one or more of the border communication device 202, the SIP server 204, the resource manager 206, the media control platform 208, the speech / text analysis system 210, the voice generator 212, the voice gateway 214, the media augmentation system 216, the chatbot 218, and / or the voice data storage 220 (and / or portions thereof) may be embodied as or executed by a computing device similar to the computing device 300. Depending on the particular embodiment, computing device 300 may be embodied as a server, a desktop computer, a laptop computer, a tablet computer, a notebook, a netbook, an Ultrabook™, a mobile phone, a mobile computing device, a smartphone, a wearable computing device, a personal digital assistant, an Internet of Things (IoT) device, a processing system, a wireless access point, a router, a gateway, and / or any other computing, processing, and / or communications device capable of performing the functions described herein.

[0060] The computing device 300 includes a processing device 302 that executes algorithms and / or processes data in accordance with operational logic 308, an input / output device 304 that enables communication between the computing device 300 and one or more external devices 310, and a memory 306 that stores data received from the external device 310 via, for example, the input / output device 304.

[0061] The input / output devices 304 enable the computing device 300 to communicate with external devices 310. For example, the input / output devices 304 may include a transceiver, a network adapter, a network card, an interface, one or more communications ports (e.g., a USB port, a serial port, a parallel port, an analog port, a digital port, VGA, DVI, HDMI, FireWire, CAT5, or any other type of communications port or interface), and / or other communications circuitry. The communications circuitry of the computing device 300 may be configured to perform such communications using any one or more communications technologies (e.g., wireless or wired communications) and associated protocols (e.g., Ethernet, Bluetooth, Wi-Fi, WiMAX, etc.) depending on the particular computing device 300. The input / output devices 304 may include hardware, software, and / or firmware suitable for implementing the techniques described herein.

[0062] The external device 310 may be any type of device that allows data to be input or output from the computing device 300. For example, in various embodiments, the external device 310 may be embodied as the cloud-based system 102, the contact center system 106, the user device 108, and / or portions thereof. Additionally, in some embodiments, the external device 310 may be embodied as another computing device, a switch, a diagnostic tool, a controller, a printer, a display, an alarm, a peripheral device (e.g., a keyboard, a mouse, a touch screen display, etc.), and / or any other computing, processing, and / or communication device capable of performing the functions described herein. Additionally, it should be understood that in some embodiments, the external device 310 may be integrated into the computing device 300.

[0063] The processing device 302 may be embodied as any type of processor capable of performing the functions described herein. In particular, the processing device 302 may be embodied as one or more single-core or multi-core processors, microcontrollers, or other processors or processing / control circuits. For example, in some embodiments, the processing device 302 may include or be embodied as an arithmetic logic unit (ALU), a central processing unit (CPU), a digital signal processor (DSP), and / or another suitable processor. The processing device 302 may be of a programmable type, a dedicated hardwired state machine, or a combination thereof. A processing device 302 having multiple processing units may utilize distributed processing, pipeline processing, and / or parallel processing in various embodiments. Furthermore, the processing device 302 may be dedicated to performing only the operations described herein, or may be utilized in one or more additional applications. In an exemplary embodiment, the processing device 302 is programmable and executes algorithms and / or processes data according to operating logic 308 defined by programming instructions (e.g., software or firmware) stored in memory 306. Additionally or alternatively, the operating logic 308 of the processing device 302 may be defined at least in part by hardwired logic or other hardware. Further, the processing device 302 may include one or more components of any type suitable for processing signals received from the input / output device 304 or from other components or devices and providing a desired output signal. Such components may include digital circuits, analog circuits, or a combination thereof.

[0064] The memory 306 may be one or more types of non-transitory computer-readable media, such as solid-state memory, electromagnetic memory, optical memory, or combinations thereof. Additionally, the memory 306 may be volatile and / or non-volatile, and in some embodiments, some or all of the memory 306 may be of a portable type, such as a disk, tape, memory stick, cartridge, and / or other suitable portable memory. In operation, the memory 306 may store various data and software used during operation of the computing device 300, such as an operating system, applications, programs, libraries, and drivers. It should be appreciated that the memory 306 may store data that is manipulated by the operating logic 308 of the processing device 302, such as, for example, data representing signals received from and / or transmitted to the input / output device 304, in addition to or instead of storing programming instructions that define the operating logic 308. As shown in FIG. 3, the memory 306 may be included in and / or coupled to the processing device 302, depending on the particular embodiment. For example, in some embodiments, the processing device 302, memory 306, and / or other components of the computing device 300 may form part of a system-on-a-chip (SoC) and be integrated into a single integrated circuit chip.

[0065] In some embodiments, various components of computing device 300 (e.g., processing device 302 and memory 306) may be communicatively coupled via an I / O subsystem, which may be embodied as circuits and / or components for facilitating I / O operations with processing device 302, memory 306, and other components of computing device 300. For example, the I / O subsystem may be embodied as or otherwise include a memory controller hub, an I / O control hub, firmware devices, communications links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems for facilitating I / O operations.

[0066] The computing device 300 may, in other embodiments, include other or additional components, such as those typically found in a typical computing device (e.g., various input / output devices and / or other components). It should be further understood that one or more of the components of the computing device 300 described herein may be distributed across multiple computing devices. In other words, the techniques described herein may be employed by a computing system including one or more computing devices. Additionally, while only a single processing device 302, I / O device 304, and memory 306 are illustratively shown in FIG. 3, it should be understood that a particular computing device 300 may, in other embodiments, include multiple processing devices 302, I / O devices 304, and / or memory 306. Furthermore, in some embodiments, two or more external devices 310 may communicate with the computing device 300.

[0067] 4-6, in use, the system 100, or more specifically the cloud-based system 102, may execute a system flow 400 for leveraging a global confidence classifier for information retrieval in a contact center. It should be understood that certain blocks of the system flow 400 are illustrated by way of example, and that such blocks may be combined or divided, added or removed, and / or reordered, in whole or in part, depending on the particular embodiment, unless stated to the contrary.

[0068] It should be appreciated that the system flow 400 may be described in stages distributed across Figures 4-6. Specifically, the portion of the system flow 400 shown in Figure 4 may be a training stage, the portion of the system flow 400 shown in Figure 5 may be an inference stage, and the portion of the system flow 400 shown in Figure 6 may be a global document collection phase. Additionally, the portion of the system flow 400 shown in Figures 4-5 may be described as a knowledge-only bot framework, and the portion of the system flow 400 shown in Figure 6 may be described as a global confidence classifier. In some embodiments, the knowledge-only bot framework may be deployed to multiple organizations.

[0069] 4, in a document collection phase 402, the system 100 collects organizational FAQs, product documentation, user manuals, and / or other relevant documents from a document warehouse or data storage. In some embodiments, one or more of the documents include questions, answers, alternative questions, similar questions, categories, and / or other relevant data.

[0070] In a natural language processing (NLP) phase 404, the system 100 uses natural language understanding / processing techniques to clean documents and / or segment data. It should be understood that natural language refers to the way humans communicate with each other (e.g., speech and text). In some embodiments, NLP may include the use of statistical methods, machine learning (e.g., deep learning neural networks), and / or other techniques to perform inference on a particular task and / or develop a robust end-to-end system. In some embodiments, in the NLP phase 404, the system 100 processes the questions, answers, and alternatives for each document and cleans the data (e.g., by removing unnecessary characters / words, removing stop words, tokenization, word stemming, and / or performing other natural language processing).

[0071] In a document classifier phase 406, the system 100 trains a machine learning model to classify documents. It should be understood that a classifier in machine learning is an algorithm that automatically orders or categorizes data into one or more of a set of classes. For example, an email classifier may scan emails and filter them by class label: spam or non-spam. In an exemplary embodiment, the machine learning model predicts input features (e.g., TF-IDF score) and a target (e.g., document identifier). Since latency may be important, in some embodiments, the system 100 leverages a linear classifier to learn associations between input features and target document identifiers. In some embodiments, the model may be used to identify a subset of documents (e.g., top-N elements) that are most relevant for a query. In other embodiments, the document classifier may include creating a searchable index of all documents based on individual keywords or combinations of words. It should be understood that document indexing allows documents to be searched using the text contained within the document (e.g., including complete phrases / clauses). Full-text indexing may consider the entire content within a document as a piece of text, whereas field-based indexing may involve tagging with metadata (e.g., information about the data itself) that aids in retrieval of documents based on search characteristics such as document type, creation date, and / or other document-related parameters.

[0072] In the feature extraction phase 408, the system 100 converts the clean text into one or more feature vectors (i.e., converts the text format into a numerical representation). For example, in some embodiments, word embeddings and / or sentence embeddings may be used. It should be understood that generic and / or custom embeddings may be used depending on the particular embodiment. Custom embeddings may be trained from available documents and may capture organization-specific vocabulary (e.g., reducing out-of-vocabulary problems). There may be an assumption that the numerical representations of the text are similar in high dimensions. Word embeddings convert each word into a vector representation and average across the utterance to provide a single vector. Sentence embeddings may directly convert the entire utterance into a single vector based on the context. It should be understood that machine learning algorithms may learn from a predefined set of features from the training data to generate outputs for the test data. However, machine learning algorithms may work more efficiently with matrix / vector representations than raw text. Thus, by converting the text data into a numerical representation, the system 100 can more efficiently and effectively leverage machine learning and artificial intelligence techniques.

[0073] It should be appreciated that the results of the document classifier phase 406 and the feature extraction phase 408 may be stored in the data storage 410. For example, the data storage 410 may store the model from the document classifier, custom embeddings, confidence classifier data, document indexes, and / or other related data.

[0074] 5, as described above, the system 100 may perform an inference stage to draw inferences based on user input. Thus, in an NLP phase 412 of FIG. 5, the system 100 may receive a user query to a knowledge-only bot (e.g., "What is the customer care number?") and use natural language understanding / processing techniques to clean the query and / or segment the data in a manner similar to that described with reference to the NLP phase 404 of FIG. 4.

[0075] In the document identification phase 414, the system 100 may identify a subset of documents (e.g., the top N elements / documents) that are most relevant to the user query. In doing so, in some embodiments, the system 100 may calculate a TF-IDF score for a given query, process the score through a document classifier, and provide the top N possible documents for a given query. In another embodiment, the system 100 may search through a document indexer.

[0076] In the feature extraction phase 416, the system 100 may convert the user input query terms into a numerical vector representation (e.g., [0.45623, 0.34544, 0.5565, 0.98098, ...]) in a manner similar to that described above with reference to the feature extraction phrases 408 of Figure 4. It should be appreciated that during the feature extraction phase 416 and / or the document identification phase 414, the system 100 may retrieve data from and / or update data on the data storage 410 (see Figure 4).

[0077] In a document re-ranking phase 418, the system 100 may re-rank the top-N documents based on the input query and output the most relevant responses. In some embodiments, the system 100 generates the necessary input features between a given query and the possible top-N documents, processes the features through a confidence classifier, which outputs the probability of each class. The system 100 may generate a confidence score between each query and each possible document. The document with the highest score may be identified as the most likely document (e.g., the most likely question / answer document). It should be appreciated that the system 100 may utilize and / or update a global confidence classifier model 430 (see FIG. 6) when re-ranking the documents. In other embodiments, a similarity function (e.g., a cosine similarity function) may be utilized to generate the similarity scores.

[0078] 6, as described above, system 100 may execute a global confidence classifier stage to train a confidence classifier (e.g., confidence classifier 428) for use with multiple knowledge bases (e.g., for different organizations, different languages, etc.). As described above, in an exemplary embodiment, the global confidence classifier stage includes training a model to predict five-class probabilities based on five numeric input features.

[0079] In a document collection phase 420, the system 100 collects organizational FAQs, product documentation, user manuals, and / or other relevant documents from a document warehouse or data storage for multiple organizations. It should be appreciated that by relying on knowledge bases from multiple different organizations, the system 100 gains access to a larger number of documents, alternatives, categories, etc. In an NLP phase 422, the system 100 may use natural language understanding / processing techniques to clean the documents and / or segment the data in a manner similar to that described with reference to the NLP phase 404 of FIG.

[0080] In a document classifier phase 424, the system 100 may train a machine learning model to classify documents in a manner similar to that described with reference to the document classifier phase 406 of Figure 4. In a feature extraction phase 426, the system 100 may convert the clean text into one or more feature vectors (i.e., convert the textual form into a numerical representation) in a manner similar to that described with reference to the feature extraction phase 408 of Figure 4.

[0081] In the confidence classifier phase 428, the system 100 measures the confidence in each document (e.g., of the top N documents). It should be understood that the confidence classifier is useful when dealing with more than two classes in the classifier, and the confidence classifier evaluates the confidence of a particular class for a particular query. For example, when performing weather classification (e.g., summer, winter, fall, spring), given a weather condition, the confidence classifier may estimate how confidently the weather condition corresponds to various classes. In an exemplary embodiment, the confidence classifier is trained as a five-class problem using five input features.

[0082] The input features may include a first alignment score, a second alignment score, a similarity score, a document score, and a word overlap score. The first alignment score (F0) may be a word alignment score between the user query and the document, and the score may be calculated according to the following formula:

[0083]

number

[0084] The second alignment score (F1) is calculated using Equations 1-4 for calculating the first alignment score (F0), but exchanging the word vectors of the query and the document. It should be appreciated that the second alignment score provides how similar the document is to the query. The similarity score between the query and the document is calculated using the cosine similarity to the average word embedding vectors of the query and the document. The document scores are calculated from the document classifier, whereby each document predicted by the document classifier is also given a score for the document.

[0085] The word overlap score (OS) calculates how many n-grams (e.g., bigrams, in this context) are similar between the query and the document. The word overlap score (OS) is calculated according to the following formula:

[0086]

number

[0087] As shown above, the exemplary confidence classifier has five classes. In the first class, the predicted document identifier for a given query does not match the predicted document identifier with a very low score (e.g., satisfied due to insufficient input provided). In the second class, the predicted identifier for a given query does not match the predicted document identifier with a moderate score. In the third class, the predicted document identifier for a given query does not match the predicted document identifier, but the category of the predicted document is the same in terms of the predicted document identifier category. The document surfaces in the third or fourth position. In the fourth class, the predicted document identifier for a given query does not match the predicted document identifier, but the category of the predicted document is the same in terms of the predicted document identifier category. The document surfaces in the first or second position. In the fifth class, the predicted document identifier for a given query is the same as the predicted document identifier.

[0088] It should be appreciated that the results of the confidence classifier 428 may be used to further update or train the global confidence classifier model 430. Additionally, as shown, the global confidence classifier model 430 may be leveraged by the document re-ranking phase 418 (see FIG. 5).

[0089] As shown in the tables of Figures 7-8, the global classifier provides improved results over conventional approaches to classification using knowledge-only bots, both in terms of speed / duration of execution and accuracy of the approach. The table illustrates four different variations of knowledge base types and corresponding data: knowledge bases with categories and alternatives, knowledge bases with categories but no alternatives, knowledge bases with alternatives but no categories, and knowledge bases with neither alternatives nor categories. The table also illustrates results for knowledge bases of different sizes (i.e., 10 documents, 20 documents, 50 documents, 80 documents, 100 documents, and a "full" database with more than 100 documents) and different languages ​​(i.e., English and German). As shown, the global confidence classifier according to the techniques described herein provided improved results across almost all datasets.

Claims

1. 1. A method for utilizing a global confidence classifier for information retrieval in a contact center, the method comprising: receiving, by a computing system, a user query from a contact center client communicating with a knowledge-only bot of the computing system; performing, by the computing system, feature extraction on the user query by converting query terms of the user query into a numeric vector representation of the user query; identifying, by the computing system, a subset of documents that are most likely to respond to the user query; and re-ranking, by the computing system, the subset of documents that are most likely to respond to the user query based on a global confidence classifier model.

2. 2. The method of claim 1, further comprising: in response to the computing system re-ranking the subset of documents that are most likely to respond to the user query, providing a response to the contact center client based on the re-ranked subset of documents via the knowledge-only bot.

3. The method of claim 1 , wherein the global confidence classifier model is trained based on document sets for multiple organizations.

4. The method of claim 3 , wherein the global confidence classifier model is trained based on document sets in multiple human languages.

5. The method of claim 1 , wherein the global confidence classifier model is trained based on a confidence classifier that uses at least five input features.

6. 6. The method of claim 5, wherein the at least five input features include a first alignment score between the user query and the corresponding document, a second alignment score between the corresponding document and the user query, a cosine similarity score between the user query and the corresponding document, a document score from a document classifier, and a word overlap score that calculates a number of similar n-grams between the user query and the corresponding document.

7. The method of claim 1 , wherein the global confidence classifier model is trained based on a confidence classifier having multiple classes.

8. The method of claim 1 , wherein the global confidence classifier model is a machine learning model.

9. The method of claim 1 , wherein performing the feature extraction on the user query comprises performing the feature extraction on the user query using a custom embedding.

10. 2. The method of claim 1, further comprising: natural language processing, by the computing system, the user query to generate processed data, the natural language processing comprising at least one of cleaning or segmenting the user query data.

11. The method of claim 10 , wherein performing the feature extraction on the user query comprises performing feature extraction on the processed data.

12. The method of claim 10 , wherein identifying the subset of documents most likely to be responsive to the user query comprises identifying the subset of documents based on the processed data.

13. 1. A system for utilizing a global confidence classifier for information retrieval in a contact center, the system comprising: At least one processor; At least one memory containing a plurality of instructions stored therein, the instructions, upon execution by the at least one processor, providing the system with: receiving a user query from a contact center client in communication with a knowledge-only bot of the system; performing feature extraction on the user query by converting query terms of the user query into a numeric vector representation of the user query; identifying a subset of documents most likely to respond to said user query; re-ranking the subset of documents most likely to respond to the user query based on the feature extraction and a global confidence classifier model.

14. 14. The system of claim 13, wherein the instructions further cause the system to, in response to re-ranking the subset of documents that are most likely to respond to the user query, provide a response to the contact center client via the knowledge-only bot based on the re-ranked subset of documents.

15. The system of claim 13 , wherein the global confidence classifier model is trained based on document sets for multiple organizations in multiple human languages.

16. The system of claim 13 , wherein the global confidence classifier model is trained based on a confidence classifier that uses at least five input features.

17. 17. The system of claim 16, wherein the at least five input features include a first alignment score between the user query and the corresponding document, a second alignment score between the user query and the corresponding document, a cosine similarity score between the user query and the corresponding document, a document score from a document classifier, and a word overlap score that calculates a number of similar n-grams between the user query and the corresponding document.

18. The system of claim 13 , wherein the global confidence classifier model is a machine learning model.

19. The system of claim 13 , wherein performing the feature extraction on the user query comprises performing the feature extraction on the user query using a custom embedding.

20. The instructions further cause the system to apply natural language processing to the user query to generate processed data, the natural language processing including at least one of cleaning or segmenting the user query data; performing the feature extraction on the user query includes performing feature extraction on the processed data; 14. The system of claim 13, wherein identifying the subset of documents most likely to be responsive to the user query comprises identifying the subset of documents based on the processed data.

Images