A global confidence classifier for information retrieval in contact centers.

Abstract

A method of leveraging a global confidence classifier for information retrieval in a contact center, according to one embodiment, includes receiving, by a computing system, a user query from a contact center client communication with a knowledge-only bot of the computing system; performing, by the computing system, feature extraction on the user query by converting query words of the user query into a numeric vector representation of the user query; identifying, by the computing system, a subset of documents that are most likely to be responsive to the user query; and re-ranking, by the computing system, the subset of documents that are most likely to be responsive to the user query based on the global confidence classifier model.

Description

[[Technical Field]]

[0001] (Cross-Reference to Related Applications) This application claims priority to U.S. Non-Provisional Patent Application No. 17 / 564,783, filed on December 29, 2021, which is incorporated herein by reference in its entirety. [[Background Art]]

[0002] Chatbots have become a ubiquitous tool for businesses and contact centers to deliver improved customer experience and responsiveness to their clients. Given the advent of deep learning techniques, improved hardware, and artificial intelligence platforms, the development of chatbots has surged. There exist multiple styles of chatbots, including knowledge-only bots, self-service bots, and hybrid bots. Knowledge-only bots rely on a knowledge base that may be created using organizational FAQs, product documents, user manuals, and / or other related documents. Self-service bots are dynamic in that they understand a user's intent, collect necessary information from the user, and provide a suitable answer based on available resources. Hybrid bots combine approaches of both knowledge-only bots and self-service bots. [[Summary of the Invention]]

[0003] One embodiment is directed to unique systems, components, and methods for leveraging a global confidence classifier for information retrieval in a contact center. Other embodiments are directed to apparatuses, systems, devices, hardware, methods, and combinations thereof for leveraging a global confidence classifier for information retrieval in a contact center.

[0004] According to one embodiment, a method of utilizing a global confidence classifier for information retrieval in a contact center may include: a computing system receiving user queries from contact center client communications with a bot that only knows the computing system; a computing system performing feature extraction on user queries by converting query words of user queries into numerical vector representations of user queries; a computing system identifying a subset of documents most likely to respond to user queries; and a computing system re-ranking the subset of documents most likely to respond to user queries based on a global confidence classifier model.

[0005] In some embodiments, the method may further include providing a response to a contact center client based on a subset of documents reranked by a knowledge-only bot, in response to a computing system reranking a subset of documents that are most likely to respond to a user query.

[0006] In some embodiments, a global confidence classifier model may be trained based on document sets for multiple organizations.

[0007] In some embodiments, a global confidence classifier model may be trained on a set of documents in multiple human languages.

[0008] In some embodiments, a global confidence classifier model may be trained based on a confidence classifier that uses at least five input features.

[0009] In some embodiments, at least five input features may include a first alignment score between a user query and a corresponding document, a second alignment score between the corresponding document and the user query, a cosine similarity score between the user query and the corresponding document, a document score from a document classifier, and a word overlap score that calculates the number of similar n-grams between the user query and the corresponding document.

[0010] In some embodiments, a global confidence classifier model may be trained based on a confidence classifier having multiple classes.

[0011] In some embodiments, the global confidence classifier model can be a machine learning model.

[0012] In some embodiments, performing feature extraction on a user query may include performing feature extraction on a user query using custom embeddings.

[0013] In some embodiments, the method may further include a computing system performing natural language processing on a user query to generate processed data, the natural language processing including at least one of cleaning or segmenting the data of the user query.

[0014] In some embodiments, performing feature extraction on a user query may include performing feature extraction on processed data.

[0015] In some embodiments, identifying a subset of documents most likely to respond to a user query may involve identifying a subset of documents based on processed data.

[0016] According to another embodiment, a system for leveraging a global confidence classifier for information retrieval in a contact center may include at least one processor and at least one memory storing a set of instructions which, depending on the execution by the at least one processor, causes the system to receive user queries from a contact center client communicating with a system-knowledge-only bot, perform feature extraction on the user queries by converting the query words of the user queries into a numerical vector representation of the user queries, identify a subset of documents most likely to respond to the user queries, and re-rank the subset of documents most likely to respond to the user queries based on the feature extraction and a global confidence classifier model.

[0017] In some embodiments, the multiple instructions may further cause the system to re-rank a subset of documents that are most likely to respond to a user query, and then, via a knowledge-only bot, provide a response to the contact center client based on the re-ranked subset of documents.

[0018] In some embodiments, a global confidence classifier model may be trained on a set of documents for multiple organizations in multiple human languages.

[0019] In some embodiments, a global confidence classifier model may be trained based on a confidence classifier that uses at least five input features.

[0020] In some embodiments, at least five input features may include a first alignment score between a user query and a corresponding document, a second alignment score between the corresponding document and the user query, a cosine similarity score between the user query and the corresponding document, a document score from a document classifier, and a word overlap score that calculates the number of similar n-grams between the user query and the corresponding document.

[0021] In some embodiments, the global confidence classifier model may be a machine learning model.

[0022] In some embodiments, performing feature extraction on a user query may include performing feature extraction on the user query using custom embeddings.

[0023] In some embodiments, the plurality of instructions may further cause the system to apply natural language processing to the user query to generate processed data, wherein the natural language processing includes at least one of cleaning or segmenting data of the user query, performing feature extraction on the user query may include performing feature extraction on the processed data, and identifying the subset of documents most likely to respond to the user query may include identifying the subset of documents based on the processed data.

[0024] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter. Further embodiments, forms, features, and aspects of the present application will become apparent from the description and the drawings provided herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The concepts described herein are illustrative by way of example and not by way of limitation in the accompanying drawings. For simplicity and clarity of illustration, elements illustrated in the figures are not necessarily drawn to scale. Where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements. [Figure 1] It is a simplified block diagram of at least one embodiment of a system for using and training a global confidence classifier for information retrieval in a contact center. [Figure 2] It is a simplified block diagram of at least one embodiment of a cloud-based system. [Figure 3] It is a simplified block diagram of at least one embodiment of a computing device. [Figure 4] It is a simplified block diagram of at least one embodiment of a system flow for utilizing a global confidence classifier for information retrieval in a contact center. [Figure 5] It is a simplified block diagram of at least one embodiment of a system flow for utilizing a global confidence classifier for information retrieval in a contact center. [Figure 6] It is a simplified block diagram of at least one embodiment of a system flow for utilizing a global confidence classifier for information retrieval in a contact center. [Figure 7] It is a table detailing statistics comparing conventional classification techniques with the global confidence classification technique of the present disclosure. [Figure 8] It is a table detailing statistics comparing conventional classification techniques with the global confidence classification technique of the present disclosure. DETAILED DESCRIPTION OF EMBODIMENTS

[0026] While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. It should be understood, however, that there is no intent to limit the concepts of the present disclosure to the particular forms disclosed; on the contrary, the intent is to cover all modifications, equivalents, and alternatives consistent with the present disclosure and the appended claims.

[0027] References in this specification such as “one embodiment,” “one embodiment,” and “exemplary embodiment” indicate that the embodiments described may include certain features, structures, or characteristics, but not all embodiments may necessarily include or may not include certain features, structures, or characteristics. Furthermore, such phrases do not necessarily refer to the same embodiment. References to “preferred” components or features may indicate that certain components or features are desirable in relation to one embodiment, but it should be further understood that this disclosure does not limit itself in this way with respect to other embodiments in which such components or features may be omitted. Furthermore, where certain features, structures, or characteristics are described in relation to an embodiment, implementation of such features, structures, or characteristics in relation to other embodiments, whether explicitly described or not, should be understood as being within the knowledge of those skilled in the art. Additionally, it should be understood that items included in the list of “at least one of A, B, and C” may mean (A), (B), (C), (A and B), (B and C), (A and C), or (A, B, and C). Similarly, it should be understood that items listed in the form of “at least one of A, B, or C” can mean (A)(B), (C), (A and B), (B and C), (A and C), or (A, B, and C). Furthermore, with respect to the claims, the use of words and phrases such as “one (a),” “one (an),” “at least one,” and / or “at least one portion” should not be interpreted as limiting to only one of such elements unless specifically stated to the contrary, and the use of phrases such as “at least a portion,” and / or “a portion,” should be interpreted as encompassing both embodiments that include only a portion of such elements and embodiments that include all of such elements, unless specifically stated to the contrary.

[0028] The disclosed embodiments may, depending on the circumstances, be implemented in hardware, firmware, software, or a combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored in one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media that can be read and executed by one or more processors. The machine-readable storage media may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., volatile or non-volatile memory, media disk, or other media device).

[0029] In the drawings, some structural or methodological features may be shown in a specific arrangement and / or sequence. However, it should be understood that such a specific arrangement and / or sequence may not always be necessary. Rather, in some embodiments, such features may be arranged in a different manner and / or sequence than those shown in the illustrative drawings, unless otherwise indicated. In addition, the inclusion of structural or methodological features in a particular drawing does not mean that such features are required in all embodiments, and in some embodiments they may not be included or may be combined with other features.

[0030] Referring here to Figure 1, in an exemplary embodiment, System 100 for leveraging a global confidence classifier for information retrieval in a contact center includes a cloud-based system 102, a network 104, a contact center system 106, and a user device 108. Although only one cloud-based system 102, one network 104, one contact center system 106, and one user device 108 are shown in the exemplary embodiment of Figure 1, System 100 may include multiple cloud-based systems 102, networks 104, contact center systems 106, and / or user devices 108 in other embodiments. For example, in some embodiments, multiple cloud-based systems 102 (e.g., related or unrelated systems) may be used to perform various functions described herein. Furthermore, in some embodiments, one or more of the systems described herein may be excluded from System 100, one or more of the systems described as independent may form part of another system, and / or one or more of the systems described as forming part of another system may be independent.

[0031] It should be understood that the techniques described herein can improve the confidence classification of knowledge-only bots. Classification challenges for such bots may arise from the simplicity of the corresponding framework, the insufficient performance of machine learning models due to lack of data, the lack of information about documents (e.g., metadata), and / or other factors. The global confidence classification techniques described herein involve collecting data from various sources and using the same model for multiple bots, which provides higher accuracy and lower latency than conventional methods (see, for example, the tables in Figures 7-8). In exemplary embodiments, as described below, the global confidence classifier trains a model to predict five-class probabilities based on five numerical input features. It should be understood that the global confidence classifier can be trained using multiple datasets with a larger number of documents, categories, and alternative phrases, for example, due to the fact that text such as words or bigrams are not directly used as feature parts. However, it should be understood that in other embodiments, the global confidence classifier can be structured differently.

[0032] It should be understood that each of the cloud-based system 102, the network 104, the contact center system 106, and / or user device 108 may be embodied as any type of device / system, set of devices / systems, or part thereof that is suitable for performing the functions described herein.

[0033] The cloud-based system 102 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in an exemplary embodiment, the cloud-based system 102 is configured to leverage a global confidence classifier for information retrieval in a contact center, as described herein. Specifically, the cloud-based system 102 may include one or more data stores or databases configured to store various documents, models, embeddings, classifiers, and / or other data related to the features described herein. Furthermore, the cloud-based system 102 may be configured to perform natural language processing (NLP) on various documents (e.g., document cleaning, data segmentation, etc.), document classification (e.g., using machine learning), feature extraction (e.g., using custom word embeddings, generic embeddings, etc.), confidence classification, document ranking / re-ranking, and / or other features related to the global confidence classification described herein.

[0034] Although the cloud-based system 102 is described in the singular form herein, it should be understood that in some embodiments, the cloud-based system 102 may be embodied as multiple servers / systems or may include multiple servers / systems. Furthermore, although the cloud-based system 102 is described herein as a cloud-based system, it should be understood that in other embodiments, the system 102 may be embodied as one or more servers / systems located outside the cloud computing environment. In some embodiments, the cloud-based system 102 may be embodied as the cloud-based system 200 described with reference to Figure 2, or similarly.

[0035] In a cloud-based embodiment, the cloud-based system 102 may be embodied as a server-ambiguous computing solution that, for example, executes multiple instructions on demand, executes instructions only when prompted by a specific activity / trigger, and does not consume computing resources when not in use. That is, system 102 may be embodied as a virtual computing environment that resides "on top of" a computing system (e.g., a distributed network of devices) on which various virtual functions (e.g., lambda functions, Azure functions, Google Cloud functions, and / or other suitable virtual functions) can be executed in accordance with the functionality of system 102 as described herein. For example, when an event occurs (e.g., data is transferred to system 102 for processing), the virtual computing environment may be communicated (e.g., via requests to the virtual computing environment's API), thereby allowing the API to route requests to the correct virtual function (e.g., a specific server-ambiguous computing resource) based on a set of rules. Thus, when a request for data transmission is made by a user (e.g., via a suitable user interface to system 102), the appropriate virtual function may be executed to perform the action before deleting the instance of the virtual function.

[0036] Network 104 can be embodied as any one or more types of communication networks that can facilitate communication between various devices connected commutably via Network 104. Thus, Network 104 may include one or more networks, routers, switches, access points, hubs, computers, and / or other intervening network devices. For example, Network 104 may be embodied as one or more cellular networks, telephone networks, local or wide-area networks, generally available global networks (e.g., the Internet), ad-hoc networks, short-range communication links, or a combination thereof, or otherwise include them. In some embodiments, Network 104 may include circuit-switched voice or data networks, packet-switched voice or data networks, and / or any other networks capable of carrying voice and / or data. Specifically, in some embodiments, Network 104 may include Internet Protocol (IP)-based and / or asynchronous transfer mode (ATM)-based networks. In some embodiments, the network 104 may process voice traffic (e.g., via a Voice over IP (VOIP) network), web traffic (e.g., hypertext transfer protocol (HTTP) traffic and hypertext markup language (HTML) traffic), and / or other network traffic, depending on the particular embodiment and / or the devices of the system 100 communicating with each other.In various embodiments, network 104 may include analog or digital wired and wireless networks (e.g., IEEE 802.11 networks, Public Switched Telephone Networks (PSTN), Integrated Services Digital Networks (ISDN), and Digital Subscriber Lines (xDSL)), 3G mobile networks, 4G mobile networks, 5G mobile networks, wired Ethernet networks, private networks (e.g., intranets), radio, television, cable, satellite, and / or any other delivery or tunneling mechanisms for carrying data, or any suitable combination of such networks. Network 104 may enable connectivity between various devices / systems 102, 106, and 108 of system 100. It should be understood that various devices / systems 102, 106, and 108 can communicate with each other via different networks 104, depending on the source and / or destination device / system 102, 106, and 108.

[0037] It should be understood that in some embodiments, the cloud-based system 102 may be communicatively coupled to the contact center system 106, may form part of the contact center system 106, and / or may be used in conjunction with the contact center system 106 in other ways. For example, the contact center system 106 may include a chatbot (for example, similar to the chatbot 218 in Figure 2) configured to communicate with users (for example, via a user device 108). Furthermore, in some embodiments, the user device 108 may communicate directly with the cloud-based system 102.

[0038] The contact center system 106 can be embodied as any system capable of providing contact center services (e.g., call center services) to end users (e.g., contact center clients) and performing functions otherwise described herein. Depending on the particular embodiment, the contact center system 106 may be located on the premises / campus of the organization utilizing the contact center system 106, and / or remotely from the organization (e.g., in a cloud-based computing environment). In some embodiments, a portion of the contact center system 106 may be located on the premises / campus of the organization, while other portions of the contact center system 106 may be located remotely from the organization. Thus, the contact center system 106 may be deployed in equipment dedicated to the organization or its third-party service providers, and / or in a remote computing environment such as a private or public cloud environment with infrastructure to support multiple contact centers for multiple enterprises, for example. In some embodiments, the contact center system 106 includes resources (e.g., workers, computers, and telecommunications equipment) to enable the provision of services via telephone and / or other communication mechanisms. Such services may include, for example, technical support, help desk support, emergency response, and / or other contact center services, depending on the type of contact center.

[0039] User device 108 can be embodied as any type of device capable of running an application and performing functions otherwise described herein. For example, in some embodiments, user device 108 is configured to run an application for participating in a conversation with a personal bot, an automated agent, a chatbot, or other automated system. Thus, user device 108 may have various input / output devices that the user can interact with and provide and receive audio, text, video, and / or other forms of data. It should be understood that the application can be embodied as any type of application suitable for performing functions described herein. Specifically, in some embodiments, the application can be embodied as a mobile application (e.g., a smartphone application), a cloud-based application, a web application, a thin client application, and / or another type of application. For example, in some embodiments, the application may function as a client-side interface (e.g., via a web browser) for a web-based application or service.

[0040] It should be understood that each of the cloud-based system 102, network 104, contact center system 106, and / or user device 108 may be embodied (and / or include) one or more computing devices similar to the computing device 300 described below with reference to Figure 3. For example, in an exemplary embodiment, each of the cloud-based system 102, network 104, contact center system 106, and / or user device 108 may include a processing device 302 and a memory 306 that stores operational logic 308 (e.g., multiple instructions) that the processing device 302 is to execute for the operation of the corresponding device.

[0041] Referring here to Figure 2, a simplified block diagram of at least one embodiment of a cloud-based system 200 is shown. The exemplary cloud-based system 200 includes a Vodafone communication device 202, a SIP server 204, a resource manager 206, a media control platform 208, a speech / text analysis system 210, a speech generator 212, a voice gateway 214, a media augmentation system 216, a chatbot 218, and voice data storage 220. The exemplary embodiment in Figure 2 shows only one Vodafone communication device 202, one SIP server 204, one resource manager 206, one media control platform 208, one speech / text analysis system 210, one voice generator 212, one voice gateway 214, one media augmentation system 216, one chatbot 218, and one voice data storage 220. However, in other embodiments, the cloud-based system 200 may include multiple Vodafone communication devices 202, SIP servers 204, resource managers 206, media control platform 208, speech / text analysis system 210, voice generator 212, voice gateway 214, media augmentation system 216, chatbots 218, and / or voice data storage 220. For example, in some embodiments, multiple chatbots 218 may be used to communicate on different subjects processed by the same cloud-based system 200. Furthermore, in some embodiments, one or more of the components described herein may be excluded from the system 200, one or more of the components described as independent may form part of another component, and / or one or more of the components described as forming part of another component may be independent.

[0042] The border communication device 202 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in some embodiments, the border communication device 202 may be configured to control signaling and media streams involved in, for example, the setup, execution, and teardown of voice conversations and other media communications between an end user and a contact center system. In some embodiments, the border communication device 202 may be a session border controller (SBC) that controls the signaling and media exchanged during a media session (also referred to as a “call,” “telephone call,” or “communication session”) between an end user and a contact center system. In some embodiments, the signaling exchanged during a media session may include SIP, H.323, Media Gateway Control Protocol (MGCP), and / or any other voice-over IP (VoIP) call signaling protocol. The media exchanged during a media session may include media streams that carry the audio, video, or other data of the call along with call statistics and quality information.

[0043] In some embodiments, the Vodafone communication device 202 may operate according to a standard SIP back-to-back user agent (B2BUA) configuration. In this regard, the Vodafone communication device 202 may be inserted into the signaling and media paths established between the caller and the called party in a VoIP call. It should be understood that in some embodiments, other intermediate software and / or hardware devices may be invoked when establishing the signaling and / or media paths between the caller and the called party.

[0044] In some embodiments, the Vodafone communication device 202 may exercise control over signaling (e.g., SIP messages) and media streams (e.g., RTP data) routed across a network (e.g., network 104) to end-user devices (e.g., user device 108) and contact center systems (e.g., contact center system 106). In this regard, the Vodafone communication device 202 may be coupled to a trunk that carries signals and media for calls to and from user devices over the network, and to a trunk that carries signals and media to and from contact center systems over the network.

[0045] The SIP server 204 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. For example, in some embodiments, the SIP server 204 may operate as a SIP B2UBA and control the flow of SIP requests and responses between SIP endpoints. In other embodiments, any other controller configured to set up and tear down VoIP communication sessions may be contemplated in addition to or instead of the SIP server 204. The SIP server 204 may be a separate logical component or may be combined with the resource manager 206. In some embodiments, the SIP server 204 may be hosted in a contact center system (e.g., contact center system 106). While the SIP server 204 is used in exemplary embodiments, in other embodiments, another call server configured with another VoIP protocol, such as the H.232 protocol, Media Gateway Control Protocol, Skype protocol, and / or other preferred technologies, may be used in addition to or instead of SIP.

[0046] The resource manager 206 may be embodied as any one or more types of devices / systems capable of performing the functions described herein. In exemplary embodiments, the resource manager 206 may be configured to allocate and monitor a pool of media control platforms to provide load balancing and high availability for each resource type. In some embodiments, the resource manager 206 may monitor and select a media control platform 208 from a cluster of available platforms. The selection of the media control platform 208 may be dynamic, for example, based on the identification of the calling end-user's location, the type of media service to be rendered, the detected quality of the current media service, and / or other factors.

[0047] In some embodiments, the resource manager 206 may be configured to process requests for media services and, for example, interact with a configuration server having a configuration database to determine service profiles that can deliver interactive voice response (IVR) profiles, voice applications (e.g., Voice Extensible Markup Language (Voice XML) applications), announcements, and conferencing applications, resources, and services such as a media control platform. According to some embodiments, the resource manager may provide a hierarchical multi-tenant configuration to a service provider, enabling the service provider to allocate a selected number of resources to each tenant.

[0048] In some embodiments, the resource manager 206 may be configured to operate as a SIP proxy, SIP registrar, and / or SIP notifier. In this regard, the resource manager 206 may act as a proxy for SIP traffic between two SIP components. As a SIP registrar, the resource manager 206 may accept registration of various resources, for example, via SIP REGISTER messages. In this way, the cloud-based system 200 may support transparent relocation of call processing components. In some embodiments, components such as the media control platform 208 do not register with the resource manager 206 at startup. The resource manager 206 may discover instances of the media control platform 208 through configuration information retrieved from a configuration database. If the media control platform 208 is configured for monitoring, the resource manager 206 may monitor resource health, for example, by using SIP OPTIONS messages. In some embodiments, to determine whether resources in a group are alive, the resource manager 206 may periodically send SIP OPTIONS messages to each media control platform 208 resource in the group. If the resource manager 206 receives an OK response, the resource is considered alive. It should be understood that the resource manager 206 can be configured to perform various other functions that are omitted for brevity. The resource manager 206 and the media control platform 208 are sometimes collectively referred to as the media controller.

[0049] In some embodiments, the resource manager 206 may act as a SIP notifier, for example, by accepting SIP SUBSCRIBE requests from the SIP server 204 and maintaining multiple independent subscriptions for the same or different SIP devices. Subscription notifications are targeted to tenants managed by the resource manager 206. In this role, the resource manager 206 may periodically generate SIP NOTIFY requests to subscribers (or tenants) regarding port usage and the number of available ports. The resource manager 206 may support multi-tenancy by sending notifications that include the tenant name, the current status (in or out of service) of the media control platform 208 associated with the tenant, and the tenant's current capacity.

[0050] The media control platform 208 may be embodied as any service or system capable of providing media services and performing functions otherwise described herein. For example, in some embodiments, the media control platform 208 may be configured to provide call and media services upon request from a service user. Such services may include, but are not limited to, initiating outbound calls, playing music or other media while a call is on hold, call logging, conference, call progress detection, playing audio / video prompts during a customer self-service session, and / or other call and media services. One or more of these services may be defined by a voice application (e.g., a VoiceXML application) that is performed as part of the process of establishing a media session between the media control platform 208 and the end user.

[0051] The speech / text analytics system (STAS) 210 may be embodied as any service or system capable of providing various speech analysis and text processing functions (e.g., text-to-speech) and performing functions otherwise described herein, as will be understood by those skilled in the art. The speech / text analytics system 210 may perform automated speech and / or text recognition and grammar matching for end-user communication sessions processed by a cloud-based system 200. The speech / text analytics system 210 may include one or more processors and instructions stored in a machine-readable medium that are executed by the processors to perform various operations. In some embodiments, the machine-readable medium may include non-temporary storage media such as hard disks and hardware memory systems.

[0052] The voice generator 212 may be embodied as any service or system capable of generating voice communications and performing functions otherwise described herein. In some embodiments, the voice generator 212 may generate voice communications based on a specific voice signature.

[0053] The voice gateway 214 can be embodied as any service or system capable of performing the functions described herein. In exemplary embodiments, the voice gateway 214 receives or makes calls to voice communication devices, such as end-user devices, and responds to calls according to a voice program corresponding to the communication routing configuration of the contact center system. In some embodiments, the voice program may include a voice avatar. The voice program may be accessed from local memory within the voice gateway 214 or from other storage media within the cloud-based system 200. In some embodiments, the voice gateway 214 may process a voice program that is a script-based voice application. Thus, the voice program may be a script written in a scripting language such as Voice Enhanced Markup Language (VoiceXML) or speech application language tag (SALT). The cloud-based system 200 can also communicate with voice data storage 220 to read and / or write user interaction data (e.g., state variables for data communication sessions) in a shared memory space.

[0054] The media augmentation system 216 can be embodied as any service or system capable of specifying how parts of the cloud-based system 200 (e.g., the Vodafone communication device 202, the SIP server 204, the resource manager 206, the media control platform 208, the speech / text analysis system 210, the speech generator 212, the voice gateway 214, the media augmentation system 216, the chatbot 218, the voice data storage 220, and / or one or more of those parts) interact with each other, and performing functions otherwise described herein. In some embodiments, the media augmentation system 216 can be embodied as or include an application program interface (API). In some embodiments, the media augmentation system 216 enables the integration of different parameters and / or protocols used with various planned applications and media types utilized within the cloud-based system 200.

[0055] The chatbot 218 can be embodied as any automated service or system that uses automation to interact with end users and perform functions otherwise described herein. For example, in some embodiments, the chatbot 218 may operate as an executable program that can be invoked, for example, in response to a request to a particular chatbot. In some embodiments, the chatbot 218 simulates and processes human (written or spoken) conversations, enabling a human to interact with a digital device as if communicating with another human. In some embodiments, the chatbot 218 may be as simple as a rudimentary program that answers simple queries with single-line responses, or it may be as sophisticated as a digital assistant that learns and evolves to deliver an increasing level of personalization as it collects and processes information. In some embodiments, the chatbot 218 includes and / or leverages artificial intelligence, adaptive learning, bots, cognitive computing, and / or other automation technologies. The chatbot 218 may also be referred to herein as one or more chat robots, AI chatbots, automated chat robots, chatterbots, dialogue systems, conversational agents, automated chat resources, and / or bots.

[0056] The advantage of using automated chatbots to engage in chat conversations with end users is that it can help contact centers use valuable and costly resources, such as human resources, more efficiently while maintaining end-user satisfaction. For example, a chatbot may be called in to initially handle a chat conversation without the human end user knowing they are speaking with a robot. The chat conversation can then be escalated to human resources where appropriate. Thus, human resources do not need to be unnecessarily tied down to handling simple requests, but can instead be used more effectively to handle more complex requests or to monitor the progress of many different automated communications simultaneously.

[0057] As described herein, in exemplary embodiments, the chatbot 218 may be embodied as a knowledge-only bot that relies on a knowledge base created using, for example, organizational FAQs, product documentation, user manuals, and / or other relevant documents.

[0058] The voice data storage 220 may be embodied as one or more databases, data structures, and / or data storage devices that can store data in the cloud-based system 200 or otherwise facilitate the storage of such data for the cloud-based system 200. For example, in some embodiments, the voice data storage 220 may include one or more cloud storage buckets. In other embodiments, it should be understood that the voice data storage 220 may additionally or alternatively include other types of voice data storage mechanisms that enable the cloud-based system 200 to dynamically scale the amount of data storage available. In some embodiments, the voice data storage 220 may store scripts (e.g., pre-programmed scripts). Although the voice data storage 220 is described herein as data storage and databases, it should be understood that the voice data storage 220 may include both a database (or other organized collection of data and structures) and data storage for the actual storage of the underlying data. The voice data storage 220 may store a variety of data useful for performing the functions described herein.

[0059] Referring here to Figure 3, a simplified block diagram of at least one embodiment of computing device 300 is shown. Exemplary computing device 300 represents at least one embodiment of a cloud-based system, a contact center system, and / or user device that may be used in connection with the cloud-based system 102, a contact center system 106, and / or user device 108 (and / or parts thereof) illustrated in Figure 1. Furthermore, in some embodiments, one or more of the following may be embodied as a computing device similar to computing device 300, or may be run by such a device: a vodafone communication device 202, a SIP server 204, a resource manager 206, a media control platform 208, a speech / text analysis system 210, a speech generator 212, a voice gateway 214, a media augmentation system 216, a chatbot 218, and / or voice data storage 220 (and / or parts thereof). Depending on the particular embodiment, the computing device 300 may be embodied as a server, desktop computer, laptop computer, tablet computer, notebook, netbook, Ultrabook®, mobile phone, mobile computing device, smartphone, wearable computing device, personal digital assistant, Internet of Things (IoT) device, processing system, wireless access point, router, gateway, and / or any other computing, processing, and / or communication device capable of performing the functions described herein.

[0060] The computing device 300 includes a processing device 302 that executes algorithms and / or processes data according to operational logic 308, an input / output device 304 that enables communication between the computing device 300 and one or more external devices 310, and a memory 306 that stores data received from the external devices 310, for example, via the input / output device 304.

[0061] The input / output device 304 enables the computing device 300 to communicate with the external device 310. For example, the input / output device 304 may include a transceiver, a network adapter, a network card, an interface, one or more communication ports (e.g., a USB port, a serial port, a parallel port, an analog port, a digital port, VGA, DVI, HDMI, FireWire, CAT5, or any other type of communication port or interface), and / or other communication circuits. The communication circuits of the computing device 300 may be configured to perform such communication depending on the specific computing device 300 using any one or more communication technologies (e.g., wireless or wired communication) and associated protocols (e.g., Ethernet, Bluetooth®, Wi-Fi®, WiMAX, etc.). The input / output device 304 may include hardware, software, and / or firmware suitable for implementing the technologies described herein.

[0062] The external device 310 may be any type of device that enables data to be input to or output from the computing device 300. For example, in various embodiments, the external device 310 may be embodied as a cloud-based system 102, a contact center system 106, a user device 108, and / or a part thereof. Furthermore, in some embodiments, the external device 310 may be embodied as another computing device, a switch, a diagnostic tool, a controller, a printer, a display, an alarm, a peripheral device (e.g., a keyboard, a mouse, a touchscreen display, etc.), and / or any other computing, processing, and / or communication device capable of performing the functions described herein. Furthermore, it should be understood that in some embodiments, the external device 310 may be integrated with the computing device 300.

[0063] The processing device 302 may be embodied as any type of processor capable of performing the functions described herein. Specifically, the processing device 302 may be embodied as one or more single-core or multi-core processors, microcontrollers, or other processors or processing / control circuits. For example, in some embodiments, the processing device 302 may include, or be embodied as, an arithmetic logic unit (ALU), a central processing unit (CPU), a digital signal processor (DSP), and / or another suitable processor. The processing device 302 may be of a programmable type, a dedicated hardwired state machine, or a combination thereof. A processing device 302 having multiple processing units may, in various embodiments, utilize distributed processing, pipelined processing, and / or parallel processing. Furthermore, the processing device 302 may be dedicated solely to performing the operations described herein, or it may be used in one or more additional applications. In exemplary embodiments, the processing device 302 is programmable and executes algorithms and / or processes data according to operational logic 308 defined by programming instructions (such as software or firmware) stored in memory 306. Additionally or alternatively, the operational logic 308 of the processing device 302 may be defined at least in part by hardwired logic or other hardware. Furthermore, the processing device 302 may include one or more components of any type suitable for processing signals received from the input / output device 304 or from other components or devices and providing a desired output signal. Such components may include digital circuits, analog circuits, or a combination thereof.

[0064] Memory 306 may be one or more types of non-temporary computer-readable media, such as solid-state memory, electromagnetic memory, optical memory, or a combination thereof. Furthermore, memory 306 may be volatile and / or non-volatile, and in some embodiments, part or all of memory 306 may be of a portable type, such as disks, tapes, memory sticks, cartridges, and / or other suitable portable memory. During operation, memory 306 may store various data and software used during the operation of the computing device 300, such as operating systems, applications, programs, libraries, and drivers. It should be understood that in addition to storing programming instructions that define the operational logic 308, memory 306 may also store data operated by the operational logic 308 of the processing device 302, such as data representing signals received from and / or transmitted to the input / output device 304. As shown in Figure 3, memory 306 may be included in and / or coupled to the processing device 302, depending on the particular embodiment. For example, in some embodiments, the processing device 302, the memory 306, and / or other components of the computing device 300 may form part of a system-on-a-chip (SoC) and be incorporated into a single integrated circuit chip.

[0065] In some embodiments, various components of the computing device 300 (e.g., the processing device 302 and the memory 306) may be communicatively coupled via an input / output subsystem, which may be embodied as circuits and / or components for facilitating input / output operations with the processing device 302, the memory 306, and other components of the computing device 300. For example, the input / output subsystem may be embodied as a memory controller hub, an input / output control hub, a firmware device, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems for facilitating input / output operations, or may otherwise include them.

[0066] The computing device 300 may include other or additional components, such as those commonly found in typical computing devices (e.g., various input / output devices and / or other components) in other embodiments. It should be further understood that one or more components of the computing device 300 described herein may be distributed across multiple computing devices. In other words, the techniques described herein may be employed by a computing system comprising one or more computing devices. Additionally, while only a single processing device 302, I / O device 304, and memory 306 are shown exemplary in Figure 3, it should be understood that a particular computing device 300 may, in other embodiments, include multiple processing devices 302, I / O devices 304, and / or memory 306. Furthermore, in some embodiments, two or more external devices 310 may communicate with the computing device 300.

[0067] Referring here to Figures 4-6, when in use, System 100, or more specifically, the cloud-based System 102, may execute a system flow 400 for leveraging a global confidence classifier for information retrieval in a contact center. Certain blocks of the system flow 400 are illustrated as examples, and it should be understood that such blocks may be combined, divided, added, removed, and / or rearranged, in whole or in part, depending on the particular embodiment, unless otherwise stated.

[0068] It should be understood that the system flow 400 can be described in stages distributed across Figures 4 to 6. Specifically, the portion of the system flow 400 shown in Figure 4 may be the training stage, the portion shown in Figure 5 may be the inference stage, and the portion shown in Figure 6 may be the global document collection phase. Furthermore, the portions of the system flow 400 shown in Figures 4 and 5 may be described as a knowledge-only bot framework, and the portion shown in Figure 6 may be described as a global confidence classifier. In some embodiments, the knowledge-only bot framework may be deployed across multiple organizations.

[0069] As shown in Figure 4, in document collection phase 402, system 100 collects organizational FAQs, product documentation, user manuals, and / or other relevant documents from a document warehouse or data storage. In some embodiments, one or more of the documents include questions, answers, alternative questions, similar questions, categories, and / or other relevant data.

[0070] In the Natural Language Processing (NLP) phase 404, System 100 uses natural language understanding / processing techniques to clean documents and / or segment data. Natural language should be understood as the way humans communicate with each other (e.g., speech and text). In some embodiments, NLP may include the use of statistical methods, machine learning (e.g., deep learning neural networks), and / or other techniques for performing inference on a particular task and / or developing a robust end-to-end system. In some embodiments, in the NLP phase 404, System 100 processes the questions, answers, and alternatives for each document and cleans the data (e.g., by removing unwanted characters / words, removing stop words, tokenization, word stemming, and / or other natural language processing).

[0071] In document classifier phase 406, system 100 trains a machine learning model to classify documents. It should be understood that a classifier in machine learning is an algorithm that automatically orders or classifies data into one or more classes. For example, an email classifier might scan emails and filter them by class label: spam or non-spam. In exemplary embodiments, the machine learning model predicts input features (e.g., TF-IDF scores) and a target (e.g., document identifiers). Since latency may be critical, in some embodiments, system 100 leverages a linear classifier to learn the association between input features and target document identifiers. In some embodiments, the model may be used to identify a subset of documents (e.g., the top N elements) that are most relevant to the query. In other embodiments, the document classifier may include creating a searchable index for all documents based on individual keywords or combinations of words. It should be understood that document indexing allows searching for documents using text contained within them (e.g., including complete phrases / clauses). Full-text indexing can treat the entire content within a document as a single piece of text, whereas field-based indexing can include tagging using metadata (e.g., information about the data itself), which helps in searching for documents based on search characteristics such as document type, creation data, and / or other document-related parameters.

[0072] In feature extraction phase 408, system 100 converts the clean text into one or more feature vectors (i.e., converts the text format into a numerical representation). For example, in some embodiments, word embeddings and / or sentence embeddings may be used. It should be understood that, depending on the particular embodiment, generic and / or custom embeddings may be used. Custom embeddings may be trained from available documents and may capture organization-specific vocabulary (e.g., reducing out-of-lexical issues). There may be assumptions that the numerical representations of the text are similar in high dimensions. Word embeddings convert each word into a vector representation and average across utterances to provide a single vector. Sentence embeddings may directly convert the entire utterance into a single vector based on context. It should be understood that machine learning algorithms may learn from a predefined set of features from training data to generate output for test data. However, machine learning algorithms may function more efficiently using matrix / vector representations than raw text. Therefore, by converting text data into numerical representations, system 100 can utilize machine learning and artificial intelligence techniques more efficiently and effectively.

[0073] It should be understood that the results of the document classifier phase 406 and the feature extraction phase 408 may be stored in the data storage 410. For example, the data storage 410 may store the model from the document classifier, custom embeddings, confidence classifier data, document index, and / or other related data.

[0074] Referring now to Figure 5, as described above, system 100 may perform an inference stage to derive inferences based on user input. Thus, in NLP phase 412 of Figure 5, system 100 may receive a user query to a knowledge-only bot (e.g., "What is the customer care number?") and use natural language understanding / processing techniques to clean the query and / or segment the data in a manner similar to that described with reference to NLP phase 404 of Figure 4.

[0075] In document identification phase 414, system 100 may identify a subset of documents (e.g., the top N elements / documents) most relevant to the user query. In doing so, in some embodiments, system 100 may calculate a TF-IDF score for a given query, process the score through a document classifier, and provide the top N possible documents for the given query. In other embodiments, system 100 may search through a document indexer.

[0076] In the feature extraction phase 416, the system 100 may convert user-input query words into numerical vector representations (e.g., [0.45623, 0.34544, 0.5565, 0.98098, ...]) in a manner similar to that described above with reference to the feature extraction phrase 408 in Figure 4. It should be understood that during the execution of the feature extraction phase 416 and / or the document identification phase 414, the system 100 may retrieve data from and / or update data on the data storage 410 (see Figure 4).

[0077] In the document reranking phase 418, system 100 may rerank the top N documents based on the input query and output the most relevant response. In some embodiments, system 100 generates the necessary input features between a given query and the top N possible documents, processes the features through a confidence classifier, and the confidence classifier outputs a probability for each class. System 100 may generate a confidence score between each query and each possible document. The document with the highest score may be identified as the most likely document (e.g., the most likely question / answer document). It should be understood that system 100 may utilize and / or update a global confidence classifier model 430 (see Figure 6) when reranking documents. In other embodiments, a similarity function (e.g., a cosine similarity function) may be used to generate similarity scores.

[0078] Referring now to Figure 6, as described above, System 100 may perform a global confidence classifier stage to train a confidence classifier (e.g., confidence classifier 428) for use with multiple knowledge bases (e.g., for different organizations, different languages, etc.). As described above, in an exemplary embodiment, the global confidence classifier stage includes training a model to predict five-class probabilities based on five numerical input features.

[0079] In document collection phase 420, system 100 collects organizational FAQs, product documentation, user manuals, and / or other relevant documents from a document warehouse or data storage for multiple organizations. It should be understood that by relying on knowledge bases from multiple different organizations, system 100 gains access to a greater number of documents, alternatives, categories, etc. In NLP phase 422, system 100 may use natural language understanding / processing techniques to clean documents and / or segment data in a manner similar to that described with reference to NLP phase 404 in Figure 4.

[0080] In document classifier phase 424, system 100 may train a machine learning model to classify documents in a manner similar to that described with reference to document classifier phase 406 in Figure 4. In feature extraction phase 426, system 100 may convert the clean text into one or more feature vectors (i.e., convert the text format to a numerical representation) in a manner similar to that described with reference to feature extraction phase 408 in Figure 4.

[0081] In confidence classifier phase 428, system 100 measures the confidence level of each document (e.g., the top N documents). Confidence classifiers are useful when dealing with more than two classes in a classifier, and it should be understood that confidence classifiers evaluate the confidence level of a particular class for a given query. For example, when performing weather classification (e.g., summer, winter, autumn, spring), given weather conditions, a confidence classifier may estimate how confidently the weather conditions correspond to the various classes. In an exemplary embodiment, the confidence classifier is trained as a five-class problem using five input features.

[0082] Input features may include a first alignment score, a second alignment score, a similarity score, a document score, and a word repetition score. The first alignment score (F0) may be the word alignment score between the user query and the document, and the score may be calculated according to the following formula.

[0083]

number

[0084] The second alignment score (F1) is calculated using equations 1-4, which are used to calculate the first alignment score (F0), but with the word vectors of the query and the document swapped. It should be understood that the second alignment score provides how similar the document is to the query. The similarity score between the query and the document is calculated using the cosine similarity to the mean word embedding vectors of the query and the document. The document score is calculated from the document classifier, so that each document predicted by the document classifier is also given a score for that document.

[0085] The overlap score (OS) calculates how many n-grams (e.g., bigrams in this context) are similar between the query and the document. The overlap score (OS) is calculated according to the following formula:

[0086]

number

[0087] As shown above, the exemplary confidence classifier has five classes. In the first class, the predicted document identifier for a given query does not match a predicted document identifier with a very low score (e.g., satisfied because the provided input is insufficient). In the second class, the predicted document identifier for a given query does not match a predicted document identifier with a moderate score. In the third class, the predicted document identifier for a given query does not match a predicted document identifier, but the category of the predicted document is the same with respect to the predicted document identifier category. The document surfaces in the third or fourth position. In the fourth class, the predicted document identifier for a given query does not match a predicted document identifier, but the category of the predicted document is the same with respect to the predicted document identifier category. The document surfaces in the first or second position. In the fifth class, the predicted document identifier for a given query is the same as the predicted document identifier.

[0088] It should be understood that the results of confidence classifier 428 can be used to further update or train the global confidence classifier model 430. Additionally, as shown, the global confidence classifier model 430 can be utilized by the document reranking phase 418 (see Figure 5).

[0089] As shown in the tables in Figures 7 and 8, the global classifier yields improved results compared to conventional methods for classification using knowledge-only bots, both in terms of execution speed / duration and methodological accuracy. The tables illustrate four different variations of knowledge base types and corresponding data: knowledge bases with categories and substitutes, knowledge bases with categories but no substitutes, knowledge bases with substitutes but no categories, and knowledge bases with neither substitutes nor categories. The tables also illustrate results for knowledge bases of different sizes (i.e., 10, 20, 50, 80, 100, and a "full" database with over 100 documents) and different languages ​​(i.e., English and German). As shown, the global confidence classifier using the techniques described herein yielded improved results across virtually all datasets.

Claims

1. A method for utilizing a global confidence classifier for information retrieval in a contact center, wherein the method is The computing system receives user queries from contact center client communications with a bot that possesses only the knowledge of the computing system, The computing system performs feature extraction on the user query by converting the query words of the user query into a numerical vector representation of the user query, The computing system identifies a subset of documents that are most likely to respond to the user query, The computing system includes re-ranking a subset of documents most likely to respond to the user query by inputting input features, at least in part, generated based on the numerical vector representation obtained by the feature extraction, into a global confidence classifier model, The global confidence classifier model is trained based on a confidence classifier that uses at least five input features. A method comprising: a first alignment score which is an indicator of how similar the user query is to a corresponding document; a second alignment score which is an indicator of how similar the corresponding document is to the user query; a cosine similarity score between the user query and the corresponding document; a document score from a document classifier; and a word overlap score which calculates the number of similar n-grams between the user query and the corresponding document.

2. The method according to claim 1, further comprising providing a response to the contact center client via a knowledge-only bot based on the re-ranked subset of documents, in response to the computing system re-ranking the subset of documents that are most likely to respond to the user query.

3. The method according to claim 1, wherein the global confidence classifier model is trained on a set of documents for multiple organizations.

4. The method according to claim 3, wherein the global confidence classifier model is trained on a set of documents in multiple human languages.

5. The method according to claim 1, wherein the global confidence classifier model is trained based on a confidence classifier having multiple classes.

6. The method according to claim 1, wherein the global confidence classifier model is a machine learning model.

7. The method according to claim 1, wherein performing the feature extraction on the user query includes performing the feature extraction on the user query using custom embeddings that are trained from available documents and capable of capturing organization-specific vocabulary.

8. The method according to claim 1, further comprising processing the user query using a computing system to generate processed data, wherein the natural language processing includes at least one of cleaning or segmenting the data of the user query.

9. The method according to claim 8, wherein performing the feature extraction on the user query includes performing the feature extraction on the processed data.

10. The method according to claim 8, wherein identifying the subset of documents most likely to respond to the user query includes identifying the subset of documents based on the processed data.

11. A system for utilizing a global confidence classifier for information retrieval in a contact center, wherein the system At least one processor, A memory containing a plurality of stored instructions, wherein the instructions are, in response to execution by the at least one processor, the system Receiving user queries from a contact center client that communicates with a bot that only has knowledge of the aforementioned system, The process involves performing feature extraction on the user query by converting the query words of the user query into a numerical vector representation of the user query, Identifying a subset of documents most likely to respond to the aforementioned user query, The system includes at least one memory which inputs input features, which are generated based on the numerical vector representation obtained by the feature extraction, into a global confidence classifier model to rerank the subset of documents most likely to respond to the user query, and which performs the following: The global confidence classifier model is trained based on a confidence classifier that uses at least five input features. A system comprising: a first alignment score which is an indicator of how similar the user query is to a corresponding document; a second alignment score which is an indicator of how similar the corresponding document is to the user query; a cosine similarity score between the user query and the corresponding document; a document score from a document classifier; and a word overlap score which calculates the number of similar n-grams between the user query and the corresponding document.

12. The system according to claim 11, wherein the plurality of instructions further cause the system to rerank the subset of documents that are most likely to respond to the user query, and then, via the knowledge-only bot, cause the system to provide a response to the contact center client based on the reranked subset of documents.

13. The system according to claim 11, wherein the global confidence classifier model is trained on a set of documents for multiple organizations in multiple human languages.

14. The system according to claim 11, wherein the global confidence classifier model is a machine learning model.

15. The system according to claim 11, wherein performing the feature extraction on the user query includes performing the feature extraction on the user query using custom embeddings that are trained from available documents and capable of capturing organization-specific vocabulary.

16. The plurality of instructions further cause the system to apply natural language processing to the user query to generate processed data, and the natural language processing includes at least one of cleaning or segmenting the data of the user query. Performing the feature extraction on the user query includes performing the feature extraction on the processed data, The system according to claim 11, wherein identifying the subset of documents most likely to respond to the user query includes identifying the subset of documents based on the processed data.

17. One or more non-temporary machine-readable storage media containing a plurality of stored instructions, wherein the plurality of instructions are, in response to execution by a computing system, to the computing system Receiving user queries from a contact center client that communicates with a bot that only has knowledge of the aforementioned system, The process involves performing feature extraction on the user query by converting the query words of the user query into a numerical vector representation of the user query, Identifying a subset of documents most likely to respond to the aforementioned user query, The system re-ranks the subset of documents most likely to respond to the user query by inputting the input features, which are generated based at least in part on the numerical vector representation obtained by the feature extraction, into a global confidence classifier model. The global confidence classifier model is trained based on a confidence classifier that uses at least five input features. One or more non-temporary machine-readable storage media, wherein the at least five input features include a first alignment score which is an indicator of how similar the user query is to a corresponding document, a second alignment score which is an indicator of how similar the corresponding document is to the user query, a cosine similarity score between the user query and the corresponding document, a document score from a document classifier, and a word overlap score which calculates the number of similar n-grams between the user query and the corresponding document.

Images