Quality assurance system based on speaker intent detection

Abstract

A semantic similarity-based configurable system for automatic scenario detection in customer-agent conversations is disclosed. The system understands intent from semantic similarities in vector space between speaker sentences, does not rely on the use of synonyms, and tolerates a large amount of paraphrasing. This approach is easily scalable to a large number of customers and can be fed more data to increase accuracy and precision. Furthermore, the system is configurable in real time, allowing clients to control which intents are detected and how. In some embodiments, the semantic similarity-based configurable system includes a scenario detection system, a conversation tagging system, a bi-encoder, and a cross-encoder. The scenario detection system receives inputs of sample phrases and customer and agent utterances and generates results. The sample phrases may be phrases and keywords that describe scenarios representing customer or call agent behavior.

Description

[Technical Field]

[0001] REFERENCE TO RELATED APPLICATIONS If an Application Data Sheet (ADS) or PCT Request Form ("Request") was filed as of the filing date of this application, it is incorporated herein by reference. Any application claimed in the ADS, or claim for priority under 35 U.S.C. §§ 119, 120, 121, or 365(c), and any and all parent applications, grandparent applications, great-grandparent applications, etc. of such application, are also incorporated by reference, including any priority claims made in those applications and any material incorporated by reference, to the extent such subject matter is not inconsistent with this specification.

[0002] Additionally, this application is related to the following listed U.S. patent applications from which it claims priority, each of which is incorporated herein by reference in its entirety as if fully set forth herein:

[0003] ●U.S. Provisional Patent Application No. 63 / 442,159 (Docket No. LEV-1002P), entitled "Configurable Intent Phrase Based Quality Assurance Systems," filed January 31, 2023, which describes a semantic similarity-based configurable system for automatic scenario detection in customer-agent conversations.

[0004] • U.S. Provisional Patent Application No. 63 / 521,105 (Docket No. LEV-1001P), entitled "Customer Agent Screen Recording Systems and Methods," filed June 15, 2023, which describes methods for triggered recording and analysis of customer-agent interactions using a customer relationship management (CRM) system.

[0005] ●U.S. Provisional Patent Application No. 63 / 597,343 (Docket No. LEV-1003P), filed November 9, 2023, entitled "Transformer-Based Assistant for Identifying, Organizing, and Responding to Customer Concerns," describes a transformer-based agent assistant system that functions as a machine learning-based customer service tool that analyzes past customer-agent conversations to build a knowledge base of problem-solving steps.

[0006] Copyright and Trade Dress Notice A portion of the disclosure of this patent document contains material that is subject to copyright protection. This patent document may display and / or describe matter that is or may become the trade dress of the Proprietors. The copyright and trade dress owners have no objection to the complete facsimile reproduction of the patent disclosure, as it appears in the U.S. Patent and Trademark Office applications and records, but otherwise reserve all copyright and trade dress rights.

[0007] The present disclosure relates to machine learning, and more particularly to a system that recognizes semantics and identifies intent in conversations. [Background technology]

[0008] The statements in the background of the invention are provided to aid in the understanding of the invention and its applications and uses and may not constitute prior art.

[0009] As companies grow in terms of employees, products, and complexity, maintaining a positive customer-company relationship is crucial. A typical scenario involves a customer contacting a company's customer service hotline. The customer is redirected to a call agent, who assists the customer with their question or concern. Some methods for ensuring a satisfactory customer experience include properly training call agents and understanding the frequency of specific complaints to minimize future occurrences. However, current methods for addressing quality assurance (QA) and gaining conversational insights involve implementing algorithms that monitor calls and use precise keyword matching. For example, to detect customer dissatisfaction, systems might examine client calls for the presence of keywords such as "disgust" or "anger." A drawback of such an approach is that it may not detect certain scenarios, such as customers using synonyms that are not on a preset keyword list (e.g., "I'm frustrated" is a phrase that is not on the list) or customers expressing implicit thoughts (e.g., "It's already January, and I don't see any signs of my paycheck getting paid"). Therefore, customer complaints may be underreported and go unaddressed. Another problem with keyword-based systems is that they must maintain a list of synonyms in a database and continually check for the occurrence of such words during conversations between customers and call agents. When a company receives hundreds of calls per day, such systems are inefficient and unscalable because they cannot exhaustively check all synonyms. Therefore, this approach is always limited by the size and diversity of the keyword list.

[0010] It is from this background that the present invention has been developed. Summary of the Invention

[0011] This Summary of the Invention provides a broad overview of the invention, its applications and uses, and is not intended to limit the scope of the invention, which will become apparent from the Detailed Description when read in conjunction with the drawings.

[0012] Therefore, considering the background technology, developing a scalable, highly accurate system that identifies speaker intent from utterances (call center agents or their customers) based on the semantics of the customer-call agent conversation would be an advancement of the state of the art. Such a system could be implemented by understanding intent from semantic similarities in the vector space between the speaker's sentences, does not rely on the use of synonyms, and tolerates a large amount of paraphrasing. This approach is easily scalable to large numbers of customers and can be fed more data to increase accuracy and precision. Furthermore, developing a system that is configurable in real time, allowing companies to control which intents are detected and how.

[0013] Thus, a semantic similarity-based configurable system for automatic scenario detection in customer-agent conversations is disclosed. The system understands intent from semantic similarities in vector space between speaker sentences, does not rely on the use of synonyms, and tolerates a large amount of paraphrasing. This approach is easily scalable to a large number of customers and can be fed more data to increase accuracy and precision. Furthermore, the system is configurable in real time, allowing clients to control which intents are detected and how. In some embodiments, the semantic similarity-based configurable system includes a scenario detection system, a conversation tagging system, a bi-encoder, and a cross-encoder, where the scenario detection system receives inputs of sample phrases and customer and agent utterances and generates results. The sample phrases may be phrases and keywords that describe scenarios representing customer or call agent behavior.

[0014] Accordingly, various methods, processes, systems, and non-transitory storage media for storing program code for executing a process for determining a scenario name label that best matches an utterance during automatic scenario detection in a customer-agent conversation are provided. In one embodiment, a non-transitory physical storage medium for storing program code is provided. The program code is executable by a hardware processor. Execution of the program code by the hardware processor causes the hardware processor to execute a computer-implemented process for determining a scenario name label that best matches an utterance during automatic scenario detection in a customer-agent conversation. The program code includes: receiving, by an acquisition stage including a bi-encoder neural network, a plurality of scenarios, a plurality of scenario name labels, and a list of a plurality of sample phrases, wherein each scenario in the plurality of scenarios is associated with a scenario name label of the plurality of scenario name labels and associated with a list of sample phrases of the plurality of list of sample phrases; encoding, by the acquisition stage, each sample phrase in the list of the plurality of sample phrases into a phrase encoding to generate a list of a plurality of phrase encodings; generating, by the acquisition stage, the plurality of scenario encodings, wherein each scenario encoding in the plurality of scenario encodings is associated with a scenario of the plurality of scenarios, associated with a scenario name label of the plurality of scenario name labels, and associated with a list of phrase encodings of the list of the plurality of phrase encodings, wherein each scenario encoding in the plurality of scenario encodings is based on normalizing and determining a centroid of the list of phrase encodings associated with a scenario of the plurality of scenarios; storing the list of the plurality of scenario encodings, the plurality of scenario name labels, and the plurality of phrase encodings in a database; receiving, by the acquisition stage, an utterance; encoding, by the acquisition stage, a conversation context vector of the utterance;generating a plurality of similarity scores for a conversational context vector of an utterance, each similarity score in the plurality of similarity scores being associated with a scenario encoding among a plurality of scenario encodings stored in a database; determining, by an acquisition stage, a best-matching scenario encoding from among the plurality of scenario encodings by selecting a scenario encoding among the plurality of scenarios having a highest similarity score among the plurality of similarity scores; and generating, by the acquisition stage, a plurality of ordered pairs, a first component of each ordered pair in the plurality of ordered pairs being the utterance, and a second component of each ordered pair in the plurality of ordered pairs being a phrase encoding from a list of phrase encodings associated with the best-matching scenario encoding. and generating, by a reordering stage including a cross-encoder neural network, a plurality of similarity probabilities, each similarity probability in the plurality of similarity probabilities being associated with an ordered pair of the plurality of ordered pairs; determining, by the reordering stage, whether at least one similarity probability among the plurality of similarity probabilities exceeds a predetermined threshold; assigning, by the reordering stage, to the utterance a best-matching scenario name label from among the plurality of scenario name labels associated with the best-matching scenario encoding if at least one similarity probability among the plurality of similarity probabilities exceeds the predetermined threshold; and assigning, by the reordering stage, an unintended scenario name label to the utterance if none of the similarity probabilities in the plurality of similarity probabilities exceeds the predetermined threshold.

[0015] In one embodiment, the bi-encoder neural network includes a masked and permutation pre-training (MPNet) based model for language understanding, multiple encoder stacks, and a multi-layer perceptron (MLP).

[0016] In one embodiment, the cross-encoder neural network includes a large-scale language model (LLM) based on the Bidirectional Encoder Representation via Transformers (BERT) language model, multiple encoder stacks, and a multi-layer perceptron (MLP).

[0017] In one embodiment, encoding the sample phrases includes generating a vector of real numbers.

[0018] In one embodiment, the plurality of similarity scores is a plurality of cosine similarity scores.

[0019] In one embodiment, the program code further includes code for triggering a conversation tag based on the best matching scenario name label and the plurality of configuration options, the conversation tag including a text string.

[0020] In one embodiment, the plurality of configuration options includes speaker identity, and the program code that triggers the conversation tag is further based on the identity of the speaker of the utterance.

[0021] In one embodiment, the program code that triggers the conversation tag is further based on whether the sequence of agent sentences is followed by a customer sentence.

[0022] In one embodiment, the plurality of configuration options includes speaker behavior, and the program code that triggers the conversation tag is further based on whether the speaker of the utterance mentioned a particular phrase.

[0023] In one embodiment, the plurality of configuration options includes timing, and the program code that triggers the conversation tag is further based on whether the utterance occurs within a predetermined period of time after the conversation begins.

[0024] In another embodiment, a non-transitory physical storage medium is provided that stores program code, the program code being executable by a hardware processor that, when executed, causes the hardware processor to perform a computer-implemented process for determining a scenario name label that best matches an utterance during automatic scenario detection in a conversation between a customer and an agent.The program code includes: receiving an utterance by an acquisition stage including a bi-encoder neural network; encoding, by the acquisition stage, a conversational context vector of the utterance; generating, by the acquisition stage, a plurality of similarity scores for the conversational context vector of the utterance, wherein each similarity score in the plurality of similarity scores is associated with a scenario encoding among a plurality of scenario encodings, associated with a scenario name label among a plurality of scenario name labels, and associated with a list of phrase encodings among a plurality of list of phrase encodings; determining, by the acquisition stage, a best matching scenario encoding from the plurality of scenario encodings by selecting, by the acquisition stage, a scenario encoding among the plurality of scenario encodings having a highest similarity score among the plurality of similarity scores; and generating, by the acquisition stage, a plurality of ordered pairs, wherein a first component of each ordered pair in the plurality of ordered pairs is the utterance and a second component of each ordered pair in the ordered pairs is a phrase encoding from a list of phrase encodings associated with the best-matching scenario encoding; and generating, by a reordering stage including a cross-encoder neural network, a plurality of similarity probabilities, each similarity probability in the plurality of similarity probabilities being associated with an ordered pair of the plurality of ordered pairs; determining, by the reordering stage, whether at least one similarity probability among the plurality of similarity probabilities exceeds a predetermined threshold; assigning, by the reordering stage, a best-matching scenario name label from among the plurality of scenario name labels associated with the best-matching scenario encoding to the utterance if at least one similarity probability among the plurality of similarity probabilities exceeds a predetermined threshold; and assigning, by the reordering stage, an unintended scenario name label to the utterance if none of the similarity probabilities in the plurality of similarity probabilities exceeds the predetermined threshold.

[0025] In yet another embodiment, a computer-implemented method for determining a best-matching scenario name label for an utterance during automatic scenario detection in a customer-agent conversation is provided. The method includes receiving, by an acquisition stage including a bi-encoder neural network, a plurality of scenarios, a plurality of scenario name labels, and a list of a plurality of sample phrases, where each scenario in the plurality of scenarios is associated with a scenario name label of the plurality of scenario name labels and with a list of sample phrases of the plurality of list of sample phrases; encoding, by the acquisition stage, each sample phrase in the list of the plurality of sample phrases into a phrase encoding to generate a list of a plurality of phrase encodings; generating, by the acquisition stage, the plurality of scenario encodings, where each scenario encoding in the plurality of scenario encodings is associated with a scenario of the plurality of scenarios, associated with a scenario name label of the plurality of scenario name labels, and with a phrase encoding of the list of a plurality of phrase encodings. generating, by the acquisition stage, a scenario encoding from the plurality of scenarios having the highest similarity score from the plurality of scenarios;determining a best-matching scenario encoding from among the plurality of scenario encodings; generating a plurality of ordered pairs by an acquisition stage, wherein a first component of each ordered pair in the plurality of ordered pairs is an utterance and a second component of each ordered pair in the plurality of ordered pairs is a phrase encoding from a list of phrase encodings associated with the best-matching scenario encoding; generating a plurality of similarity probabilities by a reordering stage including a cross-encoder neural network, wherein each similarity probability in the plurality of similarity probabilities is associated with an ordered pair in the plurality of ordered pairs; determining by the reordering stage whether at least one similarity probability among the plurality of similarity probabilities exceeds a predetermined threshold; assigning by the reordering stage a best-matching scenario name label from among the plurality of scenario name labels associated with the best-matching scenario encoding to the utterance if at least one similarity probability among the plurality of similarity probabilities exceeds the predetermined threshold; and assigning by the reordering stage an unintended scenario name label to the utterance if none of the similarity probabilities in the plurality of similarity probabilities exceeds the predetermined threshold.

[0026] In one embodiment, the bi-encoder neural network includes a masked and permutation pre-training (MPNet) based model for language understanding, multiple encoder stacks, and a multi-layer perceptron (MLP).

[0027] In one embodiment, the cross-encoder neural network includes a large-scale language Roberta-based model (LLM) based on the Bidirectional Encoder Representation via Transformers (BERT) language model, multiple encoder stacks, and a multi-layer perceptron (MLP).

[0028] In one embodiment, encoding the sample phrases includes generating a vector of real numbers.

[0029] In one embodiment, the plurality of similarity scores is a plurality of cosine similarity scores.

[0030] In one embodiment, the method further includes triggering a conversation tag based on the best-matching scenario name label and the plurality of setting options, the conversation tag comprising a text string.

[0031] In one embodiment, the plurality of configuration options includes speaker identity, and triggering the conversation tag is further based on the identity of the speaker of the utterance and whether the sequence of agent sentences is followed by a customer sentence.

[0032] In one embodiment, the plurality of setting options includes speaker behavior, and triggering a conversation tag is further based on whether the speaker of the utterance mentioned a particular phrase.

[0033] In one embodiment, the plurality of configuration options includes timing, and triggering the conversation tag is further based on whether the utterance occurs within a predetermined period of time after the conversation begins.

[0034] In various embodiments, a computer program product is disclosed that may be used to determine best-matching scenario name labels of utterances during automatic scenario detection in a customer-agent conversation and may include a computer-readable storage medium having program instructions or program code embodied thereon, the program instructions being executable by a processor to cause the processor to perform the steps described above.

[0035] In various embodiments, a system is described that includes a memory that stores computer-executable components and a hardware processor operatively coupled to the memory and that executes the computer-executable components stored in the memory, where the computer-executable components may include components communicatively coupled to the processor that perform the steps described above.

[0036] In another embodiment, the invention is a non-transitory computer-readable storage medium storing executable instructions that, when executed by a processor, cause the processor to perform a process to do something, the instructions causing the processor to perform the steps described above.

[0037] In another embodiment, the present invention is a system for configurable intent phrases based on a quality assurance system as shown and described herein, the system comprising: a user device having a processor, a display, and a first memory; a server having a second memory and a data repository; a communication link between the user device and the server; and a plurality of computer codes embodied in the first memory and the second memory of the user device and the server, the plurality of computer codes, when executed, causing the server and the user device to perform a process including the steps described above.

[0038] In yet another embodiment, the invention is a computerized server comprising at least one processor, a memory, and a plurality of computer codes embodied in said memory, said plurality of computer codes, when executed, causing said processor to perform a process including the steps set forth above. Other aspects and embodiments of the invention include methods, processes, and algorithms including the steps described herein, and also include the processes and modes of operation of the systems and servers described herein.

[0039] Further aspects and embodiments of the present invention will become apparent from the detailed description when read in conjunction with the accompanying drawings. Features described in the context of separate aspects and / or embodiments of the invention may be used together and / or interchangeable where possible. Similarly, features that are described for brevity in the context of a single embodiment may also be provided separately or in any suitable subcombination. Features described in the context of a non-transitory physical storage medium may have corresponding features that can be defined and / or combined with respect to systems and / or methods and / or systems, or vice versa, and these embodiments are specifically contemplated.

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the disclosed embodiments. For the sake of clarity, conciseness, and flexibility, not every element, component, or specification is defined in every drawing. Not every drawing corresponding to a specific step or embodiment of the present invention is drawn to scale. Instead, emphasis is placed on describing the nature, function, and product of the manufacturing methods and devices described herein.

[0041] The embodiments of the invention described herein are illustrative and not limiting. Embodiments will now be described, by way of example, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0042] [Figure 1] 1 illustrates an exemplary high-level diagram of a scenario detection system and a conversation tagging system according to embodiments disclosed herein. [Figure 2] 1 illustrates an exemplary schematic diagram of a scenario detection system according to embodiments disclosed herein. [Figure 3] 1 illustrates an exemplary detailed diagram of a scenario detection system according to an embodiment disclosed herein. [Figure 4]1 shows a schematic diagram of a bi-encoder operation according to an embodiment disclosed herein; [Figure 5] 1 illustrates an exemplary block diagram of a bi-encoder according to embodiments disclosed herein. [Figure 6] 1 shows a schematic diagram of a cross-encoder operation according to an embodiment disclosed herein; [Figure 7] 1 illustrates an exemplary block diagram of a cross-encoder according to embodiments disclosed herein. [Figure 8] 1 shows an exemplary flow diagram detailing the operation of a scenario detection and conversation tagging system according to embodiments disclosed herein. [Figure 9] 10 shows another flow diagram detailing the operation of the scenario detection and conversation tagging system according to embodiments disclosed herein. [Figure 10] 10 shows yet another flow diagram detailing the operation of the scenario detection and conversation tagging system according to embodiments disclosed herein. [Figure 11] 1 shows a system block diagram illustrating a scenario detection and conversation tagging system according to an embodiment disclosed herein. [Figure 12] 1 illustrates a first exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system, in accordance with an exemplary embodiment of the present invention. [Figure 13] 10 illustrates a second exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system, in accordance with an exemplary embodiment of the present invention. [Figure 14] 10 illustrates a third exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system, in accordance with an exemplary embodiment of the present invention. [Figure 15] 10 illustrates a fourth exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system, in accordance with an exemplary embodiment of the present invention. [Figure 16]1 illustrates the operation of a basic neural network for a scenario detection and conversation tagging system, according to an exemplary embodiment of the present invention. [Figure 17] 1 illustrates an overview of a neural network training process that can be used in a scenario detection and conversation tagging system, according to an exemplary embodiment of the present invention. [Figure 18] FIG. 1 is an exemplary flow diagram illustrating the various phases and datasets involved in training a machine learning algorithm for a scenario detection and conversation tagging system, according to an exemplary embodiment of the present invention. [Figure 19] 1 provides an illustrative schematic diagram of a server (administrative computing entity) and a client (user computing entity) used in a scenario detection and conversation tagging system, according to some embodiments of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0043] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without these specific details. In other instances, structures, devices, activities, methods, and processes are shown using schematics, use cases, and / or diagrams to avoid obscuring the present invention. While the following description includes many details for purposes of illustration, those skilled in the art will understand that many variations and / or modifications to the presented details are within the scope of the present invention. Similarly, while many of the features of the present invention are described in terms of or in conjunction with each other, those skilled in the art will understand that many of these features can be provided independently of other features. Thus, this description of the present invention is presented without loss of generality and without imposing limitations on the present invention.

[0044] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly indicates otherwise. Thus, for example, reference to "fiber" includes a single fiber, a mixture of two or more different fibers, etc. Also, as used herein, the term "about" with respect to a measured quantity refers to normal variation in that measured quantity that would be expected by one of ordinary skill in the art when making the measurement and exercising a level of care commensurate with the purpose of the measurement and the precision of the measuring device. In certain embodiments, the term "about" includes the recited number + / - 10%, such that "about 10" includes 9 to 11.

[0045] Scenario detection system and conversation tagging system In the following description, "client" refers to the owner or operator of the system, such as an organization providing a service or product, "customer" refers to the caller (e.g., a user of the service or product), and "agent" refers to the responder (e.g., a customer service representative, account manager, etc.).

[0046] 1 illustrates an exemplary high-level diagram of a scenario detection system 102 and a conversation tag system 104, according to an embodiment disclosed herein. The integrated "scenario and conversation tag" system allows clients to configure various types of events to detect. The "scenario" portion of the integrated system is a behavior detection system, and the "conversation tag" portion of the integrated system is an alarm system. Alarms are triggered contingent on the detection of a scenario and, in some embodiments, a few other configuration options.

[0047] In some embodiments, a "scenario" is defined as the behavior that the detection system 102 detects in each sentence within a given conversation. The system allows the client to describe the scenario using a set of representative phrases. For example, a "customer is disappointed" scenario intended to capture customer dissatisfaction may be described by the following phrases: "I simply hate this," "This has never worked well for me," "Oh no," and "Enough is enough." Additionally, the customer may also add "negative phrases" to the description, which are phrases that seem close to describing the desired scenario, but which the client does not want the scenario to trigger. For example, in the "customer is disappointed" scenario, the client may not want this scenario to trigger a sentence similar to "This is confusing," and therefore, such a sentence may be added to the set of negative phrases.

[0048] By itself, a scenario detected by the sentence detection system 102 may not be displayed to the client. In contrast, a scenario may trigger a "conversation tag," which, depending on set configuration options, may be displayed to the client. In some embodiments, a "conversation tag" is a text string label assigned to a conversation piece from a conversation. A "conversation tag" is assigned to a conversation piece based on the following criteria:

[0049] 1. Who is speaking the conversation part? The user may set this to either "Agent" or "Customer".

[0050] 2. (a) Whether the conversational part conveys a specific intent (as described in a "scenario"). For example, a tag called "stressed customer" may be based on a "scenario" that describes a situation in which a customer is complaining about pricing; or (b) whether the conversation contains any specific keywords;

[0051] Whether the tag should apply to conversation segments that occur at any time during the conversation, or whether the tag should only apply to conversation segments that occur within the first N seconds (where N is set by the user).

[0052] Some possible configuration options are listed below:

[0053] 1. Speaker: Whether the tag should only trigger for the agent's sentence, or the customer's sentence, or the sequence of an agent's sentence followed by a customer's sentence. The last option is called "Dynamic Tags."

[0054] 2. Speaker Action: Whether the tag should trigger when the "speaker" mentions a specific phrase, or whether it should trigger when a specific scenario is detected in the speaker's sentence. Since this is a configurable feature, a "speaker action" can refer to any behavior on the part of the speaker. For example, (A) "complaining about pricing" is an action in which a customer complains about high prices or a better price elsewhere. (B) "appropriate introductory greeting" is an action in which an agent greets a customer by stating, for example, "Thank you for calling ABC Company. How can we help you?" (C) "requesting a refund" is an action in which a customer, for any reason, inquires about the process for returning a product and getting a refund.

[0055] 3. Timing: Whether the tag will trigger all the time during the conversation, or only for the first X seconds of the conversation (where X is preset by the client).

[0056] As shown in FIG. 1 , a client begins by configuring the scenario detection system 102 to include any number of scenarios, e.g., scenario 1 106, scenario 2 108, and scenario m 110. Each scenario encapsulates some behavior of a customer or agent. To configure a scenario, the client provides a list of several (e.g., 5-15) relevant example phrases that capture how the customer or agent exhibits the behavior. For example, to detect the "Customer Disappointment" scenario shown in scenario 1 106, the following phrases are associated: "I don't like this," "It's a hassle," and "That's totally ridiculous." To detect the "Customer Dissatisfaction" scenario, an example phrase might be, "I simply hate your after-sales service."

[0057] Once a set of scenarios and a list of associated phrases are configured in the scenario detection system 102 (behavior detection system), the client can configure the "conversation tag" system 104 (alarm system) to trigger whenever a scenario is detected in a customer / agent's speech. Note that conversation tags are not necessarily contingent on the detection of a scenario; clients may configure them to trigger only under certain circumstances, such as only when certain exact keywords are detected in a customer / agent's speech. For example, Tag 1 112 sets a conversation tag that triggers when Scenario 1 106 is detected, and Tag 2 114 sets a conversation tag that triggers when a particular keyword is detected. Other tags may also be established, such as Tag 3 116 and Tag n 118.

[0058] In some embodiments, triggering the conversation tag is based on the label of the closest scenario and a plurality of configuration options. In some embodiments, the plurality of configuration options includes speaker identity (i.e., whether the speaker is an agent or a customer), and triggering the conversation tag is further based on the identity of the speaker of the utterance. In some embodiments, triggering the conversation tag is further based on whether a sequence of agent sentences is followed by a customer sentence. In some embodiments, the plurality of configuration options includes speaker behavior, and triggering the conversation tag is further based on whether the speaker of the utterance mentioned a particular phrase. In some embodiments, the plurality of configuration options includes timing, and triggering the conversation tag is further based on whether the utterance occurred within a predetermined period of time after the conversation began. Timing refers to whether the tag may trigger at any time during the conversation or whether it should only trigger portions of the conversation that occur within the first N seconds, where N is user configurable.

[0059] Advantages of the integrated scenario detection system 102 and conversation tagging system 104 include (1) the ability to tag based on the sequence of customer and agent scenarios ("dynamic tags"), and (2) the ability to create tags conditional on the presence of one of a set of keywords or the presence of a scenario. By separating the behavior detection (via scenario detection) and alarm systems (conversation tags), clients can independently configure multiple tags for the same underlying scenario but with different configurations. For example, the system may function differently depending on whether the utterance is from a customer or an agent, or whether the system has history regarding a particular customer's behavior or state of mind.

[0060] Acquisition and reordering stages of a scenario detection system FIG. 2 shows an exemplary schematic diagram of a scenario detection system according to embodiments disclosed herein. The scenario detection system includes two stages: an acquisition stage and a reordering stage. The acquisition stage acts as a coarse sieve and, in some embodiments, is implemented with a bi-encoder model 202, a neural network that encodes human language sentences into embedding vectors (or simply "embeddings"), which can be an ordered sequence of real numbers. The reordering stage acts as a fine sieve and, in some embodiments, is implemented with a cross-encoder model 204. This acquisition-reordering framework is commonly used in text-based semantic search; however, it can be adapted for scenario detection as described in this disclosure.

[0061] The scenario detection system operates as follows. First, client sample phrases 206 and their associated scenarios are input to the bi-encoder 202. For example, a scenario "The customer is disappointed" can be described by the following phrases: "I'm not happy with this," "This doesn't work for me," and / or "This is stupid." The bi-encoder 202 encodes such phrases for all scenarios and then stores them in the database 208 (e.g., "phrase encoding"). In some embodiments, scenarios are encoded as normalized centroids of embeddings of all input phrases, and speech segments are encoded during test time as a single normalized embedding. In some embodiments, the centroid of a set of vectors in a vector space is a vector in the vector space that minimizes a weighted sum of the generalized squared distances from each vector in the set of vectors to a point in the vector space. In some embodiments, the distance here is a Euclidean distance. In other embodiments, other geometric shapes are used. In some embodiments, the weighted sum is an equal sum, where each vector is equally weighted. During the training phase, encoding involves: That is, for every scenario, N phrases are obtained from the user, where N can vary from a minimum of 3 to any number of phrases the user may wish to provide. The N phrases are then encoded separately using a bi-encoder, which generates a vector of size [M x 1] (e.g., M = 768) for every phrase. The normalized centroid of all N vectors for this particular scenario (essentially the average of all vectors) is then calculated, and this encoding forms the encoding of the scenario. For example, suppose a user establishes a scenario named "greetings" and provides five associated phrases. All five phrases are then encoded separately to generate five vectors of size [768 x 1], and then the normalized centroid of these five encoded vectors is calculated to form the encoding of the scenario "greetings." During the inference phase, encoding involves the following:That is, when a new query Q is received, it is directly encoded using a bi-encoder to generate a vector of size [M × 1] (e.g., M = 768). Furthermore, the vector is then normalized to run against the scenario phrase and compare the two to find matches.

[0062] Next, a new sentence from the conversation (e.g., "unseen test utterance" 210) is encoded using the bi-encoder 202 to obtain the scenario phrase encoding 208 that is closest to the sentence. For example, the sentences "I expected better service than this" or "I was disappointed" may result in a search for phrases associated with the "The customer was disappointed" scenario. Finally, the cross-encoder 204 calculates a score associated with the closest scenario sentence and all client phrases. If the score exceeds a certain threshold, the label of the closest scenario is assigned to the sentence as a prediction 212. Otherwise, nothing is assigned to the sentence.

[0063] The advantage of our implementation of retrieve-reorder over traditional keyword matching techniques is that retrieve-reorder does not rely on the presence of keywords to detect intent. Instead, it focuses on the semantics of the sentence. That is, it maps each sentence to a list of numbers, or embedding vector, such that the distance between the embedding vectors of sentences with similar meanings is small. This is independent of whether two sentences with similar meanings have the same set of keywords.

[0064] 3 shows an exemplary detailed diagram of a scenario detection system according to an embodiment disclosed herein. The scenario detection system takes action during training time (“Training Time”) 302 (i.e., when a client first sets up a scenario) and during testing time (“Test Time”) 304 (i.e., when the system attempts to detect the presence of a scenario in a customer or agent utterance).

[0065] During training time 302, the bi-encoder 306 populates a list 308 of example phrases that can describe a particular scenario. For example, the phrases "I'm not happy with this," "This doesn't work for me," and "This is stupid" can be used to describe scenarios within the "Customer is disappointed" scenario. The bi-encoder 306 then encodes each phrase provided by the user and stores these phrase embeddings in a database ("phrase encoding") 310. The encoding is performed as described with reference to FIG. 2.

[0066] During test time 304, the system is presented with a random or arbitrary utterance 312 from a customer or agent. The utterance is encoded using a bi-encoder 306. In some embodiments, a first similarity score (e.g., cosine similarity) is calculated between the encoding of the utterance 312 and each stored phrase embedding. In other embodiments, a first similarity score (e.g., cosine similarity) is calculated between the encoding of the utterance 312 and the centroid of the phrase encoding 310 associated with each stored phrase embedding scenario. The phrase embedding scenario with the highest first similarity score to the utterance is retrieved, and a similarity score between the two embeddings is calculated as the cosine distance between the two embedding vectors, resulting in a real number between 0 and 1. If this highest first similarity is below any first threshold score, a "no intent" (i.e., no intent match) is returned. Otherwise (i.e., this highest first similarity exceeds any first threshold score), the cross encoder 314 is used to calculate second similarity scores between the utterance 312 and all phrases belonging to the superordinate intent.

[0067] Regarding "highest" first similarity: For a single conversation part, its embedding is calculated, and then its similarity score with the stored embeddings of all scenarios programmed for this client is also calculated. The scenario (among all scenarios) with the highest similarity score with this conversation part embedding is selected. Regarding the concept of "no intention" or "unintended": Each client programs N intentions, where N is any positive integer, and scenario is synonymous with intention. "No intention" or "unintended" means that the current conversation part does not indicate any of the N intentions programmed by the client. However, in this case, the conversation part may indicate some true intentions that are not in the set of N programmed intentions.

[0068] If the second similarity score with any phrase exceeds any of the second threshold scores, an intent label is obtained and generated. Otherwise, i.e., if the second similarity score does not exceed any of the second threshold scores, "no intent" ("NO_INTENT", i.e., lack of intent, the utterance is unintended) is returned. Regarding the "first threshold score" and "second threshold score," the thresholds are chosen as follows: A subset from the set of utterances is selected as a test set, and the model is run on this test set. Next, various threshold levels are set, and the threshold that produces the best precision and recall on this test set is selected as the appropriate threshold score. After thorough testing with various clients, a specific threshold of 0.55 seems to perform sufficiently well across all clients. Of course, the threshold may be adjusted between clients depending on the quality of precision and recall. One of the performance measures in these experiments is the average F1 score (i.e., the harmonic mean of precision and recall) across all intents present in the test set.

[0069] Bi-encoder 4 shows a schematic diagram of bi-encoder operation during "training time" 402, according to an embodiment disclosed herein. Phrases 410 are input to a bi-encoder 412, which generates phrase embeddings 414. In some embodiments, the bi-encoder 412 includes a transformer encoder 422, which includes a large-scale language model (LLM) trained with a contrastive loss. Arbitrary-length sentences 420 are input to the transformer encoder 422, which generates fixed-size embeddings 424.

[0070] The token space of the bi-encoder 412 is a finite, countable set of all tokens, each of which is assigned a unique "token ID." In Transformer terminology, a token is a sequence of characters. During tokenization, a string is represented as a sequence of tokens. For example, "hello world" can be represented as these three tokens: ("he," "-llo," "world"). Thus, each input phrase or sentence is mapped to a single embedding vector. The Transformer Encoder 422 receives the representation of a sentence as a sequence of token IDs and produces a "context embedding," or a sequence of vectors of real numbers. The number of resulting embedding vectors is equal to the number of token IDs in the input sequence. The average over these vectors is then calculated to produce a single vector representing the embedding of a given phrase or sentence. Thus, the Transformer Encoder 422 encodes a sentence into a sequence of vectors that capture the relationships between the various words in the sentence.

[0071] To convert input phrases into phrase embeddings, the bi-encoder 412 uses a transformer encoder architecture containing several layers, each of which contains two main subcomponents: a multi-head self-attention mechanism and a position-wise feedforward network. Each of these subcomponents is followed by residual connections and layer normalization, which help stabilize the training process. The output of the bi-encoder 412 is configured to generate a fixed-length embedding, i.e., vector length 768, regardless of the length of the input phrase or sentence.

[0072] The contrastive loss is calculated via a triplet function. Given an anchor sentence a, a positive sentence p, and a negative sentence n, the triplet loss adjusts the network so that the distance between a and p is smaller than the distance between a and n. Specifically, the following loss function is minimized: max(||sa-sp||-||sa-sn||+epsilon,0), where sx represents the sentence embedding for a, n, and p, and ||·|| represents the distance metric. The margin epsilon ensures that sp is at least closer to sa than sn. The Euclidean distance is used as the metric, and the margin is set to 1.

[0073] FIG. 5 illustrates an exemplary block diagram of a bi-encoder 500 according to embodiments disclosed herein. Bi-encoders belong to a class of transformer-based language models specifically designed to generate sentence embeddings. These tools were first introduced in the paper "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks" by Reimerset et al., published in August 2019, available at https: / / arxiv.org / abs / 1908.10084. These tools were introduced as a modification of pre-trained BERT (Bidirectional Encoder Representations from Transformers) networks that use Siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine similarity 502. A description of BERT can be found in the paper "A Primer in BERTology: What we know about how BERT works" by Rogers et al., available at https: / / arxiv.org / abs / 2002.12327. The goal of these models is to transform sentences into vectors such that similar sentence vectors are close to each other in vector space and dissimilar sentence vectors are far from each other. The models have been successfully applied to many tasks, such as large-scale semantic similarity comparison, clustering, and information retrieval via semantic search. While various embodiments using BERT-based models are described herein, other LLM algorithms or other AI-based approaches are within the scope of the present invention.

[0074] In some embodiments, bi-encoder 500 includes a first component 550, a second component 560, and a third component 570. First component 550 includes one or more MPNet-based models 506, one or more pooling operations 504, and a cosine similarity operation 502, where MPNet stands for "Masked and Permuted Pre-training for Language Understanding" (see Song et al., April 2020 paper, available at https: / / arxiv.org / abs / 2004.09297). Second component 560 includes an input encoding 522, several (e.g., nine) encoder stacks 514 (i.e., a stack of encoders in series, including encoder 512, encoder 516, and encoder 518), and an output encoding state 510. The third component 570 includes layer normalization 530, another layer normalization 538, multi-head attention 534, and multi-layer perceptron (MLP) 528.

[0075] Siamese network architectures can derive fixed-size vectors of input sentences. Similarity measures such as cosine similarity502 or Manhattan / Euclidean distance can be used to find semantically similar sentences. Bi-encoder class models add pooling504 operations to the output of BERT / RoBERTa to derive fixed-size sentence embeddings, where RoBERTa stands for "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (see Liuet, et al., July 2019, available at https: / / arxiv.org / abs / 1907.11692).

[0076] The triplet function trains the network so that the distance between anchor sentence a and positive sentence p is smaller than the distance between a and negative sentence n. The loss function for this triplet function is described above with reference to Figure 4. Among the bi-encoder models available in the Sentence-BERT and HUGGING FACE collaborative platform, the all-mpnet-base-v2 model 506 (see https: / / huggingface.co / sentence-transformers / all-mpnet-base-v2) has been applied to implement the embodiments described in this disclosure. This model was developed using a pre-trained MICROSOFT / mpnet-based model 506 (see https: / / huggingface.co / microsoft / mpnet-base) and fine-tuned on a dataset of 1 billion sentence pairs using the following contrastive learning objective: Given a sentence from a pair, the model predicts which sentence from a set of randomly sampled other sentences was actually paired with that sentence in the dataset. Formally, the cosine similarity 502 from each possible sentence pair from the batch is calculated and then cross-entropy loss is applied by comparing it with that of the true pair. Given an input sentence, it maps the sentence to a multi-dimensional (e.g., 768-dimensional) dense vector that captures the semantic information of the given sentence.

[0077] Cross-entropy loss is calculated as follows: Given an input sentence, the model calculates similarity scores between that input sentence and N other sentences (where N is a positive integer) where one sentence is semantically similar to the input sentence. A softmax operation is applied to these scores to generate a categorical distribution across the N sentences (i.e., the "softmaxed" scores now sum to unity, i.e., 1). To teach the model to maximize the softmaxed score of the sentence that was semantically closest to the input, the cross-entropy loss is calculated as the negative logarithm of the softmaxed score of this closest sentence. Minimizing this cross-entropy loss is equivalent to maximizing the softmaxed score.

[0078] Specifically, the model encodes a given query (i.e., any sentence from a conversation between an agent and a user) and intent phrases (i.e., phrases that describe the intent), and then uses the cosine similarity 502 between these phrases to determine the best candidate intent for the given query.

[0079] Cross Encoder FIG. 6 shows a schematic diagram of cross-encoder operation during “test time” according to embodiments disclosed herein. In some embodiments, the cross-encoder 616 is a large-scale language model (LLM) including a transformer encoder block and classification head trained to detect similarity between two sentences. A phrase 610 is input to the bi-encoder 612, which generates top matches 614. The top matches 614 are sent to the cross-encoder 616, which returns either “no intention” 618 or “intent” 620. The transformer encoder 634 can then accept sentences (e.g., sentence A 630 and sentence B 632) and generate a similarity score 636.

[0080] Since the model's output is a similarity metric, the cross-encoder takes two sentences as input and produces an output x, where x is a real number between 0 and 1. The value of x represents the probability of similarity between two given inputs, where 0 represents dissimilarity and 1 represents identical inputs. A threshold mechanism on the output x given to the model determines whether a match is "intent" or "no intent." Heuristically, a threshold of 0.55 seems appropriate for many applications. For example, suppose the intent match from the bi-encoder phase is "greetings." Then, all phrases in the intent "greetings" are paired with the query. Therefore, if there are five phrases associated with the intent "greetings," five phrase-query pairs (e.g., [phrase_a,query], [phrase_b,query]) are generated. These are sent to the cross-encoder, which determines a score for each phrase-query pair. If any of these five scores is above a threshold (e.g., 0.55), the query is given the intent "greetings." Otherwise, the query is said to have "no intent." The classification head contains a simple feedforward layer, takes the embeddings from the final transformer encoder layer, and produces a single number that represents the probability of how similar both input sentences are.

[0081] FIG. 7 illustrates an exemplary block diagram of a cross-encoder according to embodiments disclosed herein. A cross-encoder model also belongs to the transformer-based language model class. However, unlike a bi-encoder, which generates a sentence embedding for a given sentence, a cross-encoder generates an output value (e.g., between 0 and 1) indicating the similarity of an input sentence pair. While a user or system can provide sentences A and B to a bi-encoder alone, resulting in respective sentence embeddings u and v, in the case of a cross-encoder, a user or system simultaneously provides two sentences by pairing them. A cross-encoder does not produce sentence embeddings. While a cross-encoder generally produces better results than a bi-encoder in terms of sentence similarity, it can be computationally expensive, making it impractical for very large-scale semantic search-type operations. A cross-encoder is often used whenever a user has a predefined set of sentence pairs to score. For example, a user desires similarity scores for 100 sentence pairs.

[0082] Among the cross-encoder models available on the Sentence-BERT and HUGGING FACE collaborative platforms, the stsb-roberta-base model 702 (see https: / / huggingface.co / cross-encoder / stsb-roberta-base) has been applied to implement the present disclosure. This model was developed using the pre-trained Roberta-based model 702 (see https: / / huggingface.co / roberta-base) and fine-tuned on a sentence similarity dataset. Given a sentence pair, the model's intended output is a value between 0 and 1, where values ​​closer to 0 indicate dissimilarity between the two sentences and values ​​closer to 1 indicate similarity between the two sentences. Specifically, the model helps to verify whether the candidates tagged by the bi-encoder as intents for a given query properly belong to that particular intent class.

[0083] In some embodiments, the cross-encoder 700 includes a first component 740, a second component 750, a third component 760, and a fourth component 770. The first component 740 includes a linear layer 744 that accepts CLS ("classification") token embeddings and a sigmoid operator 742. The second component 750 includes a Roberta-based model 702 that accepts a sentence (e.g., sentence A, sentence B) and a classification head 704 that outputs a value 706 (e.g., between 0 and 1). The third component 760 includes an input encoding 722, several (e.g., nine) encoder stacks 714 (i.e., a stack of encoders in series, including encoder 712, encoder 716, and encoder 718), and an output encoding state 710. The fourth component 770 includes a layer normalization 730, another layer normalization 738, multi-head attention 734, and a multi-layer perceptron (MLP) 728.

[0084] The bi-encoder and cross-encoder coordinate their actions, with the acquisition stage using the bi-encoder and the reordering stage using the cross-encoder. In particular, the bi-encoder model is trained with cosine similarity loss and strives to generate high-quality sentence embeddings. In this stage, the system marks the previous query as a specific intent among a set of intents. This step can be thought of as an "intent-intent classifier," i.e., its role is to determine which intent a new query belongs to. The reordering stage then becomes the final decision maker. The model distinguishes between "intent" and "no intent" ("NO_INTENT," i.e., lack of intent, the utterance is unintended). However, if there is no model trained for that specific task, a proxy task is performed instead, in which phrases of a given intent are compared with the previous query. Based on which phrase is most similar, "intent" or "no intent" ("NO_INTENT," i.e., lack of intent, the utterance is unintended) is determined.

[0085] While the concept of semantic search is limited to information retrieval and knowledge retrieval, this disclosure uses embedding-based search as a pseudo-task for intent detection. In particular, the system does not directly match queries to phrases in the bi-encoder stage; instead, it matches query embeddings to intent embeddings. The intent embedding is found by taking the centroid of all phrase embeddings for that particular intent. While bi-encoders are commonly used in information retrieval or knowledge retrieval to fetch the top few (e.g., 5-10) matching sentences or paragraphs, this disclosure typically returns only the top selections. Furthermore, a threshold, i.e., a hyperparameter, for the similarity between the query and the intent centroid may be used. In some embodiments, the threshold is based on a validation set that maximizes micro-average precision and micro-average recall.

[0086] Finally, while a cross-encoder is typically used to reorder and reorder the predictions provided by the bi-encoder to generate relevant outputs on top, in this disclosure, we use the similarity output of the cross-encoder as a "pseudo" to determine "intended" versus "not intended" ("NO_INTENT", i.e., lack of intent, the utterance is unintended) given the intent recommendation from the bi-encoder. A query is classified as "intended" or "not intended" ("NO_INTENT", i.e., lack of intent, the utterance is unintended) using a threshold on the similarity score from the cross-encoder. Again, this threshold is a hyperparameter and can be obtained by optimizing micro-average precision and micro-average recall on the validation set.

[0087] FIG. 8 shows an exemplary flow diagram detailing the operation of a scenario detection and conversation tagging system according to embodiments disclosed herein. For each scenario 800 associated with a client, the operation performs steps 802, 804, 806, and 808. In step 802, the system initializes the scenario by associating it with a name label (e.g., a text string) and n phrases, where n is any positive integer. The number of n can vary from scenario to scenario. In step 804, the system uses a bi-encoder to encode all n phrases into n embedding vectors (i.e., "phrase encoding"). In step 806, the system normalizes the n embedding vectors and determines their centroids to generate a scenario embedding vector (i.e., "scenario encoding"). Finally, in step 808, the system stores the scenario embedding vectors in a database. In some embodiments, in step 808, the system stores a list of the scenario embedding vector, the name label, and the n embedding vectors as an associated group in the database.

[0088] Next, for each utterance 810 in a conversation between an agent and a customer associated with the client, and for each defined scenario, the operations perform steps 812, 814, 816, 818, 820, 822, 824, 826, and 828. In some embodiments, the system decomposes the conversation into a set of utterances. That is, the conversation includes the utterance, and the bi-encoder is ready to receive the utterance. In alternative embodiments, the conversation has already been pre-decomposed into a set of utterances. For each utterance 810, the following steps are performed: In step 812, the bi-encoder receives the utterance. In step 814, the system uses the bi-encoder to encode the utterance into a conversation context vector. In step 816, the system compares the conversation context vector with scenario embedding vectors stored in the database and generates a similarity score for each scenario embedding vector stored in the database. In some embodiments, the similarity score is a cosine similarity score. In step 818, the system uses the bi-encoder to determine a best-match scenario embedding vector, i.e., the scenario embedding vector that best matches the conversation context vector of the utterance. In some embodiments, determining the best match involves identifying the scenario embedding vector with the highest similarity score. In step 820, the system pairs the utterance with each of the n phrases associated with the best-matching scenario embedding vector to generate ordered pairs such as "(utterance A, phrase 1 of matched scenario)," "(utterance A, phrase 2 of matched scenario)," and "(utterance A, phrase 3 of matched scenario)." In step 822, the system uses a cross-encoder to generate a similarity probability for each generated ordered pair of (utterance, phrase). In some embodiments, this similarity probability corresponds to the second similarity score referenced in FIG. 3. In step 824, the system determines whether the probability for at least one phrase associated with the best-matching scenario embedding vector exceeds a predetermined threshold (e.g., 0.55).If yes (step 826), the bi-encoder is deemed "correct" and the best-matching scenario name label associated with the best-matching scenario embedding vector is assigned to the utterance. In some embodiments, other information associated with the best-matching scenario, such as intent, is also assigned to the utterance. Otherwise (step 828), the bi-encoder is deemed "incorrect" and an unintended name label (i.e., "no intent") is assigned to the utterance.

[0089] 9 shows another flow diagram detailing the operation of the scenario detection and conversation tagging system according to embodiments disclosed herein. In particular, the flow diagram determines the best-matching scenario name label of an utterance during automatic scenario detection in a customer-agent conversation.

[0090] The operation begins in step 901 .

[0091] In step 902, the system receives, by an acquisition stage including a bi-encoder neural network, a plurality of scenarios, a plurality of scenario name labels, and a list of a plurality of sample phrases, where each scenario in the plurality of scenarios is associated with a name label from the plurality of name labels and with a list of sample phrases from the list of a plurality of sample phrases.

[0092] In step 904, the system, via an acquisition stage, encodes each sample phrase in the list of sample phrases into a phrase encoding to generate a list of phrase encodings.

[0093] In step 906, the system generates, by an acquisition stage, a plurality of scenario encodings, each scenario encoding in the plurality of scenario encodings associated with a scenario from the plurality of scenarios, associated with a name label from the plurality of scenario name labels, and associated with a list of phrase encodings from the plurality of lists of phrase encodings, and each scenario encoding in the plurality of scenario encodings is based on normalizing and determining the center of gravity of the list of phrase encodings associated with the scenario from the plurality of scenarios.

[0094] In step 908, the system stores a list of the plurality of scenario encodings, the plurality of scenario name labels, and the plurality of phrase encodings in a database.

[0095] In step 912, the system receives an utterance via an acquisition stage.

[0096] In step 914, the system encodes the conversational context vector of the utterance through the acquisition stage.

[0097] In step 916, the system generates, via an acquisition stage, a plurality of similarity scores for the conversational context vector of the utterance, each similarity score in the plurality of similarity scores being associated with a scenario encoding from among a plurality of scenario encodings stored in the database.

[0098] In step 918, the system determines a best-matching scenario encoding from among the multiple scenario encodings by selecting, via the acquisition stage, a scenario encoding from the multiple scenarios that has the highest similarity score from the multiple similarity scores.

[0099] In step 920, the system generates a plurality of ordered pairs through an acquisition stage, where a first component of each ordered pair in the plurality of ordered pairs is an utterance and a second component of each ordered pair in the plurality of ordered pairs is a phrase encoding from a list of phrase encodings associated with the best-matching scenario encoding.

[0100] In step 922, the system generates a plurality of similarity probabilities by a reordering stage including a cross-encoder neural network, where each similarity probability in the plurality of similarity probabilities is associated with an ordered pair of the plurality of ordered pairs.

[0101] In step 924, the system determines, via the reordering stage, whether at least one similarity probability of the plurality of similarity probabilities exceeds a predetermined threshold.

[0102] In step 926, the system assigns a best-matching scenario name label from among the multiple scenario name labels associated with the best-matching scenario encoding to the utterance by a reordering stage if at least one similarity probability of the multiple similarity probabilities exceeds a predetermined threshold.

[0103] Finally, in step 928, the system assigns an unintentional scenario name label to the utterance through a reordering stage if none of the similarity probabilities in the plurality of similarity probabilities exceeds a predetermined threshold.

[0104] This completes the operation (step 930).

[0105] 10 shows yet another flow diagram detailing the operation of the scenario detection and conversation tagging system, according to embodiments disclosed herein. In particular, the flow diagram determines the best-matching scenario name label of an utterance during automatic scenario detection in a customer-agent conversation.

[0106] The operation begins at step 1001 .

[0107] In step 1012, the system receives the utterance by an acquisition stage that includes a bi-encoder neural network.

[0108] In step 1014, the system encodes the conversational context vector of the utterance through the acquisition stage.

[0109] In step 1016, the system generates, by an acquisition stage, a plurality of similarity scores for the conversational context vector of the utterance, each similarity score in the plurality of similarity scores being associated with a scenario encoding from the plurality of scenario encodings, associated with a scenario name label from the plurality of scenario name labels, and associated with a list of phrase encodings from the plurality of lists of phrase encodings.

[0110] In step 1018, the system determines a best-matching scenario encoding from among the multiple scenario encodings by selecting, via the acquisition stage, a scenario encoding from the multiple scenarios that has the highest similarity score from the multiple similarity scores.

[0111] In step 1020, the system generates a plurality of ordered pairs through an acquisition stage, where a first component of each ordered pair in the plurality of ordered pairs is an utterance and a second component of each ordered pair in the plurality of ordered pairs is a phrase encoding from a list of phrase encodings associated with the best-matching scenario encoding.

[0112] In step 1022, the system generates a plurality of similarity probabilities by a reordering stage including a cross-encoder neural network, where each similarity probability in the plurality of similarity probabilities is associated with an ordered pair of the plurality of ordered pairs.

[0113] In step 1024, the system determines, via the reordering stage, whether at least one similarity probability of the plurality of similarity probabilities exceeds a predetermined threshold.

[0114] In step 1026, the system assigns a best-matching scenario name label from among the multiple scenario name labels associated with the best-matching scenario encoding to the utterance by a reordering stage if at least one similarity probability of the multiple similarity probabilities exceeds a predetermined threshold.

[0115] Finally, in step 1028, the system assigns an unintentional scenario name label to the utterance through a reordering stage if none of the similarity probabilities in the plurality of similarity probabilities exceeds a predetermined threshold.

[0116] This completes the operation (step 1030).

[0117] 11 shows a system block diagram illustrating a scenario detection and conversation tagging system according to an embodiment disclosed herein. The system includes access to at least one hardware processor 1110 responsible for executing program code 1122 to implement modules 1130, described below. The system includes access to at least one non-transitory physical storage medium 1120 accessible by the at least one hardware processor 1110, which stores program code 1122 executable by the hardware processor 1110. The program code may be stored and distributed among two or more non-transitory physical storage media and executed by two or more processors. The system may further include a graphical user interface (GUI) 1104 that allows a user 1102 (e.g., a client or agent) to interact with the system modules 1130.

[0118] In various embodiments, a scenario detection and conversation tagging system may incorporate the modules and databases described in this disclosure: A client application 1132 for coordinating the scenario detection and conversation tagging system and for performing the various methods described herein; A bi-encoder ML model 1134 trained to generate embeddings or context vectors from sentences, as described in further detail in this disclosure; A cross-encoder ML model 1136 trained to compare embeddings and / or context vectors, as described in further detail in this disclosure; and and a scenario database 1138 containing data associated with one or more scenarios (e.g., 1140, 1150). For any given scenario, the scenario data 1138 includes one centroid embedding vector, one name label, and multiple phrase embedding vectors associated with that scenario. For example, for scenario #1 (1140), FIG. 11 shows centroid embedding vector 1142, name label 1144, and phrase embedding vectors 1146, 1148, etc., and for scenario #2 (1150), FIG. 11 shows centroid embedding vector 1152, name label 1154, and phrase embedding vectors 1156, 1158, etc.

[0119] Graphical User Interface (GUI) implementation 12 illustrates a first exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system in accordance with an exemplary embodiment of the present invention. A default organization 1214 can use the GUI to train the scenario detection and conversation tagging system. The left menu includes options for Home 1202, Analytics 1204, Canvas 1206, Coaching 1208, Calibration 1210, and Settings 1212. Manage Organizational Units 1216 includes options for Users 1218, Teams 1220, Report Sharing 1222, Custom Fields 1224, QA Case Assignment 1226, Notifications 1228, and Audit Log 1230. Scripting Engine 1232 includes options for Scenarios 1234, Conversation Tags 1236, Metric Tags 1238, Sentiment Tags 1240, Categories 1242, Rubric Builder 1244, and Automation 1246. Knowledge Center 1248 includes an optional FAQ 1250.

[0120] Upon selecting scenario 1234, a scenario 1252 window allows the user to enter a scenario name, e.g., "Positive Vacuum Experience" 1254 for a vacuum cleaner product. If no scenarios exist that are similar to the entered scenario, the GUI indicates "No Similar Scenarios Found" 1258. The user may then select 1260 to whom the scenario applies, choosing from customers only 1262, agents only 1264, and both agents and customers 1266. The user is then prompted 1268 to enter the number of training sentences (e.g., 5) that should trigger the scenario. The entered phrase 1270, e.g., "good at picking up dirt," is then displayed. The user may also provide training sentences that should not trigger the scenario 1280.

[0121] The GUI indicates 1274 whether the input scenario matches any of the accessible stored conversations. In the illustrated case, there are zero matches.

[0122] 13 illustrates a second exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system in accordance with an exemplary embodiment of the present invention. A default organization 1314 can use the GUI to train the scenario detection and conversation tagging system. The left menu includes options for Home 1302, Analytics 1304, Canvas 1306, Coaching 1308, Calibration 1310, and Settings 1312. Manage Organizational Units 1316 includes options for Users 1318, Teams 1320, Report Sharing 1322, Custom Fields 1324, QA Case Assignment 1326, Notifications 1328, and Audit Log 1330. Scripting Engine 1332 includes options for Scenarios 1334, Conversation Tags 1336, Metric Tags 1338, Sentiment Tags 1340, Categories 1342, Rubric Builder 1344, and Automation 1346. Knowledge Center 1348 includes an optional FAQ 1350.

[0123] Upon selecting scenario 1334, a scenario 1352 window allows the user to enter a scenario name, e.g., "Greetings" 1354 for a phrase associated with a greeting. If no scenarios exist that are similar to the entered scenario, the GUI indicates "No Similar Scenarios Found" 1358. The user may then select 1360 to whom the scenario applies, choosing from customers only 1362, agents only 1364, and both agents and customers 1366. The user is then prompted 1368 to enter the number of training sentences (e.g., 5) that should trigger the scenario. The entered phrase 1370, e.g., "thank you for calling us," is then displayed, and a relevance 1372 percentage is provided. The user may also provide 1380 training sentences that should not trigger the scenario.

[0124] The GUI indicates 1374 whether the input scenario matches any of the accessible stored conversations. In the illustrated case, the "greetings" scenario matched 2,912 of the 10,000 conversations. The most relevant (e.g., top 10) matches 1376 are shown, e.g., "thanks for calling in," "well, thanks for calling," and so on. A set of near misses 1378, e.g., "thank you so much for calling away," are also displayed.

[0125] 14 illustrates a third exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system, according to an exemplary embodiment of the present invention. A default organization 1414 can use the GUI to train the scenario detection and conversation tagging system. The left menu includes options for Home 1402, Analytics 1404, Canvas 1406, Coaching 1408, Calibration 1410, and Settings 1412. Manage organizational units 1416 includes options for Users 1418, Teams 1420, Report Sharing 1422, Custom Fields 1424, QA Case Assignment 1426, Notifications 1428, and Audit Log 1430. Scripting engine 1432 includes options for Scenarios 1434, Conversation Tags 1436, Metric Tags 1438, Sentiment Tags 1440, Categories 1442, Rubric Builder 1444, and Automation 1446. Knowledge Center 1448 includes an optional FAQ 1450.

[0126] Upon selecting scenario 1434, a scenario 1452 window allows the user to enter a scenario name, e.g., "Register" 1454, a phrase associated with registering a new account with an organization. If no scenarios exist that are similar to the entered scenario, the GUI indicates "No Similar Scenarios Found" 1458. The user may then select 1460 to whom the scenario applies, choosing from customers only 1462, agents only 1464, and both agents and customers 1466. The user is then prompted 1468 to enter the number of training sentences (e.g., 5) that should trigger the scenario. The entered phrase 1470, e.g., "I want to sign up," is then displayed, and a relevance 1472 percentage is provided. The user may also provide 1480 training sentences that should not trigger the scenario.

[0127] The GUI indicates 1474 whether the entered scenario matches any of the accessible stored conversations. In the illustrated case, the "Register" scenario matched 10 conversations out of 10,000. The most relevant (e.g., top 10) matches 1476 are shown, e.g., "trying to register this," "so I need to make an account," etc. A set of close matches 1478, e.g., "let me get you over to our enrollment department," are also displayed.

[0128] FIG. 15 illustrates a fourth exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system in accordance with an exemplary embodiment of the present invention. A default organization can use the GUI to view call history and analysis for a selected customer or agent, e.g., John Smith 1513, as analyzed by the scenario detection and conversation tagging system, as well as information and analysis for a specific conversation, as indicated by “Conversation ID” 1590. The left menu includes options for Home 1502, Analytics 1504, Canvas 1506, Coaching 1508, Calibration 1510, and Settings 1512. Various scores for the selected customer or agent, e.g., QA Score 1516, Insta Score 1518, CSAT Score 1520, and Sentiment Score 1522, are provided. A call rubric 1524 is provided.

[0129] The user can review specific conversation snippets 1530 and transcripts 1532. For example, a particular snippet in which an agent says, "I can help with that. Let me check. I'm sorry," can be tagged as the agent exhibiting "empathy" 1534.

[0130] Machine Learning (ML) and Neural Networks Machine learning (ML) algorithms are characterized by their ability to improve task performance over time, even when not explicitly programmed with rules for performing the task (i.e., learning). A machine learning (ML) model is the output generated when an ML algorithm is trained with data. As described herein, embodiments of the present invention use one or more artificial intelligence (AI) and machine learning (ML) algorithms for scenario detection and conversation tagging. Various exemplary ML algorithms are within the scope of the present invention. Exemplary ML techniques for implementing various embodiments of the present invention are described below.

[0131] Neural Networks A neural network is a computational model inspired by the structure of the human brain, consisting of interconnected units called neurons that work together to process information. Neural networks are a type of ML algorithm that are particularly effective at recognizing patterns and making predictions based on complex data. Due to their ability to learn from large amounts of data and improve their performance over time, neural networks are widely used in a variety of applications, such as image and speech recognition and natural language processing. Figure 16 illustrates the operation of a basic neural network for a scenario detection and conversation tagging system, according to an exemplary embodiment of the present invention.

[0132] 16 illustrates a single-layer neural network, also known as a single-layer perceptron. Such perceptrons are the building blocks of neural network layers included in bi-encoders and cross-encoders, according to embodiments of the present invention. The operation of a single-layer neural network includes the following steps:

[0133] 1. Input: element v j The system receives an input vector v 1604 with j∈[1, n] representing the jth input, where each element of the vector corresponds to a neuron 1606 in the input layer. In a scenario detection and speech tagging system, the input vector may be a character, syllable, or word of a phrase, utterance, or sentence. The input vector may also be an element of a phrase context vector or an embedding vector, as described in this disclosure.

[0134] 2. Transfer function: weights W corresponding to each input j 1608. These weighted inputs are then summed as a transfer function to produce the activation function

number

[0135] Each neuron in a neural network may have a bias value 1612, which is added to the weighted sum of the inputs to that neuron. Both the weights and bias values ​​are learned during the training process. The purpose of the bias is to provide every neuron with a learnable constant value that can help the model better fit the data. With bias, the total input to the activation function is

number

[0136] 3. Activation Function: The total input is passed through an activation function 1614. The activation function σ determines the activation value ο 1618, which is the output of the neuron. It is typically a nonlinear function such as a sigmoid function or a ReLU (rectified linear unit) function. The activation function threshold θ 1616 is a value that determines whether the neuron is activated or not. For some activation functions, such as a step function, the threshold is a specific value. If the total input is above the threshold, the neuron outputs a constant value, and if the total input is below the threshold, the neuron outputs a zero value. For other activation functions, such as a sigmoid function or a ReLU (rectified linear unit) function, the threshold is not a specific value but a transition point on the curve of the function.

[0137] 4. Output: The activation value ο 1618 is the output of the activation function. This value is passed to the next layer in the network, and in the case of the last layer, it becomes the final output. In a scenario detection and conversation tagging system, the final output of the neural network can be an embedding or context vector representing the scenario phrases, or a scenario intent Boolean variable.

[0138] FIG. 17 provides an overview of the training process for a neural network that can be used in a scenario detection and conversation tagging system, according to an exemplary embodiment of the present invention.

[0139] In the scenario detection and speech tagging system, the neural network 1702 undergoing training can be a bi-encoder or a cross-encoder, as described in this disclosure. Training a neural network involves iteratively updating the network's weights and biases 1710 to minimize the difference between a predicted output 1704 and a true or target output 1706, where the predicted output 1704 is the result produced by the network when a set of inputs from a dataset is passed through. The true or target output 1706 is the true, desired outcome. The difference between the predicted output and the true output is calculated using a loss function 1708 to quantify the error made by the network in its prediction. In the scenario detection and speech tagging system, the neural network's predicted output 1704 and true / target output 1706 can be embeddings or context vectors representing scenario phrases, or scenario intent Boolean variables.

[0140] The loss function is part of the cost function 1708 and is a measure of how well the network is performing across the entire dataset. The goal of training is to minimize the cost function 1708. This is achieved by iteratively adjusting the network's weights and biases 1710 in directions that lead to steepest descent of the cost function. The size of these adjustments is determined by the learning rate 1708, which is a hyperparameter that controls the amount the weights and biases change at each iteration. A smaller learning rate results in smaller changes and slower convergence to a minimum in the cost function. On the other hand, a larger learning rate results in larger changes and faster convergence, but at the risk of exceeding the minimum.

[0141] Training a neural network is a combined process of forward and backpropagation. Forward propagation is the process by which input data is passed through the network from the input layer to the output layer. During forward propagation, the network's weights and biases are used to calculate an output for a given input. Backpropagation, on the other hand, is the process used to update the network's weights and biases 1710 based on the output's error (e.g., cost function) 1708. After forward propagation through the neural network 1702, the network's output 1704 is compared to the true output 1706, and an error 1708 is calculated. This error is then propagated through the network, starting from the output layer and moving to the input layer. The weights and biases 1710 are adjusted in a way that minimizes this error. This process repeats for multiple iterations, or epochs, until the network is able to make accurate predictions.

[0142] The above neural network training method, in which the network is trained on a labeled dataset (e.g., example pairs of input user prompts and corresponding output recommendations) and the true output is known, is called supervised learning. In unsupervised learning, the network is trained on an unlabeled dataset, with the goal of discovering hidden patterns or structures in the data. The network is not provided with true outputs, and training is based on the inherent characteristics of the data. Furthermore, reinforcement learning is a type of learning in which an agent learns to make decisions from rewards or penalties it receives based on its actions. While reinforcement learning typically does not rely on a pre-existing dataset, some forms of reinforcement learning can use a database of past actions, states, and rewards during the learning process. As will be apparent from the following overview, any neural network training method that uses a labeled dataset is within the scope of the methods and systems described herein.

[0143] FIG. 18 provides additional details regarding the training process, in accordance with an exemplary embodiment of the present invention.

[0144] Transformer Model Architecture The transformer architecture, such as that used by the bi-encoder and cross-encoder shown in Figures 4-7, is a neural network design introduced in the paper "Attention is All You Need" by Vaswani et al., published in June 2017 (available at https: / / arxiv.org / abs / 1706.03762), which is incorporated by reference as if fully set forth herein. Large-scale language models (LLMs) rely heavily on transformer architectures.

[0145] The architecture (see Figure 1 in the referenced paper) is based on the concept of "attention," which allows the model to focus on different parts of the input sequence when producing an output. The transformer consists of an encoder and a decoder: the encoder processes the input data and the decoder generates the output. Each of these components consists of multiple layers: self-attention layers and pointwise fully connected layers.

[0146] The self-attention layer in the Transformer model allows it to compare the relevance of different parts of the input sequence when generating the output, thereby capturing long-range dependencies in the data, while the fully connected layer is used to transform the output of the self-attention layer, adding complexity and depth to the model's learning ability.

[0147] Transformer models are known for their ability to process long data sequences, making them particularly useful for tasks such as machine translation and text summarization. Transformer architectures use positional encoding to give the model information about the relative positions of words in the input sequence. Because the model itself has no inherent sense of order or sequence, positional encoding is a way to inject some order information into an order-independent attention mechanism.

[0148] Embedding Vector Space In the context of neural networks, tokenization refers to the process of converting input and output spaces, such as natural language text or programming code, into discrete units or "tokens." This process allows the network to effectively process and understand the data, converting complex structures into manageable individual elements from which the model can learn and generate.

[0149] In training neural networks, embeddings serve as a form of word distribution representation that transforms discrete categorical variables (i.e., tokens) into a continuous vector space (i.e., embedding vectors). This transformation process captures the semantic properties of tokens, allowing tokens with similar meanings to have similar embeddings. These embeddings provide a dense representation of tokens and their semantic relationships. Typically, embeddings are represented as vectors, but they may also be represented as matrices or tensors.

[0150] The input of a Transformer typically requires a transformation from an input space (e.g., a natural language token space) to an embedding space. This process, called "encoding," converts discrete inputs (tokens) into continuous vector representations (embeddings). This transformation is a prerequisite for a Transformer model to process the input data and understand the semantic relationships between tokens (e.g., words). Similarly, the output of a Transformer typically requires a transformation from an embedding space to an output space (e.g., natural language tokens, programming code tokens, etc.), in a process called "decoding." Therefore, both the training of a neural network and its evaluation (i.e., its use during deployment) occur in the embedding space.

[0151] The remainder of this document infers the processes of tokenization, encoding, decoding, and detokenization. In other words, the processes described below occur in an "embedding space." Thus, the tokenization and encoding of training data and input prompts may not be explicitly represented or described, but are implied. Similarly, the decoding and detokenization of neural network outputs are also implied.

[0152] Train and fine-tune machine learning (ML) modules FIG. 18 is an exemplary flow diagram illustrating the various phases and datasets involved in training a machine learning algorithm for a scenario detection and conversation tagging system, according to an exemplary embodiment of the present invention.

[0153] The training process begins at step 1810, where scenario and conversation data are captured, acquired, assimilated, or generated. In step 1820, the captured data is preprocessed or prepared. In step 1830, an ML model (e.g., a bi-encoder or cross-encoder) is trained using the training data 1825. In step 1840, the ML model is evaluated, validated, and tested, and further improvements to the ML model are fed back to step 1830 for additional training. If the performance is accepted, optimal model parameters are selected in step 1850.

[0154] Training data 1825 is a recorded dataset containing multiple instances of system inputs and correct results. This trains an ML model to optimize performance for a specific target task. In FIG. 18 , training data 1825 may also include a subset for validation and testing of the ML model as part of training iterations 1830 and 1840. For NN-based ML models, the quality of the output may depend on (a) the NN architecture design and hyperparameter configuration, (b) the optimization of the NN coefficients or parameters, and (c) the quality of the training dataset. These components can be refined and optimized using various methods. For example, training data 2825 may be augmented via an augmentation process.

[0155] In some embodiments, an additional fine-tuning 1860 phase, including iterative fine-tuning 1860 and evaluation, validation, and testing 1870 steps, is performed using the fine-tuning data 1855. Fine-tuning in machine learning is a process that involves taking a selected 1850 pre-trained model and further adjusting or "tuning" its parameters to better fit a specific task or fine-tuning dataset 1855. This technique is particularly useful when dealing with deep learning models that are trained on a large, general training dataset 1825 and are intended to be applied to more specialized tasks or smaller datasets. The goal is to leverage and refine the knowledge the model already acquired during its initial training (often called transfer learning) so that the model performs better at the more specialized task at hand.

[0156] The fine-tuning process typically begins with a model that has already been trained on a large benchmark training dataset 1825, such as ImageNet for image recognition tasks. The model's existing weights, learned from the original training, serve as the starting point. During fine-tuning, the model is further trained on a new fine-tuning dataset 1855, which may contain different classes or types of data than the original training set. This additional training phase allows the model to adjust its weights to better capture the characteristics of the new fine-tuning dataset 1855, thereby improving its performance on the specific task being fine-tuned.

[0157] In some embodiments, an additional testing and validation 1880 phase is performed using test and validation data 1875. Testing and validation of an ML model both refer to the process of evaluating the model's performance on a separate dataset 1875 that was not used during training to ensure that it generalizes well to new, unseen data. Validating an ML model helps prevent overfitting by ensuring that the model's performance generalizes beyond the training data.

[0158] While the validation phase is considered part of ML model development and can lead to further iterations of fine-tuning, the testing phase is the final evaluation of a model's performance after it has been trained and validated. The testing phase provides an unbiased assessment of the final model's performance that reflects how well the model is expected to perform on unseen data, and is typically conducted after the model has finished to ensure that the evaluation is unbiased.

[0159] Once a model has been trained (1830), selected (1850), optionally fine-tuned (1860), and validated / tested (1880), it is deployed (1890). Deployed ML models, such as bi-encoders and cross-encoders 1895, typically receive new data 1885 that has been preprocessed 1820.

[0160] In machine learning, data preprocessing 1820 is aligned with the model development phase. During model training 1830, preprocessing involves cleaning, normalizing, and converting raw data into a format suitable for learning patterns. For fine-tuning 1860, preprocessing adapts the data to match the distribution of the specific target task, ensuring the pre-trained model can effectively transfer its knowledge. Validation 1880 preprocessing mirrors training preprocessing to accurately assess model generalization without leaking information from the training set. Finally, during deployment (1890 and 1895), preprocessing ensures that real-world data matches the expectations of the trained model and often involves dynamic adjustments to maintain consistency with the training and validation stages.

[0161] Machine Learning Algorithms Various exemplary ML algorithms are within the scope of the present invention, including, but not limited to, random forests, nearest neighbors, decision trees, support vector machines (SVMs), Adaboost, gradient boosting, Bayesian networks, evolutionary algorithms, various neural networks (including deep learning networks (DLNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs)), and the like.

[0162] ML modules based on Transformers and Large Language Models (LLMs) are particularly well suited for the tasks described herein. The online article by S. Raschka, "Understanding Large Language Models -- A Transformative Reading List," posted February 7, 2023, and available at https: / / sebastianraschka.com / blog / 2023 / llm-reading-list.html, describes various LLM architectures that are within the scope of the methods and systems described herein, and is incorporated by reference in its entirety as if fully set forth herein.

[0163] The input to each of the listed ML modules is a feature vector containing the input data listed above for each ML module. The output of the ML module is a feature vector containing the corresponding output data listed above for each ML module.

[0164] Prior to deployment, each of the above-listed ML modules may be trained with one or more respective sample input datasets and one or more corresponding sample output datasets. The input and output training datasets may be generated from a database containing a history of input and output instances, or may be synthetically generated by a subject matter expert.

[0165] System configuration example An example embodiment of the present disclosure may include one or more servers (management computing entities), one or more networks, and one or more clients (user computing entities). Each of these components, entities, devices, and systems (similar terms used interchangeably herein) may communicate directly or indirectly with each other, for example, via the same or different wired or wireless networks.

[0166] 19 provides an illustrative schematic diagram of a server (management computing entity) 1902 and a client (user computing entity) 1930 used for communication within a scenario detection and conversation tagging system, connected via a network 1920, in accordance with some embodiments of the present invention. While FIG. 19 depicts various system entities as separate, standalone entities, various embodiments are not limited to this particular architecture. Furthermore, the terms "client device," "client computing entity," "edge device," and "edge computing system" are equivalent and are used interchangeably herein.

[0167] Exemplary Management Computing Entity 19 provides a schematic diagram of a server (managing computing entity, 1902) according to one embodiment of the present disclosure. In general, the terms computing entity, computer, entity, device, system, and / or similar words used interchangeably herein may refer to, for example, one or more computers, computing entities, desktop computers, mobile phones, tablets, phablets, notebook computers, laptops, distributed systems, game consoles, watches, glasses, iBeacons, proximity beacons, key fobs, radio frequency identification (RFID) tags, earphones, scanners, televisions, dongles, cameras, wristbands, wearable items / devices, kiosks, input terminals, servers or server networks, blades, gateways, switches, processing devices, processing entities, set-top boxes, repeaters, routers, network access points, base stations, etc., and / or any combination of devices or entities adapted to perform the functions, operations, and / or processes described herein. Such functions, operations, and / or processes may include, for example, sending, receiving, operating, processing, displaying, storing, determining, creating / generating, monitoring, evaluating, and / or comparing (similar terms used interchangeably herein). In one embodiment, these functions, operations, and / or processes may be performed on data, content, and / or information (similar terms used interchangeably herein).

[0168] As shown, in one embodiment, the managing computing entity (1902) may also include one or more communication interfaces (1910) for communicating with various computing entities, such as by communicating data, content, and / or information (similar terms used interchangeably herein), which may be transmitted, received, operated on, processed, displayed, and / or stored, etc.

[0169] As shown in FIG. 19 , in one embodiment, the managing computing entity (1902) may include or communicate with one or more processors (i.e., processing elements 1904, also referred to as processors and / or processing circuits, and similar terms used interchangeably herein) that communicate with other elements within the managing computing entity (1902), for example, via a bus. It should be appreciated that the processors (1904) may be embodied in several different ways. For example, the processors (1904) may be embodied as one or more complex programmable logic devices (CPLDs), microprocessors, multi-core processors, simultaneous processing entities, application-specific instruction set processors (ASIPs), microcontrollers, and / or controllers. Furthermore, the processors (1904) may be embodied as one or more other processing devices or circuits. The term circuitry may refer to an entire hardware embodiment or a combination of hardware and a computer program product. Accordingly, the processor (1904) may be embodied as an integrated circuit, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic array (PLA), a hardware accelerator, and / or other circuitry, etc. It will be appreciated, therefore, that the processor (1904) may be configured for a particular use or configured to execute instructions stored on a volatile or non-volatile (or non-transitory) medium or otherwise accessible to the processor (1904). Thus, whether configured by hardware or a computer program product, or a combination thereof, the processor (1904), when so configured, may be capable of performing steps or operations according to embodiments of the present disclosure.

[0170] In one embodiment, the managing computing entity (1902) may further include or be in communication with non-transitory memory (non-volatile media, non-volatile storage, non-transitory storage, memory, memory storage, and / or memory circuitry, similar terms used interchangeably herein). In one embodiment, the non-transitory memory or storage may include one or more non-transitory memory or storage media (1906), including, but not limited to, a hard disk, ROM, PROM, EPROM, EEPROM, flash memory, MMC, SD memory cards, memory sticks, CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM, millipede memory, and / or racetrack memory. As will be appreciated, the non-volatile (or non-transitory) storage or memory media may store databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, bytecode, compiled code, interpreted code, machine code, and / or executable instructions, etc. The terms database, database instance, and / or database management system (similar terms used interchangeably herein) may refer to a collection of records or data stored on a computer-readable storage medium using one or more database models, such as a hierarchical database model, a network model, a relational model, an entity-relationship model, an object model, a document model, a semantic model, and / or a graph model.

[0171] In one embodiment, the management computing entity (1902) may further include or be in communication with volatile memory (volatile storage, memory, memory storage, memory and / or circuitry, similar terms used interchangeably herein). In one embodiment, the volatile storage or memory may also include one or more volatile storage or memory media 1908, including, but not limited to, RAM, DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM, cache memory, and / or register memory, etc. As will be appreciated, the volatile storage or memory media may be used to store at least a portion of a database, database instance, database management system, data, application, program, program module, script, source code, object code, byte code, compiled code, interpreted code, machine code, and / or executable instructions, etc., executed by the processor (1904), for example. Thus, as will be appreciated, databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, byte code, compiled code, interpreted code, machine code, and / or executable instructions, etc., may be used to control certain aspects of the operation of the managing computing entity (1902) with the aid of the processor (1904) and operating system.

[0172] As shown, in one embodiment, the managing computing entity (1902) may also include one or more communication interfaces (1910) for communicating with various computing entities, such as by communicating data, content, and / or information (similar terms used interchangeably herein), which may be sent, received, acted upon, processed, displayed, and / or stored, etc. Such communication may be performed using a wired data transmission protocol, such as Fiber Distributed Data Interface (FDDI), Digital Subscriber Line (DSL), Ethernet, Asynchronous Transfer Mode (ATM), Frame Relay, Data Service Interface Standard over Cable (DOCSIS), or any other wired transmission protocol. Similarly, the management computing entity (1902) may be configured to communicate over a wireless external communications network using any of a variety of protocols, such as General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access 2000 (CDMA2000), CDMA2000 1X (1xRTT), Wideband Code Division Multiple Access (WCDMA), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Evolved Universal Terrestrial Radio Access Network (E-UTRAN), Evolution Data Optimized (EVDO), High Speed ​​Packet Access (HSPA), High Speed ​​Downlink Packet Access (HSDPA), IEEE 802.11 (Wi-Fi), Wi-Fi Direct, 802.16 (WiMAX), Ultra Wideband (UWB), Infrared (IR) protocol, Near Field Communication (NFC) protocol, Wibree, Bluetooth protocol, Wireless Universal Serial Bus (USB) protocol, and / or any other wireless protocol.

[0173] Although not shown, the managing computing entity (1902) may include or communicate with one or more input elements, such as keyboard input, mouse input, touchscreen / display input, motion input, kinetic input, audio input, pointing device input, joystick input, and / or keypad input. The managing computing entity (1902) may also include or communicate with one or more output elements (not shown), such as audio output, video output, screen / display output, motion output, and / or kinetic output.

[0174] As will be appreciated, one or more of the components of the management computing entity (1902) may be located remotely from other components of the management computing entity (1902), such as in a distributed system. Furthermore, one or more of the components may be combined, and additional components that perform the functions described herein may be included in the management computing entity (1902). Thus, the management computing entity (1902) may be adapted to accommodate a variety of needs and situations. It will be appreciated that these architectures and descriptions are provided for illustrative purposes only and are not limiting of the various embodiments.

[0175] Exemplary User Computing Entity A user may be a human individual, a company, an organization, an entity, a department within an organization, a representative of an organization and / or person, and / or an artificial user such as an algorithm, artificial intelligence, or other software interfacing therewith. Figure 19 further provides an exemplary schematic representation of a client-user computing entity 1930 that can be used in conjunction with embodiments of the present disclosure. In general, the terms device, system, computing entity, entity, and / or similar words used interchangeably herein may refer to, for example, one or more computers, computing entities, desktops, mobile phones, tablets, phablets, notebook computers, laptops, distributed systems, game consoles, watches, glasses, key fobs, radio frequency identification (RFID) tags, earphones, scanners, televisions, cameras, wristbands, kiosks, input terminals, servers or server networks, blades, gateways, switches, processing devices, processing entities, set-top boxes, repeaters, routers, network access points, base stations, etc., and / or any combination of devices or entities adapted to perform the functions, operations, and / or processes described herein. A user computing entity may be operated by a variety of parties. Such functions, operations, and / or processes may include, for example, sending, receiving, operating, crawling, extracting, processing, displaying, storing, determining, creating, generating, generating a display, monitoring, evaluating, comparing, and / or similar terms used interchangeably herein. In various embodiments, these functions, operations, and / or processes may be performed on machine-readable or human-readable data, content, information, and / or similar terms used interchangeably herein. Furthermore, in embodiments of the present invention, computing device 1930 may be a general-purpose computing device with a dedicated graphics processing unit (GPU) 1936 and artificial intelligence 1938 module.In some embodiments, computing device 1930 may be built on a backplane 1956 connected to a power source 1958. Alternatively, it may be implemented in the cloud using a logically and / or physically distributed architecture.

[0176] 19, the user computing entity 1930 may include an antenna 1970, a radio frequency (RF) transceiver 1932, and a processor unit 1940 that provides signals to and receives signals from the transceiver. The signals provided to and received from the transceiver may include signal information in accordance with the air interface standard of the applicable wireless system. In this regard, the user computing entity (1930) may be capable of operating with one or more air interface standards, communication protocols, modulation types, and access types. More specifically, the user computing entity 1930 may operate in accordance with any of several wireless communication standards and protocols, such as those described above with respect to the management computing entity 1902. In particular embodiments, the user computing entity 1930 may operate according to multiple wireless communication standards and protocols, such as UMTS, CDMA2000, 1xRTT, WCDMA, TD-SCDMA, LTE, E-UTRAN, EVDO, HSPA, HSDPA, Wi-Fi, Wi-Fi Direct, WiMAX, UWB, IR, NFC, Bluetooth, and / or USB. Similarly, the user computing entity 1930 may operate according to any of several wireless communication standards and protocols, such as those described above with respect to the management computing entity 1902.

[0177] Through these communication standards and protocols, the user computing entity 1930 can communicate with various other entities using concepts such as Unstructured Supplementary Service Data (USSD), Short Message Service (SMS), Multimedia Messaging Service (MMS), Dual Tone Multi-Frequency Signaling (DTMF), and / or a Subscriber Identity Module Dialer (SIM Dialer), etc. The user computing entity 1930 may also download modifications, add-ons, and updates to its firmware, software (including, e.g., executable instructions, applications, program modules), and operating system, for example.

[0178] In some implementations, processing unit 1940 may be embodied in several different ways. For example, processing unit 1940 may be embodied as one or more complex programmable logic devices (CPLDs), microprocessors, multi-core processors, simultaneous processing entities, application specific instruction set processors (ASIPs), microcontrollers, and / or controllers. Additionally, processing unit 1940 may be embodied as one or more other processing devices or circuits. The term circuit may refer to an entire hardware embodiment or a combination of hardware and a computer program product. Thus, processing unit 1940 may be embodied as an integrated circuit, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic array (PLA), a hardware accelerator, and / or other circuitry, etc. Thus, it will be appreciated that processing unit 1940 may be configured for a particular use or may be configured to execute instructions stored on volatile or non-volatile media or otherwise accessible to the processing unit. Thus, whether configured by hardware or a computer program product, or a combination thereof, processing unit 1940, when so configured, may be capable of performing steps or operations according to embodiments of the present invention.

[0179] In some embodiments, processing unit 1940 may include a control unit 1942 and a special purpose arithmetic unit (ALU) 1944 to perform arithmetic and logical operations. In some embodiments, user computing entity 1930 may include a graphics processing unit (GPU) 1946 for specialized parallel processing tasks, and / or an artificial intelligence (AI) module or accelerator 1948, also specialized for applications including artificial neural networks and / or machine learning. In some embodiments, processing unit 1940 may be coupled with GPU 1946 and / or AI accelerator 1948 to distribute and coordinate processing tasks.

[0180] In some embodiments, computing entity 1930 may include a user interface, which includes input interface 1950 and output interface 1952, each coupled to processing unit 1940. User input interface 1950 may include any of several devices or interfaces that allow computing entity 1930 to receive data, such as a keypad (hard or soft), a touch display, a microphone for voice / speech, and a camera for motion or posture interfacing. User output interface 1952 may include any of several devices or interfaces that allow computing entity 1930 to provide information to a user, such as via a touch display or a speaker for audio output. In some embodiments, output interface 1952 may connect computing entity 1930 to external speakers or a projector for audio or image output. In some embodiments, user interfaces 1950 and 1952 integrate multimodal data and accommodate both humans and bots / algorithms. Some examples of human interfaces include dashboard-type interfaces, workflow-based interfaces, conversational interfaces, and spatial computer interfaces. Some examples of bot / algorithm interfaces include code interfaces, text-based API interfaces, etc.

[0181] The user computing entity 1930 may also include volatile and / or nonvolatile storage or memory 1960, which may be embedded and / or removable. For example, the nonvolatile or non-transitory memory may be a hard disk, ROM, PROM, EPROM, EEPROM, flash memory, MMC, SD memory card, memory stick, CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM, Millipede memory, and / or Racetrack memory, etc. The volatile memory may be RAM, DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM, cache memory, and / or registered memory, etc. The volatile and non-volatile storage or memory may store operating system 1962, application software 1964, data 1966, databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, byte code, compiled code, interpreted code, machine code, and / or executable instructions, etc., to implement the functionality of the user computing entity (1930). As shown, this may include user applications resident on the entity or accessible via a browser or other user interface for communicating with the managing computing entity (1902) and / or various other computing entities.

[0182] In some embodiments, the user computing entity 1930 may include one or more components or functionality that are the same as or similar to the management computing entity 1902, as described in more detail above. It should be recognized that these architectures and descriptions are provided for illustrative purposes only and are not limiting of the various embodiments.

[0183] In some embodiments, computing entities 1910 and / or 1930 may communicate with other computing devices and / or external devices, such as access points, to receive information, such as software or firmware, or to transmit information from the memory of the computing entity to external systems or devices, such as servers, computers, smartphones, etc.

[0184] In some embodiments, two or more computing entities, such as 1910 and / or 1930, may establish a connection using a network, such as 1920, that utilizes any of the previously listed network protocols. In some embodiments, the computing entities may use network interfaces, such as 1910 and 1934, to communicate with each other, such as by communicating data, content, information, and / or similar terms used interchangeably herein, that may be transmitted, received, operated on, processed, displayed, and / or stored, etc.

[0185] In some embodiments of the present invention, the entire system may be implemented and provided to end users and operators via the Internet in a so-called cloud implementation. No local installation of software or hardware is required; end users and operators can access the system of the present invention directly via the Internet using either a web browser or similar software on a client, which can be a desktop, laptop, mobile device, etc. This eliminates the need for custom software installation on the client side, increasing the flexibility of the (software-as-a-service) service offering and improving user satisfaction and ease of use. Various business models, profit models, and delivery mechanisms for the present invention are envisioned, and all are considered to be within the scope of the present invention.

[0186] Additional Hardware and Software Implementation Details Although an exemplary processing system is described above, implementations of the subject matter and functional operations described herein can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, or in one or more combinations thereof, including the structures disclosed herein and their structural equivalents.

[0187] Embodiments of the subject matter and operations described herein may be implemented in digital electronic circuitry, or computer software, firmware, or hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions, encoded on a computer storage medium for execution by or to control the operation of an information / data processing apparatus. Alternatively, or in addition, the program instructions may be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information / data for transmission to an appropriate receiving device for execution by the information / data processing apparatus. A computer storage medium may be, or may be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory or device, or a combination of one or more of these. Furthermore, while a computer storage medium is not a propagated signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0188] The operations described herein may be implemented as operations performed by an information / data processing apparatus on information / data stored in one or more computer-readable storage devices or received from other sources.

[0189] The term "data processing unit" encompasses all types of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system-on-chip, or a combination thereof. An apparatus may include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also include code that creates an execution environment for underlying computer programs, such as code comprising processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement infrastructures for a variety of different computing models, such as web services, distributed computing, and grid computing infrastructures.

[0190] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or information / data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple associated files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0191] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs, operating on input information / data and generating output to perform actions. Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose processors, as well as one or more processors of any type of digital computer. Generally, a processor receives instructions and information / data from a read-only memory (ROM), a random-access memory (RAM), or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operably coupled to receive information / data from, transfer information / data to, or both of these mass storage devices. However, a computer need not have such devices. Suitable devices for storing computer program instructions and information / data include all types of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0192] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented in a computer that has a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information / data to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with a user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be acoustic, vocal, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to and from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0193] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., information / data servers, or middleware components, e.g., application servers, or front-end components, e.g., client computers having a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described herein, or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital information / data communication, e.g., a communications network. Examples of communications networks include local area networks (“LANs”) and wide area networks (“WANs”), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer networks).

[0194] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends information / data (e.g., HTML pages) to the client devices (e.g., for the purpose of displaying the information / data to and receiving user input from a user interacting with the client device). Information / data generated at the client device (e.g., a result of user interaction) can be received from the client device by the server.

[0195] Although many specific implementation details are described herein, these should not be construed as limitations on the scope of any embodiment or what may be claimed, but rather as descriptions of features inherent to particular embodiments. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as functioning in a particular combination, or even when originally claimed as such, one or more features from a claimed combination can, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0196] Similarly, although acts are depicted in the figures in a particular order, this should not be understood as requiring that such acts be performed in the particular order or sequence shown, or that all of the acts shown be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0197] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Furthermore, the processes depicted in the accompanying figures do not necessarily require the particular order or sequence depicted to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0198] In some embodiments of the present invention, the entire system can be implemented and provided to end users and operators over the Internet in a so-called cloud implementation. No local installation of software or hardware is required; end users and operators can access the system of the present invention directly over the Internet using either a web browser or similar software on a client, which can be a desktop, laptop, mobile device, etc. This eliminates the need for custom software installation on the client side, increasing the flexibility of the (software-as-a-service) service offering and improving user satisfaction and ease of use. Various business models, profit models, and delivery mechanisms for the present invention are envisioned, and all are considered to be within the scope of the present invention.

[0199] In general, methods performed to implement embodiments of the present invention may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions referred to as "computer program(s)" or "computer code(s)." A computer program typically comprises one or more instructions stored at various times in various memories and storage locations of a computer, which, when read and executed by one or more processors of the computer, cause the computer to perform the operations necessary to implement elements including various aspects of the present invention. Furthermore, while the present invention has been described in the context of fully functional computers and computer systems, those skilled in the art will understand that various embodiments of the present invention can be distributed as program products in various forms, and that the present invention applies equally regardless of the particular type of machine or computer-readable medium used to effect the distribution. Examples of computer-readable media include, but are not limited to, volatile and non-volatile (or non-transitory) memory devices, floppy and other removable disks, hard disk drives, recordable media such as optical disks, compact disk read-only memories (CD ROMs), digital versatile disks (DVDs), and the like, as well as digital and analog communications media.

[0200] conclusion Those skilled in the art will recognize that the use cases, structures, schematics, flow diagrams, and steps may be performed in any order or subcombination, while the inventive concept of the present invention remains within the broader scope of the invention. Every embodiment may be unique, and step(s) of the method(s) may be shortened or extended, overlap with other activity, deferred, delayed, and / or continued after a time lag, so that all active users and running application programs are accommodated on the server(s) implementing the method(s) of the present invention.

[0201] For ease of explanation, embodiments of the methods of the present disclosure are depicted and described as a series of acts or steps. However, acts or steps according to the present disclosure can occur in various orders and / or simultaneously, and with other acts or steps not shown and described herein. Moreover, not all illustrated acts or steps are required to implement a methodology in accordance with the disclosed subject matter. Furthermore, those skilled in the art will understand and appreciate that a methodology can alternatively be represented as a series of interrelated states via a state diagram or events or their equivalents.

[0202] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly indicates otherwise. Thus, for example, reference to a "cable" includes a single cable, a bundle of two or more different cables, etc. As used in this specification and claims, the terms "comprise," "comprising," "includes," "including," "have," "having," and the like are intended to be open-ended and non-limiting, meaning "including but not limited to."

[0203] In the foregoing description, numerous specific details are set forth, such as specific structures, dimensions, processes, and parameters, to provide a thorough understanding of the present invention. Particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments. The words "example," "exemplary," "illustrative," and the like are used herein to mean example, instance, or illustration. Any aspect or design described herein as an "example" or equivalent should not necessarily be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" or equivalents is intended to present concepts in a concrete manner.

[0204] As used in this application, "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X includes A or B" is intended to mean the natural inclusive permutation. That is, if X includes A, if X includes B, or if X includes both A and B, then "X includes A or B" is satisfied in any of the above cases.

[0205] References throughout this specification to "an embodiment," "certain embodiments," or "one embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "an embodiment," "certain embodiments," or "one embodiment" throughout this specification are not necessarily all referring to the same embodiment.

[0206] As used herein, the term "about" with respect to a measurand refers to normal variation in that measurand that would be expected by one of ordinary skill in the art when making the measurement and exercising a level of care commensurate with the purpose of the measurement and the precision of the measurement device. For example, in some exemplary embodiments, the term "about" may include the recited number + / - 10%, such that "about 10" includes 9 to 11. In other exemplary embodiments, the term "about" may not include the recited number + / - X%, where X is considered normal variation in the above measurements by one of ordinary skill in the art.

[0207] Features that are described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features that are, for brevity, described in the context of a single embodiment, may also be implemented separately or in any suitable subcombination. The applicant hereby notifies that new claims may be formulated to such features and / or combinations of such features during prosecution of this application or any further application derived therefrom. The described temporary physical storage medium features may be incorporated into or used in corresponding methods and / or systems, and vice versa.

[0208] While the present invention has been described with reference to certain exemplary embodiments, it will be apparent to those skilled in the art that various modifications and changes can be made to these embodiments without departing from the broader scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will also be apparent to those skilled in the art that the above-described embodiments are specific examples of a single broader invention that may have a broader scope than any single description taught. Many changes may be made herein without departing from the scope of the invention, as defined by the claims.

Claims

1. 1. A non-transitory physical storage medium storing program code executable by a hardware processor, the program code, when executed by the hardware processor, causing the hardware processor to perform a computer-implemented process for determining a scenario name label that best matches an utterance during automatic scenario detection in a conversation between a customer and an agent, the program code comprising: receiving, by an acquisition stage including a bi-encoder neural network, a plurality of scenarios, a plurality of scenario name labels, and a plurality of lists of sample phrases, wherein each scenario in the plurality of scenarios is associated with a name label from the plurality of name labels and with a list of sample phrases from the plurality of lists of sample phrases; encoding, by the obtaining stage, each sample phrase in the list of sample phrases into a phrase encoding to generate a list of phrase encodings; generating, by the obtaining stage, a plurality of scenario encodings, each scenario encoding in the plurality of scenario encodings associated with a scenario of the plurality of scenarios, associated with a scenario name label of the plurality of scenario name labels, and associated with a list of phrase encodings of the plurality of lists of phrase encodings, each scenario encoding in the plurality of scenario encodings based on normalizing and determining a centroid of a list of phrase encodings associated with a scenario of the plurality of scenarios; storing a list of the plurality of scenario encodings, the plurality of scenario name labels, and the plurality of phrase encodings in a database; receiving an utterance by said acquisition stage; encoding, by the acquisition stage, a conversational context vector for the utterance; generating, by the obtaining stage, a plurality of similarity scores for the conversational context vector of the utterance, each similarity score in the plurality of similarity scores being associated with a given scenario encoding from the plurality of scenario encodings stored in the database; determining, by the obtaining stage, a best-matching scenario encoding from among the plurality of scenario encodings by selecting a given scenario encoding from the plurality of scenarios that has a highest similarity score from the plurality of similarity scores; generating, by the obtaining stage, a plurality of ordered pairs, wherein a first component of each ordered pair in the plurality of ordered pairs is the utterance and a second component of each ordered pair in the plurality of ordered pairs is a given phrase encoding from a list of given phrase encodings associated with the best-matching scenario encoding; generating a plurality of similarity probabilities by a reordering stage including a cross-encoder neural network, each similarity probability in the plurality of similarity probabilities being associated with an ordered pair of the plurality of ordered pairs; determining, by the reordering stage, whether at least one similarity probability of the plurality of similarity probabilities exceeds a predetermined threshold; assigning, by the reordering stage, the best-matching scenario name label from among the plurality of scenario name labels associated with the best-matching scenario encoding to the utterance if at least one similarity probability among the plurality of similarity probabilities exceeds the predetermined threshold; The non-transitory physical storage medium includes code that, if none of the multiple similarity probabilities exceeds the predetermined threshold, assigns an unintentional scenario name label to the utterance by the reordering stage.

2. 2. The non-transitory physical storage medium of claim 1, wherein the bi-encoder neural network includes a masked and permutation pre-trained (MPNet) based model for language understanding, multiple encoder stacks, and a multi-layer perceptron (MLP).

3. 2. The non-transitory physical storage medium of claim 1, wherein the cross-encoder neural network includes a large-scale language model (LLM) based on a Bidirectional Encoder Representation by Transformers (BERT) language model, multiple encoder stacks, and a multi-layer perceptron (MLP).

4. The non-transitory physical storage medium of claim 1 , wherein encoding the sample phrases includes generating a vector of real numbers.

5. The non-transitory physical storage medium of claim 1 , wherein the plurality of similarity scores comprises a plurality of cosine similarity scores.

6. The program code 2. The non-transitory physical storage medium of claim 1, further comprising code for triggering a conversation tag based on the best-matching scenario name label and a plurality of setting options, the conversation tag including a text string.

7. The non-transitory physical storage medium of claim 6 , wherein the plurality of configuration options includes speaker identity, and the program code that triggers the conversation tag is further based on the identity of a speaker of the utterance.

8. The non-transitory physical storage medium of claim 7 , wherein the program code that triggers the conversation tag is further based on whether a sequence of agent sentences is followed by a customer sentence.

9. 7. The non-transitory physical storage medium of claim 6, wherein the plurality of setting options include speaker behavior, and the program code that triggers the conversation tag is further based on whether the speaker of the utterance mentioned a particular phrase.

10. 7. The non-transitory physical storage medium of claim 6, wherein the plurality of setting options include timing, and the program code that triggers the conversation tag is further based on whether the utterance occurs within a predetermined period of time after the conversation begins.

11. 1. A non-transitory physical storage medium storing program code executable by a hardware processor, the program code, when executed by the hardware processor, causing the hardware processor to perform a computer-implemented process for determining a scenario name label that best matches an utterance during automatic scenario detection in a conversation between a customer and an agent, the program code comprising: receiving the utterance by an acquisition stage including a bi-encoder neural network; encoding, by the acquisition stage, a conversational context vector for the utterance; generating, by the obtaining stage, a plurality of similarity scores for the conversation context vector of the utterance, each similarity score in the plurality of similarity scores being associated with a scenario encoding among a plurality of scenario encodings, a scenario name label among a plurality of scenario name labels, and a list of phrase encodings among a plurality of lists of phrase encodings; determining, by the obtaining stage, a best-matching scenario encoding from among the plurality of scenario encodings by selecting a given scenario encoding from the plurality of scenario encodings that has a highest similarity score from the plurality of similarity scores; generating, by the obtaining stage, a plurality of ordered pairs, wherein a first component of each ordered pair in the plurality of ordered pairs is the utterance and a second component of each ordered pair in the plurality of ordered pairs is a given phrase encoding from a list of given phrase encodings associated with the best-matching scenario encoding; generating a plurality of similarity probabilities by a reordering stage including a cross-encoder neural network, each similarity probability in the plurality of similarity probabilities being associated with an ordered pair of the plurality of ordered pairs; determining, by the reordering stage, whether at least one similarity probability of the plurality of similarity probabilities exceeds a predetermined threshold; assigning, by the reordering stage, the best-matching scenario name label from among the plurality of scenario name labels associated with the best-matching scenario encoding to the utterance if at least one similarity probability among the plurality of similarity probabilities exceeds the predetermined threshold; The non-transitory physical storage medium includes code that, if none of the multiple similarity probabilities exceeds the predetermined threshold, assigns an unintentional scenario name label to the utterance by the reordering stage.

12. 1. A computer-implemented method for determining a best-matching scenario name label to an utterance during automatic scenario detection in a customer-agent conversation, comprising: receiving the utterance by an acquisition stage including a bi-encoder neural network; encoding, by the acquisition stage, a conversational context vector for the utterance; generating, by the obtaining stage, a plurality of similarity scores for the conversation context vector of the utterance, each similarity score in the plurality of similarity scores being associated with a scenario encoding among a plurality of scenario encodings, a scenario name label among a plurality of scenario name labels, and a list of phrase encodings among a plurality of lists of phrase encodings; determining, by the obtaining stage, a best-matching scenario encoding from among the plurality of scenario encodings by selecting a given scenario encoding from the plurality of scenario encodings that has a highest similarity score from the plurality of similarity scores; generating, by the obtaining stage, a plurality of ordered pairs, wherein a first component of each ordered pair in the plurality of ordered pairs is the utterance and a second component of each ordered pair in the plurality of ordered pairs is a given phrase encoding from a list of given phrase encodings associated with the best-matching scenario encoding; generating a plurality of similarity probabilities by a reordering stage including a cross-encoder neural network, each similarity probability in the plurality of similarity probabilities being associated with an ordered pair of the plurality of ordered pairs; determining, by the reordering stage, whether at least one similarity probability of the plurality of similarity probabilities exceeds a predetermined threshold; assigning, by the reordering stage, the best-matching scenario name label from among the plurality of scenario name labels associated with the best-matching scenario encoding to the utterance if at least one similarity probability among the plurality of similarity probabilities exceeds the predetermined threshold; and if none of the plurality of similarity probabilities exceeds the predetermined threshold, assigning an unintentional scenario name label to the utterance by the reordering stage.

13. 13. The computer-implemented method of claim 12, wherein the bi-encoder neural network includes a masked and permutation pre-training (MPNet) based model for language understanding, multiple encoder stacks, and a multi-layer perceptron (MLP).

14. 13. The computer-implemented method of claim 12, wherein the cross-encoder neural network includes a large-scale language model (LLM) based on a Bidirectional Encoder Representation by Transformers (BERT) language model, multiple encoder stacks, and a multi-layer perceptron (MLP).

15. The computer-implemented method of claim 12 , wherein the plurality of similarity scores comprises a plurality of cosine similarity scores.

16. The computer-implemented method of claim 12 , further comprising triggering a conversation tag based on the best-matching scenario name label and a plurality of setting options, the conversation tag comprising a text string.

17. 17. The computer-implemented method of claim 16, wherein the plurality of configuration options includes speaker identity, and triggering the conversation tag is further based on a speaker identity of the utterance and whether a sequence of agent sentences is followed by a customer sentence.

18. 17. The computer-implemented method of claim 16, wherein the plurality of configuration options includes speaker behavior, and triggering the conversation tag is further based on whether a speaker of the utterance mentioned a particular phrase.

19. 17. The computer-implemented method of claim 16, wherein the plurality of configuration options includes timing, and triggering the conversation tag is further based on whether the utterance occurs within a predetermined period of time after the conversation begins.

Images