A quality assurance system based on speaker intent detection.

Abstract

A semantic similarity-based configurable system for automatic scenario detection in customer-agent conversations is disclosed. The system understands intent from semantic similarities in vector space between speaker sentences, does not rely on the use of synonyms, and tolerates a large amount of paraphrasing. This approach is easily scalable to a large number of customers and can be fed more data to increase accuracy and precision. Furthermore, the system is configurable in real time, allowing clients to control which intents are detected and how. In some embodiments, the semantic similarity-based configurable system includes a scenario detection system, a conversation tagging system, a bi-encoder, and a cross-encoder. The scenario detection system receives inputs of sample phrases and customer and agent utterances and generates results. The sample phrases may be phrases and keywords that describe scenarios representing customer or call agent behavior.

Description

Technical Field

[0001] Cross - reference to related applications If an Application Data Sheet (ADS) or PCT Request Form (“Request”) has been filed on the filing date of the present application, it shall be incorporated herein by reference. Any application claimed in the ADS, or any claim of priority under 35 U.S.C. §§ 119, 120, 121, or 365(c), and any and all parent applications of such application, its grandparent application, its great - grandparent application, etc., shall also include any claim of priority made in those applications, and any material incorporated by reference, and shall be incorporated herein by reference to the extent that such subject matter is not inconsistent with the present specification.

[0002] Furthermore, the present application is related to U.S. patent applications claiming the priorities listed below, and these U.S. patent applications are hereby incorporated herein by reference in their entirety as if fully set forth herein.

[0003] ● U.S. Provisional Patent Application No. 63 / 442,159 (Docket No. LEV - 1002P), filed on January 31, 2023, entitled “Configurable Intent Phrase Based Quality Assurance Systems,” which describes a semantic - similarity - based configurable system for automatic scenario detection in customer - agent conversations.

[0004] ● U.S. Provisional Patent Application No. 63 / 521,105 (Docket No. LEV - 1001P), filed on June 15, 2023, entitled “Customer Agent Screen Recording Systems and Methods,” which describes a method for triggering recording and analysis of customer - agent interactions using a Customer Relationship Management (CRM) system.

[0005] ● U.S. Provisional Patent Application No. 63 / 597,343 (Reference No. LEV-1003P), filed on November 9, 2023, titled "Transformer-Based Assistant for Identifying, Organizing, and Responding to Customer Concerns," describes a transformer-based agent assistant system that functions as a machine learning-based customer service tool that analyzes past customer-agent conversations to build a knowledge base of problem-solving steps.

[0006] Copyright and Trade Dress Notice Some disclosures in this patent document include copyrighted material. This patent document may indicate and / or describe matters that are or may become the rights holder's trade dress. The copyright holder and the rights holder of the trade dress shall not object to any complete copy of the patent disclosure as expressed in the U.S. Patent and Trademark Office application and records, but otherwise retain all copyright and trade dress rights.

[0007] This disclosure relates to machine learning, and more particularly to a system for recognizing semantics in conversation and identifying intent. [Background technology]

[0008] The descriptions in the background art of this invention are provided to aid in understanding the invention and its applications and uses, and may not constitute prior art.

[0009] As a company grows in terms of employees, products, and complexity, maintaining good relationships with customers is crucial. Under normal circumstances, this involves customers contacting the company's customer service hotline. The customer is redirected to a call agent, who assists the customer with their questions or concerns. Several ways to ensure a satisfactory customer experience include properly training call agents and understanding the frequency of specific complaints to minimize future incidents. However, current methods of engaging in quality assurance (QA) and gaining conversational insights involve having systems implement algorithms that monitor calls and use precise keyword matching. For example, to detect customer dissatisfaction, a client's call might be examined for keywords such as “disgust” or “anger.” The drawback of such an approach is that it may fail to detect certain scenarios, such as customers using synonyms not found in a preset keyword list (e.g., “I’m frustrated” might be an unlisted phrase) or customers implicitly expressing their thoughts (e.g., “It’s January and there’s no sign of my paycheck being paid”). Thus, customer complaint intent may be underreported and left unaddressed. Another problem with keyword-based systems is that they must maintain a list of synonyms in a database and continuously check whether such words are being spoken during customer-call agent conversations. When a company receives hundreds of calls a day, such a system becomes inefficient and lacks scalability, as it cannot comprehensively check all synonyms. Therefore, this approach is always limited by the size and diversity of the keyword list.

[0010] Based on this background technology, the present invention was developed. [Overview of the project]

[0011] This summary of the invention provides a general overview of the present invention, its applications, and use, and is not intended to limit the scope of the invention, which will become clear from the detailed description when read in conjunction with the drawings.

[0012] Therefore, considering the underlying technology, developing a scalable and highly accurate system that identifies speaker intent from utterances spoken by a speaker (a call center agent or their customer) based on the semantics of customer-call agent conversations would represent a cutting-edge technological advancement. Such a system could be implemented by understanding intent from the semantic similarities of the vector space between speaker sentences, without relying on synonym use and allowing for a large amount of paraphrasing. This approach is easily scalable to a large number of customers and can be fed more data to improve accuracy and precision. Furthermore, developing a real-time configurable system that allows the company to control which intents are detected and how would represent a further cutting-edge technological advancement.

[0013] Accordingly, a semantic similarity-based configurable system for automated scenario detection in customer-agent conversations is disclosed. The system understands intent from the semantic similarity of the vector space between speaker sentences, is independent of synonym use, and tolerates a large amount of paraphrasing. This approach is easily scalable to a large number of customers and can be fed more data to improve accuracy and precision. Furthermore, since the system is configurable in real time, clients can control which intents are detected and how. In some embodiments, the semantic similarity-based configurable system includes a scenario detection system, a conversation tagging system, a biencoder, and a crossencoder, where the scenario detection system receives sample phrases and customer and agent utterances as input to generate results. The sample phrases may be phrases and keywords that describe a scenario representing customer or call agent behavior.

[0014] Accordingly, various methods, processes, systems, and non-temporary storage media are provided for storing program code for executing a process to determine the best-matching scenario name label for an utterance during automated scenario detection in a customer-agent conversation. In one embodiment, a non-temporary physical storage medium for storing program code is provided. The program code is executable by a hardware processor. When the hardware processor executes the program code, it causes the hardware processor to execute a computer implementation process for determining the best-matching scenario name label for an utterance during automated scenario detection in a customer-agent conversation. The program code includes receiving a list of multiple scenarios, multiple scenario name labels, and multiple sample phrases by an acquisition stage that includes a biencoder neural network, wherein each scenario in the multiple scenarios is associated with a scenario name label among the multiple scenario name labels and with a list of sample phrases among the multiple sample phrases; encoding each sample phrase in the list of multiple sample phrases into a phrase encoding by the acquisition stage in order to generate a list of multiple phrase encodings; generating a list of multiple scenario encodings by the acquisition stage, wherein each scenario encoding in the multiple scenario encodings is associated with a scenario among the multiple scenarios, associated with a scenario name label among the multiple scenario name labels and with a list of phrase encodings among the multiple phrase encodings, and each scenario encoding in the multiple scenario encodings is based on normalizing and determining the centroid of the list of phrase encodings associated with the scenario among the multiple scenarios; storing the list of multiple scenario encodings, multiple scenario name labels, and multiple phrase encodings in a database; receiving an utterance by the acquisition stage; encoding the conversation context vector of the utterance by the acquisition stage; andThe process involves generating multiple similarity scores for a conversational context vector of an utterance, where each similarity score is associated with a scenario encoding from among multiple scenario encodings stored in a database; determining the best-matching scenario encoding from among multiple scenario encodings by selecting the scenario encoding with the highest similarity score from among the multiple similarity scores in the acquisition stage; and generating multiple ordered pairs by the acquisition stage, where the first component of each ordered pair is an utterance, and the second component of each ordered pair is a phrase encoding from a list of phrase encodings associated with the best-matching scenario encoding. The code includes: generating multiple similarity probabilities by a reordering stage including a cross-encoder neural network, wherein each of the multiple similarity probabilities is associated with an ordered pair among multiple ordered pairs; determining by the reordering stage whether at least one of the multiple similarity probabilities exceeds a predetermined threshold; assigning the best-matching scenario name label from among multiple scenario name labels associated with the best-matching scenario encoding to the utterance if at least one of the multiple similarity probabilities exceeds a predetermined threshold; and assigning an unintentional scenario name label to the utterance if none of the multiple similarity probabilities exceed a predetermined threshold.

[0015] In one embodiment, the biencoder neural network includes a masked and replaceable pre-trained (MPNet) based model for language understanding, a stack of encoders, and a multilayer perceptron (MLP).

[0016] In one embodiment, the cross-encoder neural network includes a large-scale language model (LLM) based on a transformer-based bidirectional encoder representation (BERT) language model, multiple encoder stacks, and a multilayer perceptron (MLP).

[0017] In one embodiment, encoding a sample phrase includes generating a vector of real numbers.

[0018] In one embodiment, the multiple similarity scores are multiple cosine similarity scores.

[0019] In one embodiment, the program code further includes code that triggers conversation tags based on the best-matching scenario name label and multiple configuration options, where the conversation tags include text strings.

[0020] In one embodiment, multiple configuration options include speaker identity, and the program code that triggers conversation tags is further based on the identity of the speaker of the utterance.

[0021] In one embodiment, the program code that triggers the conversation tag is further based on whether a customer's sentence follows a sequence of agent's sentences.

[0022] In one embodiment, the multiple configuration options include speaker behavior, and the program code that triggers conversation tags is further based on whether the speaker of the utterance referred to a particular phrase.

[0023] In one embodiment, the multiple configuration options include timing, and the program code that triggers the conversation tag is further based on whether an utterance occurred within a predetermined period after the conversation began.

[0024] In another embodiment, a non-temporary physical storage medium is provided for storing program code. The program code is executable by a hardware processor. When the hardware processor executes the program code, it causes the hardware processor to perform a computer implementation process for determining the best-matching scenario name label for an utterance during automated scenario detection in a customer-agent conversation.The program code includes an acquisition stage that includes a biencoder neural network to receive utterances, an acquisition stage that encodes the conversation context vector of the utterances, an acquisition stage that generates multiple similarity scores for the conversation context vector of the utterances, each similarity score being associated with a scenario encoding among multiple scenario encodings, a scenario name label among multiple scenario name labels, and a list of phrase encodings among multiple phrase encodings, an acquisition stage that determines the best-matching scenario encoding from among multiple scenario encodings by selecting the scenario encoding with the highest similarity score among the multiple similarity scores, and an acquisition stage that generates multiple ordered pairs, the first component of each ordered pair being an utterance, and multiple The code includes generating a second component of each ordered pair in the ordered pair is a phrase encoding from a list of phrase encodings associated with the best-match scenario encoding, and generating a plurality of similarity probabilities by a reordering stage which includes a cross-encoder neural network, where each similarity probability in the plurality of similarity probabilities is associated with an ordered pair among the plurality of ordered pairs, and by a reordering stage which determines whether at least one of the plurality of similarity probabilities exceeds a predetermined threshold, and by the reordering stage which assigns the best-match scenario name label from a plurality of scenario name labels associated with the best-match scenario encoding to the utterance if at least one of the plurality of similarity probabilities exceeds a predetermined threshold, and by the reordering stage which assigns an unintentional scenario name label to the utterance if none of the plurality of similarity probabilities exceed a predetermined threshold.

[0025] In yet another embodiment, a computer implementation method is provided for determining the best-matching scenario name label for an utterance during automated scenario detection in a customer-agent conversation. The method includes receiving a list of multiple scenarios, multiple scenario name labels, and multiple sample phrases by an acquisition stage including a biencoder neural network, wherein each scenario in the multiple scenarios is associated with a scenario name label among the multiple scenario name labels and with a list of sample phrases among the multiple sample phrases; encoding each sample phrase in the list of multiple sample phrases into a phrase encoding in order to generate a list of multiple phrase encodings; and generating a list of multiple scenario encodings by the acquisition stage, wherein each scenario encoding in the multiple scenario encodings is associated with a scenario among the multiple scenarios, a scenario name label among the multiple scenario name labels, and a phrase encoding among the list of multiple phrase encodings. The process involves generating a list of scenario encodings, a list of scenario encodings, a list of scenario name labels, and a list of phrase encodings associated with a scenario in a list of scenario encodings, based on normalizing and determining the centroid of a list of phrase encodings associated with a scenario among the multiple scenarios, and storing the list of multiple scenario encodings, a list of scenario encodings, and a list of phrase encodings in a list of scenario encodings, and receiving an utterance by an acquisition stage, encoding the conversation context vector of the utterance by an acquisition stage, and generating multiple similarity scores for the conversation context vector of the utterance by an acquisition stage, wherein each similarity score in the multiple similarity scores is associated with a scenario encoding among the multiple scenario encodings stored in the database, and by selecting the scenario encoding among the multiple scenarios that has the highest similarity score among the multiple similarity scores by an acquisition stage.Determining the best-matching scenario encoding from among a plurality of scenario encodings, generating, by an acquisition stage, a plurality of ordered pairs, wherein a first component of each ordered pair in the plurality of ordered pairs is an utterance, and a second component of each ordered pair in the plurality of ordered pairs is a phrase encoding from a list of phrase encodings associated with the best-matching scenario encoding, generating, by a reordering stage that includes a cross-encoder neural network, a plurality of similarity probabilities, wherein each similarity probability in the plurality of similarity probabilities is associated with an ordered pair among the plurality of ordered pairs, determining, by the reordering stage, whether at least one similarity probability among the plurality of similarity probabilities exceeds a predetermined threshold, and, by the reordering stage, when at least one similarity probability among the plurality of similarity probabilities exceeds a predetermined threshold, assigning the best-matching scenario name label from among a plurality of scenario name labels associated with the best-matching scenario encoding to the utterance, and, by the reordering stage, when none of the similarity probabilities in the plurality of similarity probabilities exceed a predetermined threshold, assigning an unintentional scenario name label to the utterance.

[0026] In one embodiment, the bi-encoder neural network includes a masked and replaced pre-training (MPNet)-based model for language understanding, a plurality of encoder stacks, and a multi-layer perceptron (MLP).

[0027] In one embodiment, the cross-encoder neural network includes a large language Roberta-based model (LLM) based on a bidirectional encoder representation from transformers (BERT) language model, a plurality of encoder stacks, and a multi-layer perceptron (MLP).

[0028] In one embodiment, encoding a sample phrase includes generating a vector of real numbers.

[0029] In one embodiment, the plurality of similarity scores are a plurality of cosine similarity scores.

[0030] In one embodiment, the method further includes triggering a conversation tag based on the best matching scenario name label and a plurality of setting options, and the conversation tag includes a text string.

[0031] In one embodiment, the plurality of setting options include speakerhood, and triggering the conversation tag is further based on the identity of the speaker of the utterance and whether a customer's utterance follows a sequence of agent's utterances.

[0032] In one embodiment, the plurality of setting options include speaker behavior, and triggering the conversation tag is further based on whether the speaker of the utterance referred to a specific phrase.

[0033] In one embodiment, the plurality of setting options include timing, and triggering the conversation tag is further based on whether the utterance occurred within a predetermined period after the conversation started.

[0034] In various embodiments, a computer program product is disclosed. The computer program may be used to determine the best matching scenario name label of an utterance during automatic scenario detection in a conversation between a customer and an agent, and may include a computer-readable storage medium implementing program instructions or program code, and the program instructions are executable by a processor to cause the processor to perform steps for the above-described steps.

[0035] In various embodiments, a system is described that includes a memory storing computer-executable components and a hardware processor operably coupled to the memory and executing the computer-executable components stored in the memory, and the computer-executable components may include components communicatively coupled to a processor that performs the above-described steps.

[0036] In another embodiment, the present invention is a non-temporary computer-readable storage medium for storing executable instructions, which, when executed by a processor, causes the processor to perform a process to do something, and the instructions cause the processor to perform the steps described above.

[0037] In another embodiment, the present invention is a system of configurable intent phrases based on a quality assurance system, as illustrated and described herein, the system comprising a user device having a processor, a display, and a first memory; a server having a second memory and a data repository; a communication link between the user device and the server; and a plurality of computer codes embodied in the first memory and the second memory of the user device and the server, the plurality of computer codes, when executed, cause the server and the user device to perform a process including the steps described above.

[0038] In yet another embodiment, the present invention is a computerized server comprising at least one processor, memory, and a plurality of computer codes embodied in the memory, wherein, when executed, the plurality of computer codes cause the processor to execute a process including the steps described above. Other aspects and embodiments of the present invention include methods, processes, and algorithms including the steps described herein, and also include processes and operating modes of systems and servers described herein.

[0039] Further aspects and embodiments of the present invention will become apparent from the modes for carrying out the invention when read in conjunction with the accompanying drawings. Features described in the context of different aspects and / or embodiments of the present invention may be used together and / or interchangeable wherever possible. Similarly, features described in the context of a single embodiment for brevity may also be provided separately or as any suitable small combination. Features described in relation to non-temporary physical storage media may have corresponding features that are definable and / or combinatable with respect to a system and / or a system, and / or in relation to a system, and vice versa, these embodiments specifically envision.

[0040] The accompanying drawings are incorporated into and constitute part of this specification, illustrating embodiments of the invention and are used in conjunction with this description to illustrate the principles of the disclosed embodiments. For clarity, brevity, and flexibility, not all elements, components, or specifications are defined in all drawings. Not all drawings corresponding to specific steps or embodiments of the invention are drawn to scale. Instead, emphasis is placed on the description of the properties, functions, and products of the manufacturing methods and devices described herein.

[0041] The embodiments of the present invention described herein are illustrative and not limiting. Hereinafter, embodiments will be described by reference to the accompanying drawings. [Brief explanation of the drawing]

[0042] [Figure 1] This specification shows exemplary high-level diagrams of a scenario detection system and a conversation tagging system according to the embodiments disclosed herein. [Figure 2] An illustrative schematic diagram of a scenario detection system according to the embodiments disclosed herein is shown. [Figure 3] This section shows an exemplary detailed diagram of a scenario detection system according to the embodiments disclosed herein. [Figure 4]A schematic diagram of the biencoder operation according to the embodiments disclosed herein is shown. [Figure 5] An exemplary block diagram of a biencoder according to the embodiments disclosed herein is shown. [Figure 6] A schematic diagram of the cross encoder operation according to the embodiments disclosed herein is shown. [Figure 7] An exemplary block diagram of a cross encoder according to the embodiments disclosed herein is shown. [Figure 8] This specification provides an exemplary flowchart illustrating in detail the operation of the scenario detection and conversation tagging system according to the embodiments disclosed herein. [Figure 9] Another flowchart illustrating in detail the operation of the scenario detection and conversation tagging system according to the embodiments disclosed herein is shown. [Figure 10] Further flowcharts illustrating in detail the operation of the scenario detection and conversation tagging systems according to the embodiments disclosed herein are shown. [Figure 11] A system configuration diagram showing a scenario detection and conversation tagging system according to the embodiments disclosed herein is shown. [Figure 12] A first exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention is shown. [Figure 13] A second exemplary graphical user interface (GUI) relating to a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention is shown. [Figure 14] A third exemplary graphical user interface (GUI) relating to a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention is shown. [Figure 15] A fourth exemplary graphical user interface (GUI) relating to a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention is shown. [Figure 16]This invention illustrates the operation of a basic neural network relating to a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention. [Figure 17] This document outlines the training process for a neural network that can be used in a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention. [Figure 18] This is an exemplary flowchart illustrating various phases and datasets related to training a machine learning algorithm for a scenario detection and conversation tagging system, according to an exemplary embodiment of the present invention. [Figure 19] This invention provides a schematic diagram illustrating a server (management computing entity) and a client (user computing entity) used in a scenario detection and conversation tagging system, according to several embodiments of the present invention. [Modes for carrying out the invention]

[0043] In the following description, many specific details are given for illustrative purposes to provide a complete understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be implemented without using these specific details. In other examples, structures, devices, activities, methods, and processes are shown using schematic diagrams, use cases, and / or illustrations to avoid obscuring the invention. The following description contains many details for illustrative purposes, but those skilled in the art will understand that many variations and / or modifications to the details presented are within the scope of the invention. Similarly, many of the features of the invention are described in relation to or in relation to each other, but those skilled in the art will understand that many of these features can be provided independently of others. Therefore, this description of the invention is presented without loss of generality and without limitation.

[0044] As used herein, the singular forms "a," "an," and "the" include plural references unless otherwise explicitly indicated in the context. For example, a reference to "fiber" includes a single fiber and a mixture of two or more different fibers. Also as used herein, the term "about" in relation to a measured quantity refers to the normal variation of that quantity that can be expected by a person skilled in the art when performing a measurement and exercising a level of care commensurate with the purpose of the measurement and the precision of the measuring instrument. In certain embodiments, the term "about" includes a range of + / - 10% of the stated number, such that "about 10" includes 9 to 11.

[0045] Scenario detection system and conversation tagging system In the following explanation, “client” refers to the owner or operator of the system, such as the organization providing the service or product; “customer” refers to the caller (e.g., the user of the service or product); and “agent” refers to the responder (e.g., a customer service representative, account manager, etc.).

[0046] Figure 1 shows an exemplary high-level diagram of a scenario detection system 102 and a conversation tagging system 104 according to an embodiment disclosed herein. The integrated "scenario and conversation tagging" system allows the client to configure various types of events to detect. The "scenario" portion of the integrated system is the behavior detection system, and the "conversation tagging" portion of the integrated system is the alarm system. Alarms are triggered upon detection of a scenario and, in some embodiments, detection of a small number of other configuration options.

[0047] In some embodiments, a “scenario” is defined as the behavior that the detection system 102 detects in each sentence within a given conversation. The system allows the client to describe a scenario using a set of representative phrases. For example, a “customer disappointed” scenario intended to capture customer dissatisfaction might be described by the following phrases: “I simply hate this,” “This has never worked well for me,” “Oh my god,” and “Enough already.” Furthermore, the client may also add “negative phrases” to the description, which are phrases that seem close to describing the desired scenario but that the client does not want to trigger the scenario. For example, in the “customer disappointed” scenario, the client might not want this scenario to be triggered by a sentence similar to “This is confusing,” and therefore such a sentence may be added to the set of negative phrases.

[0048] However, by itself, scenarios detected by the sentence detection system 102 may not be displayed to the client. In contrast, scenarios may trigger a “conversation tag” and, depending on the set configuration option, may be displayed to the client. In some embodiments, a “conversation tag” is a text string label assigned to a conversational portion of a conversation. “Conversation tags” are assigned to conversational portions based on the following criteria:

[0049] 1. Who is the speaker in the conversation? The user may set this to either "Agent" or "Customer".

[0050] 2. (a) Whether the conversational portion conveys a specific intent (such as that described in the “Scenario”). For example, a tag called “Stressed Customer” might be based on a “Scenario” describing a situation where a customer is complaining about pricing. Or, (b) Whether the conversation contains any specific keywords.

[0051] Whether the tag should apply to any part of the conversation that occurs at any point in the conversation, or whether the tag should apply only to the part of the conversation that occurs within the first N seconds (where N is set by the user).

[0052] Several possible configuration options are listed below.

[0053] 1. Speaker: Should the tag only be triggered for agent sentences, or customer sentences, or sequences where an agent sentence is followed by a customer sentence? The last option is called a "dynamic tag".

[0054] 2. Speaker Behavior: Whether the tag should be triggered when the "speaker" refers to a specific phrase, or when a specific scenario is detected in the speaker's sentence. Because this is a configurable feature, "speaker behavior" can refer to any action taken by the speaker. For example, (A) "Dissatisfaction with pricing" is the action of a customer complaining that the price is too high or that they can find a better price elsewhere. (B) "Appropriate introductory greeting" is the action of an agent greeting a customer, for example, by saying, "Thank you for calling ABC. How can I help you?" (C) "Requesting a refund" is the action of a customer inquiring about the process of returning a product and getting a refund for any reason.

[0055] 3. Timing: Whether the tag is triggered at all times during the conversation, or only during the first X seconds of the conversation (where X is pre-configured by the client).

[0056] As shown in Figure 1, the client begins by configuring the scenario detection system 102 to include any number of scenarios, e.g., Scenario 1 106, Scenario 2 108, and Scenario m 110. Each scenario encapsulates some behavior of a customer or agent. To set up a scenario, the client provides a list of several (e.g., 5 to 15) relevant exemplary phrases that capture how a customer or agent exhibits behavior. For example, to detect a “customer disappointment” scenario, as shown in Scenario 1 106, the following phrases would be associated: namely, “I don’t like this,” “It’s a hassle,” and “It’s ridiculous.” To detect a “customer dissatisfaction” scenario, the exemplary phrase might be, “I simply hate your after-sales service.”

[0057] Once a set of scenarios and a list of associated phrases are configured in the scenario detection system 102 (behavior detection system), the client can configure the "conversation tag" system 104 (alarm system) to trigger whenever a scenario is detected in a customer / agent's utterance. Note that conversation tags do not necessarily have to be conditional on scenario detection; the client may configure them to trigger only under specific circumstances, such as when a particular keyword is detected in a customer / agent's utterance. For example, tag 112 sets a conversation tag that triggers when scenario 106 is detected, and tag 214 sets a conversation tag that triggers when a specific keyword is detected. Other tags, such as tag 316 and tag n118, may also be established.

[0058] In some embodiments, triggering a conversation tag is based on the label of the nearest scenario and multiple configuration options. In some embodiments, the multiple configuration options include speaker identity (i.e., whether the speaker is an agent or a customer), and triggering a conversation tag is further based on the identity of the speaker of the utterance. In some embodiments, triggering a conversation tag is further based on whether a customer's sentence follows a sequence of an agent's sentence. In some embodiments, the multiple configuration options include speaker behavior, and triggering a conversation tag is further based on whether the speaker of the utterance referred to a particular phrase. In some embodiments, the multiple configuration options include timing, and triggering a conversation tag is further based on whether the utterance occurred within a predetermined period after the start of the conversation. Timing refers to whether the tag can be triggered at any time during the conversation, or only for conversational segments that occurred within the first N seconds, where N is user-configurable.

[0059] The advantages of the integrated scenario detection system 102 and conversation tagging system 104 include (1) the ability to tag based on a sequence of customer and agent scenarios ("dynamic tags"), and (2) the ability to create tags conditional on the presence of one of a set of keywords or the presence of a scenario. By separating the behavior detection (via scenario detection) and the alarm system (conversation tags), the client can independently configure multiple tags that are based on the same underlying scenario but have different configurations. For example, the system may function differently depending on whether the utterance is from a customer or an agent, or whether the system has a history of a particular customer's behavior or psychological state.

[0060] Scenario detection system acquisition stage and reordering stage Figure 2 shows an exemplary schematic diagram of a scenario detection system according to an embodiment disclosed herein. The scenario detection system comprises two stages: an acquisition stage and a reordering stage. The acquisition stage acts as a coarse sieve, and in some embodiments, the acquisition stage may be implemented by a biencoder model 202, a neural network that encodes human language sentences into embedding vectors (or simply “embeddings”), and may be an ordered sequence of real numbers. The reordering stage acts as a fine sieve, and in some embodiments, is implemented by a crossencoder model 204. This acquisition-reordering framework is commonly used in text-based semantic retrieval. However, it may be adapted for scenario detection as described herein.

[0061] The scenario detection system operates as follows: First, the client's sample phrases 206 and their associated scenarios are input to the biencoder 202. For example, the "customer was disappointed" scenario may be described by the following phrases: "I'm not happy with this," "This is useless to me," and / or "ridiculous." The biencoder 202 encodes such phrases for all scenarios and then stores them in the database 208 (e.g., "phrase encodings"). In some embodiments, a scenario is encoded as the normalized centroid of the embeddings of all input phrases, and the conversational portion is encoded as a single normalized embedding during test time. In some embodiments, the centroid of a set of vectors in vector space is the vector in vector space that minimizes the weighted sum of the generalized squared distances from each vector in the set of vectors to a point in vector space. In some embodiments, the distance here is the Euclidean distance. In other embodiments, other geometric shapes are used. In some embodiments, the weighted sum is equal, and each vector is weighted equally. During the training phase, encoding involves the following: In other words, for any scenario, N phrases are obtained from the user, where N can range from a minimum of 3 to any number of phrases the user may wish to provide. Next, the N phrases are encoded separately using a biencoder, which generates a vector of size [M × 1] (e.g., M = 768) for each phrase. Next, the normalized centroid (essentially the average value of all vectors) of all N vectors for this particular scenario is calculated, and this encoding forms the encoding of the scenario. For example, suppose the user establishes a scenario named "Greetings" and provides five relevant phrases. Next, all five phrases are encoded separately to generate five vectors of size [768 × 1], and then the normalized centroid of these five encoded vectors is calculated to form the encoding of the scenario "Greetings". In the inference phase, encoding involves the following:In other words, when a new query Q is received, it is directly encoded using a biencoder to generate a vector of size [M × 1] (e.g., M = 768). Furthermore, the vector is then normalized so that it can be applied to a scenario phrase, comparing the two to find matches.

[0062] Next, a new sentence from the conversation (e.g., "invisible utterance during testing" 210) is encoded using the biencoder 202 to obtain the closest scenario phrase encoding 208 to that sentence. For example, for the sentences "I expected better service than this" or "I was disappointed," the phrase associated with the "The customer was disappointed" scenario might be retrieved as a result. Finally, the crossencoder 204 calculates a score associated with the sentence of the closest scenario and all client phrases. If that score exceeds a certain threshold, the label of that closest scenario is assigned to the sentence as a prediction 212. Otherwise, nothing is assigned to the sentence.

[0063] The advantage of the retrieval-reordering implementation over traditional keyword matching methods is that retrieval-reordering does not rely on the presence of keywords to detect intent. Instead, it focuses on sentence semantics; that is, it maps each sentence to a numbered list, i.e., embedding vectors, such that the distance between embedding vectors of sentences with similar meanings is small. This is independent of whether two sentences with similar meanings have the same set of keywords.

[0064] Figure 3 shows an exemplary detailed diagram of a scenario detection system according to an embodiment disclosed herein. The scenario detection system takes action during training time ("training time") 302 (i.e., when the client first sets up a scenario) and during test time ("test time") 304 (i.e., when the system attempts to detect the presence of a scenario in the customer or agent's utterance).

[0065] During training time 302, the biencoder 306 takes in a list 308 of sample phrases that can describe specific scenarios. For example, the phrases “I’m not happy with this,” “This doesn’t work for me,” and “Ridiculous” could be used to describe a scenario within the “Customer Disappointed” scenario. The biencoder 306 then encodes each phrase provided by the user and stores these phrase embeddings in a database ("phrase encodings") 310. The encoding is performed as described with reference to Figure 2.

[0066] During test time 304, the system is given a random or arbitrary utterance 312 from a customer or agent. This utterance is encoded using a biencoder 306. In some embodiments, a first similarity score (e.g., cosine similarity) is calculated between the encoding of this utterance 312 and each stored phrase embedding. In other embodiments, a first similarity score (e.g., cosine similarity) is calculated between the encoding of this utterance 312 and the centroid of the phrase encoding 310 associated with each scenario of the stored phrase embedding. The scenario of the phrase embedding with the highest first similarity score to the utterance is retrieved, and the similarity score between the two embeddings is calculated as the cosine distance between the two embedding vectors, resulting in a real number between 0 and 1. If this highest first similarity falls below any first threshold score, "NO_INTENT" (i.e., no intent match, no intent) is returned. Otherwise (i.e., the highest first similarity score exceeds any of the first threshold scores), a second similarity score is calculated between the utterance 312 and all phrases belonging to the higher intent using the cross-encoder 314.

[0067] Regarding the "highest" first similarity: For a single conversational segment, its embedding is calculated, and then its similarity score is calculated against the stored embeddings of all scenarios programmed for this client. The scenario with the highest similarity score to this conversational segment embedding (from all scenarios) is selected. Regarding the concept of "no intent" or "unintended": Each client programs N intents, where N is any positive integer and scenarios are synonymous with intents. "No intent" or "unintended" means that the current conversational segment does not exhibit any of the N intents programmed by the client. However, in this case, the conversational segment may exhibit some true intent that is not in the set of N programmed intents.

[0068] If the second similarity score with any given phrase exceeds either of the second threshold scores, an intent label is retrieved and generated. Otherwise, i.e., if the second similarity score does not exceed either of the second threshold scores, "No Intent" ("NO_INTENT," i.e., lack of intent, the utterance is unintentional) is returned. With respect to the "First Threshold Score" and "Second Threshold Score," the thresholds are selected as follows: A subset of the set of utterances is selected as the test set, and the model is run on this test set. Then, various threshold levels are set, and the threshold that produces the best precision and recall on this test set is selected as the appropriate threshold score. After thorough testing with various clients, a particular threshold of 0.55 appears to work well enough for all clients. Of course, the thresholds may be adjusted between clients depending on the quality of precision and recall. One measure of the performance of these experiments is the average F1 score (i.e., the harmonic mean of precision and recall) across all intents present in the test set.

[0069] Bi-encoder Figure 4 shows a schematic diagram of the biencoder operation during a “training time” 402 according to an embodiment disclosed in the specification. A phrase 410 is input to the biencoder 412, which generates a phrase embedding 414. In some embodiments, the biencoder 412 includes a transformer encoder 422, which includes a large-scale language model (LLM) trained with a contrast loss. A sentence 420 of arbitrary length is input to the transformer encoder 422, which generates a fixed-size embedding 424.

[0070] The token space of the biencoder 412 is a finite countable set of all tokens, each token assigned a unique "token ID". In transformer terminology, a token is a sequence of characters. During tokenization, a string is represented as a sequence of tokens. For example, "hello world" can be represented as these three tokens: ("he", "-llo", "world"). Thus, each input phrase or sentence is mapped to a single embedding vector. The transformer encoder 422 receives the representation of the sentence as a sequence of token IDs and produces "context embeddings," or a sequence of real vectors. The number of resulting embedding vectors is equal to the number of token IDs in the input sequence. The average of these vectors is then calculated to produce a single vector representing the embeddings of a given phrase or sentence. Thus, the transformer encoder 422 encodes the sentence into a sequence of vectors that capture the relationships between the various words in the sentence.

[0071] To convert an input phrase into a phrase embedding, the biencoder 412 employs a transformer encoder architecture with several layers, each layer containing two main subcomponents: a multi-head self-attention mechanism and a position-based feedforward network. Each of these subcomponents is followed by residual connection and layer normalization, which helps stabilize the learning process. The output of the biencoder 412 is configured to produce a fixed-length embedding, i.e., a vector length of 768, regardless of the length of the input phrase or sentence.

[0072] The control loss is calculated via a triplet function. Given anchor sentence a, positive sentence p, and negative sentence n, the triplet loss adjusts the network so that the distance between a and p is less than the distance between a and n. Specifically, the loss function is max(||sa-sp||-||sa-sn||+epsilon, 0 is minimized), where sx represents the sentence embeddings for a, n, and p, and ||·|| represents the distance metric. The margin epsilon ensures that sp is at least closer to sa than to sn. Euclidean distance is used as the metric, and the margin is set to 1.

[0073] Figure 5 shows an exemplary block diagram of a biencoder 500 according to the embodiments disclosed herein. Biencoders belong to a class of transformer-based language models specifically designed to generate sentence embeddings. These tools were first introduced in the paper "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks" by Reimerset et al., published in August 2019 (available at https: / / arxiv.org / abs / 1908.10084). These tools were introduced as a modification of a pre-trained BERT (bidirectional encoder representation from transformers) network that uses Siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine similarity 502. A description of BERT can be found in the paper "A Primer in BERTology: What we know about how BERT works" by Rogers et al. (available at https: / / arxiv.org / abs / 2002.12327). The objective of these models is to transform sentences into vectors such that similar sentence vectors are close to each other in the vector space, and dissimilar sentence vectors are farther from each other. The models have been successfully applied to many tasks, including large-scale semantic similarity comparison, clustering, and information retrieval via semantic retrieval. While various embodiments using BERT-based models are described herein, other LLM algorithms or other AI-based techniques are also within the scope of the present invention.

[0074] In some embodiments, the biencoder 500 includes a first component 550, a second component 560, and a third component 570. The first component 550 includes one or more MPNet-based models 506, one or more pooling operations 504, and a cosine similarity operation 502, where MPNet stands for "Masked and Permuted Pre-training for Language Understanding" (see the paper by Song et al., April 2020, available at https: / / arxiv.org / abs / 2004.09297). The second component 560 includes an input encoding 522, several (e.g., nine) encoder stacks 514 (i.e., a stack of encoders in series including encoder 512, encoder 516, and encoder 518), and an output encoding state 510. The third component 570 includes layer normalization 530, another layer normalization 538, multi-head attention 534, and a multilayer perceptron (MLP) 528.

[0075] The Sham network architecture can derive a fixed-size vector of input sentences. Semantically similar sentences can be found using similarity measures such as cosine similarity 502 or Manhattan / Euclidean distance. The biencoder class model adds a pooling 504 operation to the output of BERT / RoBERTa to derive a fixed-size sentence embedding, where RoBERTa stands for "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (see Liuet.al., July 2019, available at https: / / arxiv.org / abs / 1907.11692).

[0076] The triple-item function adjusts the network such that the distance between anchor sentence a and a positive sentence p is less than the distance between a and a negative sentence n. The loss function of this triple-item function is described above with reference to Figure 4. Of the biencoder models available on the Sentence-BERT and HUGGING FACE joint platform, the all-mpnet-base-v2 model 506 (see https: / / huggingface.co / sentence-transformers / all-mpnet-base-v2) is applied to implement the embodiments described in this disclosure. This model was developed using a pre-trained MICROSOFT / mpnet-based model 506 (see https: / / huggingface.co / microsoft / mpnet-base) and fine-tuned on a dataset of one billion sentence pairs with the following contrast learning objective: Given a sentence from a pair, the model predicts which sentences from a randomly sampled set of other sentences actually paired with that sentence in the dataset. Formally, the cosine similarity of each possible sentence pair from a batch is calculated, and then the cross-entropy loss is applied by comparing it to that of a true pair. Given an input sentence, it maps the sentence to a multidimensional (e.g., 768-dimensional) dense vector that captures the semantic information of the given sentence.

[0077] The cross-entropy loss is calculated as follows: Given an input sentence, the model calculates similarity scores between that input sentence and N other sentences (where N is a positive integer) that have at least one sentence semantically similar to the input sentence. The softmax operation is applied to these scores to generate a categorical distribution across the N sentences (i.e., the "after softmax" scores here are unified in summation, i.e., 1). To instruct the model to maximize the after softmax score of the sentence that was semantically closest to the input, the cross-entropy loss is calculated as the negative logarithm of the after softmax score of this closest sentence. Minimizing this cross-entropy loss is equivalent to maximizing the softmax score.

[0078] Specifically, this model encodes a given query (i.e., any sentence from an agent-user conversation) and an intent phrase (i.e., a phrase describing the intent), and then uses the cosine similarity of 502 between these phrases to determine the best candidate intent for the given query.

[0079] Cross Encoder Figure 6 shows a schematic diagram of the cross-encoder operation during the “test time” according to the embodiments disclosed herein. In some embodiments, the cross-encoder 616 is a large-scale language model (LLM) including an encoder block and a classification head of a transformer, trained to detect the similarity between two sentences. The phrase 610 is input to the bi-encoder 612, which generates a super-match 614. The super-match 614 is sent to the cross-encoder 616, which returns either “no intent” 618 or “intent” 620. The transformer encoder 634 can then accept sentences (e.g., sentence A 630 and sentence B 632) to generate a similarity score 636.

[0080] Since the model's output is a similarity metric, the cross-encoder takes two sentences as input and generates an output x, where x is a real number between 0 and 1. The value of x represents the similarity probability between two given inputs, where 0 represents dissimilarity and 1 represents identical inputs. The threshold mechanism for the output x given to the model determines whether it is "intended" or "unintended". Heuristically, a threshold of 0.55 seems suitable for many applications. For example, suppose the intent match from the biencoder phase is "greetings". Then all phrases of the intent "greetings" are paired with queries. Thus, if there are five phrases associated with the intent "greetings", five phrase-query pairs (e.g., [phrase_a,query], [phrase_b,query]) are generated. These are sent to the cross-encoder, and a score is determined for each phrase-query pair. If any of these five scores exceeds the threshold (e.g., 0.55), the query is given the intent "greetings". Otherwise, the query is said to have "unintended". The classification head includes a simple feedforward layer that retrieves embeddings from the final transformer encoder layer and generates a single number representing the probability of how similar the two input sentences are.

[0081] Figure 7 shows an exemplary block diagram of a cross-encoder according to an embodiment disclosed herein. While cross-encoder models also belong to the transformer-based language model class, unlike bi-encoders which produce sentence embeddings for a given sentence, cross-encoders produce output values ​​(e.g., 0-1) indicating the similarity of input sentence pairs. A user or system can provide sentences A and B to a bi-encoder individually, resulting in sentence embeddings u and v, respectively. However, with a cross-encoder, the user or system provides two sentences simultaneously by pairing them. A cross-encoder does not produce sentence embeddings. Generally, cross-encoders yield better results than bi-encoders in terms of sentence similarity, but their computational cost can be high, making them impractical for very large-scale semantic retrieval-type operations. Cross-encoders are typically used whenever a user has a predefined set of sentence pairs to score. For example, a user might want similarity scores for 100 sentence pairs.

[0082] Of the cross-encoder models available on the Sentence-BERT and HUGGING FACE collaborative platform, the stsb-roberta-base model 702 (see https: / / huggingface.co / cross-encoder / stsb-roberta-base) is applied to carry out this disclosure. This model was developed by fine-tuning a sentence similarity dataset using a pre-trained Roberta-based model 702 (see https: / / huggingface.co / roberta-base). Given a sentence pair, the model's desired output is a value between 0 and 1, where values ​​closer to 0 indicate dissimilarity between the two sentences, and values ​​closer to 1 indicate similarity between the two sentences. Specifically, the model helps determine whether a candidate tagged as intent by a biencoder for a given query belongs appropriately to its particular intent class.

[0083] In some embodiments, the cross encoder 700 includes a first component 740, a second component 750, a third component 760, and a fourth component 770. The first component 740 includes a linear layer 744 that accepts CLS ("Classification") token embeddings and a sigmoid operator 742. The second component 750 includes a Roberta-based model 702 that accepts sentences (e.g., sentence A, sentence B) and a classification head 704 that outputs values ​​706 (e.g., 0 to 1). The third component 760 includes an input encoding 722, several (e.g., nine) encoder stacks 714 (i.e., a stack of encoders in series including encoder 712, encoder 716, and encoder 718), and an output encoding state 710. The fourth component 770 includes layer normalization 730, another layer normalization 738, multi-head attention 734, and a multilayer perceptron (MLP) 728.

[0084] The biencoder and crossencoder coordinate their actions, with the acquisition stage using the biencoder and the reordering stage using the crossencoder. In particular, the biencoder model is trained on cosine similarity loss and strives to generate high-quality sentence embeddings. In this stage, the system marks the previous query as belonging to a specific intent within a set of intents. This step can be thought of as an "intent vs. intent classifier," that is, its role is to determine which intent a new query belongs to. Next, the reordering stage is the final decision-maker. The model distinguishes between "intended" and "unintended" ("NO_INTENT," i.e., lack of intent, the utterance is unintentional), but if there is no model trained for that particular task, a proxy task is performed instead, and a phrase of a given intent is compared to the previous query. Based on which phrase is most similar, it is determined whether it is "intended" or "unintended" ("NO_INTENT," i.e., lack of intent, the utterance is unintentional).

[0085] While the concept of semantic retrieval is limited to information retrieval and knowledge retrieval, this disclosure uses embedding-based retrieval as a pseudo-task for intent detection. In particular, the system does not directly match queries and phrases in the biencoder stage, but instead matches query embeddings with intent embeddings. Intent embeddings are found by taking the centroid of all phrase embeddings for a particular intent. Biencoders are generally used in information retrieval or knowledge retrieval to fetch the top few (e.g., 5-10) matching sentences or paragraphs, but in this disclosure, generally only the top selection is returned. Furthermore, a threshold, i.e., a hyperparameter, on the similarity between the query and the intent centroid may be used. In some embodiments, the validation set on which the threshold is based maximizes the micro-mean precision and micro-mean recall.

[0086] Finally, while cross-encoders are generally used to reorder and reorder predictions provided by bi-encoders to generate relevant outputs at the top, in this disclosure, given intent recommendations from bi-encoders, the similarity output of the cross-encoder is used as a "pseudo" to determine "intended" versus "unintended" ("NO_INTENT," i.e., lack of intent, the utterance is unintentional). Queries are classified as either "intended" or "unintended" ("NO_INTENT," i.e., lack of intent, the utterance is unintentional) using a threshold on the similarity score from the cross-encoder. Again, this threshold is a hyperparameter and can be obtained by optimizing micro-mean precision and micro-mean recall on a validation set.

[0087] Figure 8 shows an exemplary flowchart illustrating in detail the operation of a scenario detection and conversation tagging system according to embodiments disclosed herein. For each scenario 800 associated with a client, the operation performs steps 802, 804, 806, and 808. In step 802, the system initializes the scenario by associating it with a name label (e.g., a text string) and n phrases, where n is any positive integer. The number of n may differ for each scenario. In step 804, the system uses a biencoder to encode all n phrases into n embedding vectors (i.e., "phrase encoding"). In step 806, the system normalizes the n embedding vectors and determines their centroids to generate a scenario embedding vector (i.e., "scenario encoding"). Finally, in step 808, the system stores the scenario embedding vector in a database. In some embodiments, in step 808, the system stores the scenario embedding vector, the name label, and the list of n embedding vectors in the database as associated groups.

[0088] Next, for each utterance 810 in the conversation between the client-associated agent and the customer, and for each defined scenario, the operation performs steps 812, 814, 816, 818, 820, 822, 824, 826, and 828. In some embodiments, the system decomposes the conversation into a set of utterances; that is, the conversation contains utterances and the biencoder is ready to receive them. In alternative embodiments, the conversation has already been pre-decomposed into a set of utterances. For each utterance 810, the following steps are performed: In step 812, the biencoder receives the utterance. In step 814, the system uses the biencoder to encode the utterance into a conversation context vector. In step 816, the system compares the conversation context vector to scenario embedding vectors stored in the database and generates a similarity score for each scenario embedding vector stored in the database. In some embodiments, the similarity score is a cosine similarity score. In step 818, the system uses the biencoder to determine the best-match scenario embedding vector, i.e., the scenario embedding vector that best matches the conversation context vector of the utterance. In some embodiments, determining the best match involves identifying the scenario embedding vector with the highest similarity score. In step 820, the system pairs the utterance with each of the n phrases associated with the best-match scenario embedding vector to generate ordered pairs such as "(utterance A, phrase 1 of the matched scenario)", "(utterance A, phrase 2 of the matched scenario)", and "(utterance A, phrase 3 of the matched scenario)". In step 822, the system uses a cross-encoder to generate similarity probabilities for each generated ordered pair of (utterance, phrase). In some embodiments, these similarity probabilities correspond to a second similarity score, as referenced in Figure 3. In step 824, the system determines whether the probability for at least one phrase associated with the best-match scenario embedding vector exceeds a predetermined threshold (e.g., 0.55).If the answer is "yes" (step 826), the biencoder considers it "correct" and assigns the best-match scenario name label associated with the best-match scenario embedding vector to the utterance. In some embodiments, other information associated with the best-match scenario, such as intent, is also assigned to the utterance. Otherwise (step 828), the biencoder considers it "incorrect" and assigns the unintended name label (i.e., "unintended") to the utterance.

[0089] Figure 9 shows another flowchart illustrating in detail the operation of the scenario detection and conversation tagging system according to the embodiments disclosed herein. In particular, the flowchart determines the best-matching scenario name label for an utterance during automated scenario detection in a customer-agent conversation.

[0090] The operation begins in step 901.

[0091] In step 902, the system receives a list of multiple scenarios, multiple scenario name labels, and multiple sample phrases through an acquisition stage that includes a biencoder neural network, where each scenario in the multiple scenarios is associated with a name label among the multiple name labels and with a list of sample phrases among the list of sample phrases.

[0092] In step 904, the system, through the acquisition stage, encodes each sample phrase in a list of multiple sample phrases into a phrase encoding, generating a list of multiple phrase encodings.

[0093] In step 906, the system generates multiple scenario encodings based on the acquisition stage, each of which scenario encodings is associated with a scenario among the multiple scenarios, a name label among the multiple scenario name labels, and a list of phrase encodings among the list of phrase encodings, and each of which scenario encodings is associated with a scenario among the multiple scenario encodings is determined by normalizing the centroid of the list of phrase encodings associated with the scenario among the multiple scenarios.

[0094] In step 908, the system stores a list of multiple scenario encodings, multiple scenario name labels, and multiple phrase encodings in the database.

[0095] In step 912, the system receives the utterance according to the acquisition stage.

[0096] In step 914, the system encodes the conversational context vector of the utterance based on the acquisition stage.

[0097] In step 916, the system generates multiple similarity scores for the conversational context vector of the utterance during the acquisition stage, and each of the multiple similarity scores is associated with a scenario encoding from among multiple scenario encodings stored in the database.

[0098] In step 918, the system determines the best-matching scenario encoding from among multiple scenario encodings by selecting the scenario encoding with the highest similarity score among multiple similarity scores based on the acquisition stage.

[0099] In step 920, the system generates multiple sequence pairs by the acquisition stage, where the first component of each sequence pair is an utterance, and the second component of each sequence pair is a phrase encoding from a list of phrase encodings associated with the best-match scenario encoding.

[0100] In step 922, the system generates multiple similarity probabilities through a reordering stage that includes a cross-encoder neural network, and each of the multiple similarity probabilities is associated with an ordered pair among multiple ordered pairs.

[0101] In step 924, the system determines, through the reordering stage, whether at least one of the multiple similarity probabilities exceeds a predetermined threshold.

[0102] In step 926, if at least one of the multiple similarity probabilities exceeds a predetermined threshold, the reordering stage assigns the best-matching scenario name label from among the multiple scenario name labels associated with the best-matching scenario encoding to the utterance.

[0103] Finally, in step 928, the system assigns an unintentional scenario name label to an utterance by the reordering stage if none of the multiple similarity probabilities exceed a predetermined threshold.

[0104] This completes the operation (step 930).

[0105] Figure 10 shows yet another flowchart illustrating in detail the operation of the scenario detection and conversation tagging system according to the embodiments disclosed herein. In particular, the flowchart determines the best-matching scenario name label of an utterance during automated scenario detection in a customer-agent conversation.

[0106] The operation begins in step 1001.

[0107] In step 1012, the system receives the utterance through an acquisition stage that includes a biencoder neural network.

[0108] In step 1014, the system encodes the conversational context vector of the utterance based on the acquisition stage.

[0109] In step 1016, the system generates multiple similarity scores for the conversational context vector of the utterance by the acquisition stage, and each of the multiple similarity scores is associated with a scenario encoding from among multiple scenario encodings, a scenario name label from among multiple scenario name labels, and a list of phrase encodings from among a list of multiple phrase encodings.

[0110] In step 1018, the system determines the best-matching scenario encoding from among multiple scenario encodings by selecting the scenario encoding with the highest similarity score among multiple similarity scores based on the acquisition stage.

[0111] In step 1020, the system generates multiple sequence pairs through the acquisition stage, where the first component of each sequence pair is an utterance, and the second component of each sequence pair is a phrase encoding from a list of phrase encodings associated with the best-match scenario encoding.

[0112] In step 1022, the system generates multiple similarity probabilities through a reordering stage that includes a cross-encoder neural network, and each of the multiple similarity probabilities is associated with an ordered pair among multiple ordered pairs.

[0113] In step 1024, the system determines, through the reordering stage, whether at least one of the multiple similarity probabilities exceeds a predetermined threshold.

[0114] In step 1026, if at least one of the multiple similarity probabilities exceeds a predetermined threshold, the system, through a reordering stage, assigns the best-matching scenario name label from among the multiple scenario name labels associated with the best-matching scenario encoding to the utterance.

[0115] Finally, in step 1028, the system assigns an unintentional scenario name label to an utterance by the reordering stage if none of the multiple similarity probabilities exceed a predetermined threshold.

[0116] This completes the operation (step 1030).

[0117] Figure 11 shows a system configuration diagram illustrating a scenario detection and conversation tagging system according to an embodiment disclosed herein. The system includes access to at least one hardware processor 1110 which is responsible for executing program code 1122 in order to implement module 1130, which is described later. The system includes access to at least one non-temporary physical storage medium 1120, accessible by at least one hardware processor 1110, which stores the program code 1122 that can be executed by the hardware processor 1110. The program code may be stored and distributed between two or more non-temporary physical storage mediums and may be executed by two or more processors. The system may further include a graphical user interface (GUI) 1104 which enables a user 1102 (e.g., a client or agent) to interact with the system module 1130.

[0118] In various embodiments, the scenario detection and conversation tagging system may incorporate the modules and databases described herein. ● A client application 1132 for adjusting the scenario detection and conversation tagging system and for performing the various methods described herein, ●As described in further detail in this disclosure, a biencoder ML model 1134 trained to generate embeddings or context vectors from a sentence, ●As described in further detail in this disclosure, a cross-encoder ML model 1136 trained to compare embeddings and / or context vectors, ●Includes a scenario database 1138 containing data associated with one or more scenarios (e.g., 1140, 1150). For any given scenario, the scenario data 1138 includes one centroid embedding vector, one name label, and several phrase embedding vectors associated with that scenario. For example, for scenario #1 (1140), Figure 11 shows the centroid embedding vector 1142, name label 1144, and phrase embedding vectors 1146, 1148, etc., and for scenario #2 (1150), Figure 11 shows the centroid embedding vector 1152, name label 1154, and phrase embedding vectors 1156, 1158, etc.

[0119] Implementation of a graphical user interface (GUI) Figure 12 shows a first exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention. A default organization 1214 can use the GUI to train the scenario detection and conversation tagging system. The left-hand menu includes options for Home 1202, Analysis 1204, Canvas 1206, Coaching 1208, Calibration 1210, and Settings 1212. Organizational Unit Management 1216 includes options for Users 1218, Teams 1220, Report Sharing 1222, Custom Fields 1224, QA Case Assignment 1226, Notifications 1228, and Audit Logs 1230. The Scripting Engine 1232 includes options for Scenarios 1234, Conversation Tags 1236, Metric Tags 1238, Sentiment Tags 1240, Categories 1242, Rubric Builder 1244, and Automation 1246. The Knowledge Center 1248 includes an optional FAQ 1250.

[0120] When scenario 1234 is selected, the scenario 1252 window allows the user to enter a scenario name, for example, "Positive Vacuum Experience" for a vacuum cleaner product.1254 If no scenarios similar to the entered scenario exist, the GUI displays "No similar scenarios found"1258. The user may then choose to apply the scenario to only customers1262, only agents1264, or both agents and customers1266 (1260). The user is then prompted to enter the number of training sentences that should trigger the scenario (e.g., 5) (1268). The entered phrase1270, for example, "Can suck up dirt well," is then displayed. The user may also provide training sentences that should not trigger the scenario (1280).

[0121] The GUI indicates whether the entered scenario matches any of the accessible stored conversations (1274). In the illustrated case, there are zero matches.

[0122] Figure 13 shows a second exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention. A default organization 1314 can use the GUI to train the scenario detection and conversation tagging system. The left-hand menu includes options for Home 1302, Analysis 1304, Canvas 1306, Coaching 1308, Calibration 1310, and Settings 1312. Organizational Unit Management 1316 includes options for Users 1318, Teams 1320, Report Sharing 1322, Custom Fields 1324, QA Case Assignment 1326, Notifications 1328, and Audit Logs 1330. The Scripting Engine 1332 includes options for Scenarios 1334, Conversation Tags 1336, Metric Tags 1338, Sentiment Tags 1340, Categories 1342, Rubric Builder 1344, and Automation 1346. The Knowledge Center 1348 includes an optional FAQ 1350.

[0123] When scenario 1334 is selected, the scenario 1352 window allows the user to enter a scenario name, for example, "Greetings" 1354, a phrase associated with greetings. If no scenarios similar to the entered scenario exist, the GUI displays "No similar scenarios found" 1358. The user may then choose who the scenario applies to by selecting from customers only 1362, agents only 1364, or both agents and customers 1366 (1360). The user is then prompted to enter the number of training sentences that should trigger the scenario (e.g., 5) (1368). The entered phrase 1370, for example, "Thank you for calling us," is then displayed, and a relevance percentage 1372 is provided. The user may also provide training sentences that should not trigger the scenario (1380).

[0124] The GUI indicates whether the entered scenario matches any of the accessible stored conversations (1374). In the illustrated case, the "Greetings" scenario matched 2,912 out of 10,000 conversations. The most relevant matches (e.g., the top 10) 1376, such as "Thanks for calling in" and "Well, thanks for calling," are shown. A set of near misses 1378, such as "Thank you so much for calling away," is also displayed.

[0125] Figure 14 shows a third exemplary graphical user interface (GUI) for a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention. A default organization 1414 can use the GUI to train the scenario detection and conversation tagging system. The left-hand menu includes options for Home 1402, Analysis 1404, Canvas 1406, Coaching 1408, Calibration 1410, and Settings 1412. Organizational Unit Management 1416 includes options for Users 1418, Teams 1420, Report Sharing 1422, Custom Fields 1424, QA Case Assignment 1426, Notifications 1428, and Audit Logs 1430. The Scripting Engine 1432 includes options for Scenarios 1434, Conversation Tags 1436, Metric Tags 1438, Sentiment Tags 1440, Categories 1442, Rubric Builder 1444, and Automation 1446. The Knowledge Center 1448 includes an optional FAQ 1450.

[0126] When Scenario 1434 is selected, the Scenario 1452 window allows the user to enter a scenario name, for example, "Register," 1454, a phrase associated with registering a new account in an organization. If no scenarios similar to the entered scenario exist, the GUI displays "No similar scenarios found" 1458. The user may then choose who the scenario applies to by selecting from Customers only 1462, Agents only 1464, or Both Agents and Customers 1466 (1460). The user is then prompted to enter the number of training sentences that should trigger the scenario (e.g., 5) (1468). The entered phrase 1470, for example, "I want to sign up," is then displayed, and a relevance percentage 1472 is provided. The user may also provide training sentences that should not trigger the scenario (1480).

[0127] The GUI indicates whether the entered scenario matches any of the accessible stored conversations (1474). In the illustrated case, the "Register" scenario matched 10 out of 10,000 conversations. The most relevant matches (e.g., the top 10) 1476, such as "trying to register this" and "so I need to make an account," are shown. A set of near misses 1478, such as "let me get you over to our enrollment department," is also displayed.

[0128] Figure 15 shows a fourth exemplary graphical user interface (GUI) relating to a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention. The default organization can use the GUI to view the call history and analysis of a selected customer or agent, e.g., John Smith 1513, as well as information and analysis of a specific conversation, such as indicated by the “Conversation ID” 1590, as analyzed by the scenario detection and conversation tagging system. The left-hand menu includes options for Home 1502, Analysis 1504, Canvas 1506, Coaching 1508, Calibration 1510, and Settings 1512. Various scores for the selected customer or agent are provided, e.g., QA score 1516, Insta score 1518, CSAT score 1520, and Sentiment score 1522. A phone rubric 1524 is provided.

[0129] Users can review snippets 1530 and transcripts 1532 of specific conversations. For example, a particular snippet in which an agent says, "I can help with that. Let me check. I'm sorry," might be tagged as an agent representing "empathy" 1534.

[0130] Machine learning (ML) and neural networks Machine learning (ML) algorithms are characterized by their ability to improve their performance on a task over time, even if they are not explicitly programmed with rules for performing the task (i.e., learning). A machine learning (ML) model is the output produced when an ML algorithm is trained on data. As described herein, embodiments of the present invention use one or more artificial intelligence (AI) algorithms and machine learning (ML) algorithms for scenario detection and conversation tagging. Various exemplary ML algorithms are within the scope of the present invention. Below are exemplary ML techniques for carrying out various embodiments of the present invention.

[0131] Neural Network A neural network is a computational model inspired by the structure of the human brain, consisting of interconnected units called neurons that work together to process information. Neural networks are a type of ML algorithm that is particularly effective at recognizing patterns and making predictions based on complex data. Because neural networks have the ability to learn from large amounts of data and improve their performance over time, they are widely used in various applications such as image and speech recognition, as well as natural language processing. Figure 16 shows the operation of a basic neural network relating to a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention.

[0132] Figure 16 shows a single-layer neural network, also known as a single-layer perceptron. Such a perceptron is a building block of the neural network layer included in the biencoder and crossencoder according to embodiments of the present invention. The operation of the single-layer neural network involves the following steps.

[0133] 1. Input: element v j The system receives an input vector v1604 having the following properties: j∈[1.n] represents the j-th input, and each element of the vector corresponds to neuron 1606 in the input layer. In the scenario detection and conversation tagging system, the input vector may be a character, syllable, or word of a phrase, utterance, or sentence. The input vector may also be an element of a phrase context vector or embedding vector, as described in this disclosure.

[0134] 2. Transfer function: Weights W corresponding to each input j Multiply by 1608. Then these weighted inputs are summed as a transfer function and activation function

number

[0135] Each neuron in a neural network may have a bias value of 1612, which is added to the weighted sum of inputs to that neuron. Both the weight and bias values ​​are learned during the training process. The purpose of the bias is to provide all neurons with a learnable constant value that can help the model better fit the data. When bias is present, the total input to the activation function is

number

[0136] 3. Activation Function: The total input is passed through the activation function 1614. The activation function σ determines the activation value ο1618, which is the output of the neuron. Typically, this is a nonlinear function such as the sigmoid function or the ReLU (Normalized Linear Unit) function. The threshold θ1616 of the activation function is the value that determines whether the neuron is activated or not. For some activation functions, such as the step function, the threshold is a specific value. If the total input is above the threshold, the neuron outputs a constant value, and if the total input is below the threshold, the neuron outputs a zero value. For other activation functions, such as the sigmoid function or the ReLU (Normalized Linear Unit) function, the threshold is not a specific value, but rather a transition point on the function's curve.

[0137] 4. Output: The activation value ο1618 is the output of the activation function. This value is passed to the next layer in the network, and in the case of the last layer, it becomes the final output. In scenario detection and conversation tagging systems, the final output of the neural network can be an embedding or context vector representing a scenario phrase, or a scenario intent Boolean variable.

[0138] Figure 17 shows an overview of the training process for a neural network that can be used in a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention.

[0139] In a scenario detection and conversation tagging system, the neural network 1702 being trained may be a biencoder or a crossencoder as described in this disclosure. Training the neural network involves repeatedly updating the network's weights and biases 1710 to minimize the difference between the predicted output 1704 and the true output or target output 1706, so that the predicted output 1704 is the result produced by the network as a set of inputs from the dataset passes through. The true output or target output 1706 is the true desired result. The difference between the predicted output and the true output is calculated using a loss function 1708 to quantify the error produced by the network in its prediction. In a scenario detection and conversation tagging system, the neural network's predicted output 1704 and true / target output 1706 may be embeddings or context vectors representing scenario phrases, or scenario intent Boolean variables.

[0140] The loss function is part of the cost function 1708 and is a measure of how well the network performs across the entire dataset. The goal of training is to minimize the cost function 1708. This is achieved by iteratively adjusting the network's weights and biases 1710 in a direction that leads to the steepest descent of the cost function. The size of these adjustments is determined by the learning rate 1708, a hyperparameter that controls the amount by which the weights and biases change in each iteration. A smaller learning rate results in smaller changes and slower convergence to the minimum of the cost function. Conversely, a larger learning rate results in larger changes and faster convergence, but carries the risk of exceeding the minimum.

[0141] Training a neural network is a combination of forward and backpropagation processes. Forward propagation is the process by which input data is passed through the network from the input layer to the output layer. During forward propagation, the network's weights and biases are used to compute the output for a given input. Backpropagation, on the other hand, is the process used to update the network's weights and biases 1710 based on the output error (e.g., a cost function) 1708. After forward propagation by the neural network 1702, the network's output 1704 is compared to the true output 1706, and the error 1708 is computed. This error is then propagated through the network, starting from the output layer and moving to the input layer. The weights and biases 1710 are adjusted in a way that minimizes this error. This process is repeated multiple times or epochs until the network can make accurate predictions.

[0142] The neural network training method described above, in which the network is trained on a labeled dataset (e.g., sample pairs of input user prompts and corresponding output recommendations) and the true output is known, is called supervised learning. In unsupervised learning, the network is trained on an unlabeled dataset with the aim of discovering hidden patterns or structures in the data. The network is not provided with a true output, and the training is based on the inherent properties of the data. Furthermore, reinforcement learning is a type of learning in which an agent learns to make decisions based on the rewards or penalties it receives based on its actions. Reinforcement learning is usually independent of existing datasets, although some forms of reinforcement learning can use a database of past actions, states, and rewards during the learning process. As is evident from the following overview, any neural network training method using a labeled dataset is within the scope of the methods and systems described herein.

[0143] Figure 18 provides additional details regarding the training process according to an exemplary embodiment of the present invention.

[0144] Transformer Model Architecture The transformer architectures used by the biencoders and crossencoders shown in Figures 4-7 are neural network designs introduced in the paper "Attention is All You Need" by Vaswani et al., published in June 2017 (available at https: / / arxiv.org / abs / 1706.03762), and are incorporated herein by reference as if fully described herein. Large-scale language models (LLMs) are heavily dependent on the transformer architecture.

[0145] The architecture (see Figure 1 in the reference paper) is based on the concept of "attention," which allows the model to focus on different parts of the input sequence when producing an output. The transformer consists of an encoder and a decoder. The encoder processes the input data, and the decoder generates the output. Each of these components consists of multiple layers, including a self-attention layer and a point-by-point fully connected layer.

[0146] The self-attention layer of the transformer model allows for the comparison of relationships between different parts of the input sequence when generating the output, thereby enabling the capture of long-range dependencies in the data. Meanwhile, the fully connected layer is used to transform the output of the self-attention layer, adding complexity and depth to the model's learning ability.

[0147] Transformer models are known for their ability to process long data sequences and are particularly effective for tasks such as machine translation and text summarization. In transformer architectures, positional encoding is used to provide model information about the relative positions of words within the input sequence. Since the model itself does not possess an inherent sense of order or sequence, positional encoding is a way to inject some order information into an order-independent attention mechanism.

[0148] Embedding vector space In the context of neural networks, tokenization refers to the process of transforming input and output spaces, such as natural language text or programming code, into discrete units or "tokens." This process allows the network to effectively process and understand data by transforming complex structures into manageable individual elements from which the model can learn and generate.

[0149] In training neural networks, embeddings function as a form of word embedding that transforms discrete categorical variables (i.e., tokens) into a continuous vector space (i.e., embedding vectors). This transformation process captures the semantic properties of tokens, allowing tokens with similar meanings to have similar embeddings. These embeddings provide a dense representation of tokens and their semantic relationships. Embeddings are typically represented as vectors, but may also be represented as matrices or tensors.

[0150] The input to a transformer typically requires a transformation from the input space (e.g., natural language token space) to the embedding space. This process, called "encoding," converts discrete inputs (tokens) into continuous vector representations (embeddings). This transformation is a prerequisite for the transformer model to process the input data and understand the semantic relationships between tokens (e.g., words). Similarly, the output of a transformer typically requires a transformation from the embedding space to the output space (e.g., natural language tokens, programming code tokens, etc.), a process called "decoding." Therefore, both the training and evaluation (i.e., its use in deployment) of a neural network take place within the embedding space.

[0151] The remainder of this document infers the processes of tokenization, encoding, decoding, and detokenization. In other words, the processes described below occur in the “embedding space.” Therefore, the tokenization and encoding of the training data and input prompts are implied, even if they are not explicitly stated or explained. Similarly, the decoding and detokenization of the neural network output are also implied.

[0152] Training and fine-tuning machine learning (ML) module Figure 18 is an exemplary flowchart illustrating various phases and datasets involved in training a machine learning algorithm for a scenario detection and conversation tagging system according to an exemplary embodiment of the present invention.

[0153] The training process begins in step 1810, where scenario and conversation data are captured, acquired, assimilated, or generated. In step 1820, the captured data is preprocessed or prepared. In step 1830, the ML model (e.g., a biencoder or crossencoder) is trained using the training data 1825. In step 1840, the ML model is evaluated, validated, and tested, and any areas for further improvement are fed back to step 1830 for additional training. Once its performance is acceptable, in step 1850, the optimal model parameters are selected.

[0154] The training data 1825 is a recorded dataset containing multiple instances of system inputs and correct results. This is used to train an ML model to optimize its performance on a specific target task. In Figure 18, the training data 1825 may also include a subset for validation and testing of the ML model as part of training iterations 1830 and 1840. For NN-based ML models, the quality of the output may depend on (a) the NN architecture design and hyperparameter configuration, (b) the optimization of NN coefficients or parameters, and (c) the quality of the training dataset. These components can be improved and optimized using various methods. For example, the training data 2825 may be augmented through an augmentation process.

[0155] In some embodiments, additional fine-tuning phases, including iterative fine-tuning 1860 and evaluation, validation, and test 1870 steps, are performed using fine-tuning data 1855. Fine-tuning in machine learning is a process that involves obtaining a selected 1850 pre-trained model and further adjusting or "tuned" its parameters to better suit a particular task or fine-tuning dataset 1855. This technique is particularly useful when dealing with deep learning models that are trained on a large, general training dataset 1825 and are intended to be applied to more specialized tasks or smaller datasets. The goal is to enable the model to perform better on the more specialized task before it by leveraging and refining the knowledge it has already acquired during its initial training (often called transfer learning).

[0156] The fine-tuning process typically begins with a model already trained on a large benchmark training dataset 1825, such as ImageNet, for an image recognition task. The existing weights of the model learned from the original training serve as the starting point. During fine-tuning, the model is further trained on a new fine-tuning dataset 1855, which may contain data of different classes or types than the original training set. This additional training phase allows the model to adjust its weights to better capture the characteristics of the new fine-tuning dataset 1855, thereby improving its performance on the specific task being fine-tuned.

[0157] In some embodiments, an additional test and validation phase 1880 is performed using test and validation data 1875. Both testing and validation of an ML model refer to the process of evaluating the model's performance on a separate dataset 1875 that was not used during training, in order to ensure that it generalizes well to new, unseen data. Validating an ML model helps prevent overfitting by ensuring that the model's performance generalizes beyond the training data.

[0158] The validation phase is considered part of ML model development and may lead to further iterations of fine-tuning, while the testing phase is the final evaluation of the model's performance after it has been trained and validated. The testing phase provides an unbiased final evaluation of the model's performance, reflecting how well the model is expected to perform on unseen data, and is usually conducted after the model has finished to ensure the evaluation is unbiased.

[0159] Once the model has been trained (1830), selected (1850), fine-tuned with optional selections (1860), and validated / tested (1880), it is unfolded (1890). The unfolded ML model, such as biencoders and crossencoders (1895), typically receives new data (1885) that has been preprocessed (1820).

[0160] In machine learning, data preprocessing (1820) is aligned with the model development phase. During model training (1830), preprocessing involves cleaning, normalizing, and transforming raw data into a format suitable for the learning pattern. For fine-tuning (1860), preprocessing adapts the data to match the distribution of the specific target task, ensuring that the pre-trained model can effectively transmit its knowledge. Preprocessing for validation (1880) reflects the preprocessing for training to accurately assess model generalization without losing information from the training set. Finally, in deployment (1890 and 1895), preprocessing ensures that real-world data matches the expectations of the trained model, often involving dynamic adjustments to maintain consistency with the training and validation stages.

[0161] Machine learning algorithms Various exemplary ML algorithms fall within the scope of this invention. Such machine learning algorithms include, but are not limited to, random forests, nearest neighbors, decision trees, support vector machines (SVMs), AdaBoost, gradient boosting, Bayesian networks, evolutionary algorithms, and various neural networks (including deep learning networks (DLNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs)).

[0162] ML modules based on transformers and large language models (LLMs) are particularly well-suited to the tasks described herein. The online article by S. Raschka (published February 7, 2023, and available at https: / / sebastianraschka.com / blog / 2023 / llm-reading-list.html), “Understanding Large Language Models -- A Transformative Reading List,” describes various LLM architectures within the scope of the methods and systems described herein, and its entirety is incorporated herein by reference as if it were fully described herein.

[0163] The input to each of the listed ML modules is a feature vector containing the input data described above for each ML module. The output of each ML module is a feature vector containing the corresponding output data described above for each ML module.

[0164] Before deployment, each of the ML modules listed above may be trained on one or more respective sample input datasets and one or more corresponding sample output datasets. The input and output training datasets may be generated from a database containing the history of input and output instances, or they may be generated synthetically by experts in the subject area.

[0165] System Configuration Example Exemplary embodiments of this disclosure may include one or more servers (management computing entities), one or more networks, and one or more clients (user computing entities). Each of these components, entities, devices, and systems (similar terms used interchangeably herein) may communicate with one another directly or indirectly, for example, via the same or different wired or wireless networks.

[0166] Figure 19 provides a schematic diagram illustrating several embodiments of the present invention, showing a server (management computing entity) 1902 connected via a network 1920 and a client (user computing entity) 1930 used for communication within a scenario detection and conversation tagging system. While Figure 19 shows various system entities as separate standalone entities, the various embodiments are not limited to this particular architecture. Furthermore, the terms “client device,” “client computing entity,” “edge device,” and “edge computing system” are equivalent and are used interchangeably herein.

[0167] Exemplary Management Computing Entity Figure 19 provides a schematic diagram of a server (management computing entity, 1902) according to one embodiment of the present disclosure. Generally, the terms computing entity, computer, entity, device, system, and / or similar words used interchangeably herein may refer to, for example, one or more computers, computing entities, desktop computers, mobile phones, tablets, phablets, notebooks, laptops, distributed systems, game consoles, watches, glasses, iBeacon, proximity beacon, key fob, radio frequency identification (RFID) tag, earphones, scanners, televisions, dongles, cameras, wristbands, wearable items / devices, kiosks, input terminals, servers or server networks, blades, gateways, switches, processing devices, processing entities, set-top boxes, repeaters, routers, network access points, base stations, and / or any combination of devices or entities adapted to perform the functions, operations, and / or processes described herein. Such functions, operations, and / or processes may include, for example, transmitting, receiving, operating, processing, displaying, storing, determining, creating / generating, monitoring, evaluating, and / or comparing (similar terms used interchangeably herein). In one embodiment, these functions, operations, and / or processes can be performed with data, content, and / or information (similar terms used interchangeably herein).

[0168] As shown, in one embodiment, the management computing entity (1902) may also include one or more communication interfaces (1910) for communicating with various computing entities, such as by communicating data, content, and / or information (similar terms used herein interchangeably) that can be transmitted, received, operated, processed, displayed, and / or stored.

[0169] As shown in Figure 19, in one embodiment, the management computing entity (1902) may include, or communicate with, one or more processors (i.e., processing elements 1904, also called processors and / or processing circuits, and similar terms used interchangeably herein) that communicate with other elements within the management computing entity (1902), for example, via a bus. Naturally, the processors (1904) may be embodied in several different ways. For example, the processors (1904) may be embodied as one or more complex programmable logic devices (CPLDs), microprocessors, multicore processors, simultaneous processing entities, application-specific instruction set processors (ASIPs), microcontrollers, and / or controllers. Furthermore, the processors (1904) may be embodied as one or more other processing devices or circuits. The term "circuit" may refer to an entire hardware embodiment or a combination of hardware and computer program products. Accordingly, the processor (1904) may be embodied as an integrated circuit, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic array (PLA), a hardware accelerator, and / or other circuitry. Naturally, the processor (1904) may be configured for a particular use, or may be configured to execute instructions stored in a volatile or non-volatile (or non-transient) medium, or otherwise instructions accessible to the processor (1904). Thus, whether configured by hardware or computer program products, or a combination thereof, the processor (1904), when configured in such a way, may be capable of performing steps or operations according to embodiments of the present disclosure.

[0170] In one embodiment, the management computing entity (1902) may further include, or communicate with, non-temporary memory (non-volatile media, non-volatile storage, non-temporary storage, memory, memory storage, and / or memory circuits, similar terms used interchangeably herein). In one embodiment, the non-temporary memory or storage may include, but not limited to, one or more non-temporary memory or storage media (1906), including, but not limited to, hard disks, ROMs, PROMs, EPROMs, EEPROMs, flash memory, MMCs, SD memory cards, memory sticks, CBRAMs, PRAMs, FeRAMs, NVRAMs, MRAMs, RRAMs, SONOSs, FJG RAMs, millipede memory, and / or racetrack memory. To be recognized, the non-volatile (or non-temporary) storage or memory media may store databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, bytecode, compiled code, interpreted code, machine language, and / or executable instructions. The terms database, database instance, and / or database management system (similar terms used interchangeably herein) may also refer to a collection of records or data stored on a computer-readable storage medium using one or more database models, such as a hierarchical database model, a network model, a relational model, an entity-relationship model, an object model, a document model, a semantic model, and / or a graph model.

[0171] In one embodiment, the management computing entity (1902) may further include, or communicate with, volatile memory (volatile storage, memory, memory storage, memory and / or circuitry, similar terms used interchangeably herein). In one embodiment, the volatile storage or memory may also include one or more volatile storage or memory media 1908, including but not limited to RAM, DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM, cache memory, and / or register memory. The volatile storage or memory media may be used to store at least portions of databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, bytecode, compiled code, interpreted code, machine code, and / or executable instructions, which are executed, for example, by a processor (1904). Therefore, certain aspects of the operation of a managed computing entity (1902) may be controlled with the help of a processor (1904) and an operating system, using, as recognized, databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, bytecode, compiled code, interpreted code, machine language, and / or executable instructions.

[0172] As shown, in one embodiment, a management computing entity (1902) may also include one or more communication interfaces (1910) for communicating with various computing entities, such as by communicating data, content, and / or information (similar terms used interchangeably herein) that can be transmitted, received, operated, processed, displayed, and / or stored. Such communication may be performed using wired data transmission protocols such as fiber distributed data interface (FDDI), digital subscriber line (DSL), Ethernet, asynchronous transport mode (ATM), Frame Relay, Data Services Interface Standard over Cable (DOCSIS), or any other wired transmission protocol. Similarly, the Management Computing Entity (1902) may be configured to communicate over a wireless external communication network using any of the following protocols: General-Purpose Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access 2000 (CDMA2000), CDMA2000 1X (1xRTT), Wideband Code Division Multiple Access (WCDMA), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), Long-Term Evolution (LTE), Evolutionary Universal Terrestrial Radio Access Network (E-UTRAN), Evolutionary Data Optimization (EVDO), High Speed ​​Packet Access (HSPA), High Speed ​​Downlink Packet Access (HSDPA), IEEE 802.11 (Wi-Fi), Wi-Fi Direct, 802.16 (WiMAX), Ultra-Wideband (UWB), Infrared (IR) Protocol, Near Field Communication (NFC) Protocol, Wibree, Bluetooth Protocol, Wireless Universal Serial Bus (USB) Protocol, and / or any other wireless protocol.

[0173] Although not shown, the management computing entity (1902) may include or communicate with one or more input elements such as keyboard input, mouse input, touchscreen / display input, motion input, motor input, audio input, pointing device input, joystick input, and / or keypad input. The management computing entity (1902) may also include or communicate with one or more output elements (not shown) such as audio output, video output, screen / display output, motion output, and / or motor output.

[0174] As should be understood, one or more components of the managed computing entity (1902) may be located far apart from other components of the managed computing entity (1902), such as in a distributed system. Furthermore, one or more components may be combined, and additional components performing the functions described herein may be included in the managed computing entity (1902). Thus, the managed computing entity (1902) can be adapted to meet various needs and circumstances. As should be recognized, these architectures and descriptions are provided for illustrative purposes only and are not limited to various embodiments.

[0175] Example User Computing Entity The user may be a human individual, a company, an organization, an entity, a department within an organization, a representative of an organization and / or a person, and / or an artificial user such as an algorithm, artificial intelligence, or other software that takes an interface. Figure 19 further provides an exemplary schematic representation of a client user computing entity 1930 that can be used in conjunction with embodiments of the present disclosure. Generally, the terms device, system, computing entity, entity, and / or similar words used interchangeably herein may refer to, for example, one or more computers, computing entities, desktops, mobile phones, tablets, phablets, notebooks, laptops, distributed systems, game consoles, watches, glasses, key fobs, radio frequency identification (RFID) tags, earphones, scanners, televisions, cameras, wristbands, kiosks, input terminals, servers or server networks, blades, gateways, switches, processing devices, processing entities, set-top boxes, repeaters, routers, network access points, base stations, and / or any combination of devices or entities adapted to perform the functions, operations, and / or processes described herein. The user computing entity can be operated by various parties. Such functions, operations, and / or processes may include, for example, transmitting, receiving, operating, crawling, extracting, processing, displaying, storing, determining, creating, generating, generating a display, monitoring, evaluating, comparing, and / or similar terms used interchangeably herein. In various embodiments, these functions, operations, and / or processes may be performed with machine-readable data or human-readable data, content, information, and / or similar terms used interchangeably herein. Furthermore, in embodiments of the present invention, the computing device 1930 may be a general-purpose computing device equipped with a dedicated graphics processing unit (GPU) 1936 and an artificial intelligence module 1938.In some embodiments, the computing device 1930 may be built on a backplane 1956 connected to a power supply 1958. Alternatively, it may be implemented in the cloud using a logical and / or physically distributed architecture.

[0176] As shown in Figure 19, the user computing entity 1930 may include an antenna 1970, a radio frequency (RF) transceiver 1932, and a processor unit 1940 that provides and receives signals from the transceiver. The signals provided to and received from the transceiver may include signal information in accordance with the applicable radio system air interface standard. In this regard, the user computing entity (1930) may operate in accordance with one or more air interface standards, communication protocols, modulation types, and access types. More specifically, the user computing entity 1930 may operate in accordance with any of several radio communication standards and protocols, such as those described above with respect to the management computing entity 1902. In certain embodiments, the user computing entity 1930 may operate in accordance with several wireless communication standards and protocols, such as UMTS, CDMA2000, 1xRTT, WCDMA, TD-SCDMA, LTE, E-UTRAN, EVDO, HSPA, HSDPA, Wi-Fi, Wi-FiDirect, WiMAX, UWB, IR, NFC, Bluetooth, and / or USB. Similarly, the user computing entity 1930 may operate in accordance with any of several wireless communication standards and protocols, such as those described above with respect to the management computing entity 1902.

[0177] Through these communication standards and protocols, the user computing entity 1930 can communicate with various other entities using concepts such as unstructured add-on service data (USSD), short message service (SMS), multimedia messaging service (MMS), dual-tone multi-frequency signaling (DTMF), and / or subscriber identification module dialer (SIM dialer). The user computing entity 1930 may also download changes, add-ons, and updates to its firmware, software (including, for example, executable instructions, applications, and program modules), and operating system.

[0178] In some embodiments, the processing unit 1940 may be embodied in several different ways. For example, the processing unit 1940 may be embodied as one or more complex programmable logic devices (CPLDs), microprocessors, multicore processors, simultaneous processing entities, application-specific instruction set processors (ASIPs), microcontrollers, and / or controllers. Furthermore, the processing unit 1940 may be embodied as one or more other processing devices or circuits. The term "circuit" can refer to an entire hardware embodiment or a combination of hardware and computer program products. Thus, the processing unit 1940 may be embodied as an integrated circuit, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic array (PLA), a hardware accelerator, and / or other circuits. Naturally, the processing unit 1940 may be configured for a particular use or configured to execute instructions stored in a volatile or non-volatile medium, or otherwise instructions accessible to the processing unit. Therefore, whether configured by hardware or computer program products, or a combination thereof, the processing unit 1940 may, when configured in such a way, be capable of performing steps or operations according to embodiments of the present invention.

[0179] In some embodiments, the processing unit 1940 may include a control unit 1942 and a dedicated arithmetic unit (ALU) 1944 for performing arithmetic and logical operations. In some embodiments, the user computing entity 1930 may include a graphics processing unit (GPU) 1946 for special parallel processing tasks, and / or an artificial intelligence (AI) module or accelerator 1948 also specialized for applications including artificial neural networks and / or machine learning. In some embodiments, the processing unit 1940 can be coupled with the GPU 1946 and / or the AI ​​accelerator 1948 to distribute and coordinate processing tasks.

[0180] In some embodiments, the computing entity 1930 may include a user interface, which includes an input interface 1950 and an output interface 1952, respectively, coupled to the processing unit 1940. The user input interface 1950 may include any of several devices or interfaces that enable the computing entity 1930 to receive data, such as a keypad (hard or soft), a touch display, a microphone for voice / speech, and a camera for a motion or posture interface. The user output interface 1952 may include any of several devices or interfaces that enable the computing entity 1930 to provide information to the user, such as a touch display or a speaker for audio output. In some embodiments, the output interface 1952 may connect the computing entity 1930 to an external speaker or projector for audio or image output. In some embodiments, the user interfaces 1950 and 1952 integrate multimodal data and accommodate both humans and bots / algorithms. Some examples of human interfaces include dashboard interfaces, workflow-based interfaces, conversational interfaces, and spatial computer interfaces. Some examples of bot / algorithm interfaces include code interfaces and text-based API interfaces.

[0181] The user computing entity 1930 may also include volatile and / or non-volatile storage or memory 1960 that can be embedded and / or removable. For example, non-volatile or non-temporary memory may be hard disks, ROMs, PROMs, EPROMs, EEPROMs, flash memory, MMCs, SD memory cards, memory sticks, CBRAMs, PRAMs, FeRAMs, NVRAMs, MRAMs, RRAMs, SONOS, FJG RAMs, millipede memory, and / or racetrack memory. Volatile memory may be RAMs, DRAMs, SRAMs, FPM DRAMs, EDO DRAMs, SDRAMs, DDR SDRAMs, DDR2 SDRAMs, DDR3 SDRAMs, RDRAMs, TTRAMs, T-RAMs, Z-RAMs, RIMMs, DIMMs, SIMMs, VRAMs, cache memory, and / or register memory. Volatile and non-volatile storage or memory can store operating systems 1962, application software 1964, data 1966, databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, bytecode, compiled code, interpreted code, machine language, and / or executable instructions, etc., to perform the functions of a user computing entity (1930). As shown, this may include user applications residing in the entity, or user applications accessible via a browser or other user interface for communicating with the administrative computing entity (1902) and / or various other computing entities.

[0182] In some embodiments, the user computing entity 1930 may include one or more components or functions identical or similar to those of the management computing entity 1902, as described in more detail above. As recognized, these architectures and descriptions are provided for illustrative purposes only and are not limited to various embodiments.

[0183] In some embodiments, the computing entities 1910 and / or 1930 may communicate with other computing devices and / or external devices such as access points to receive information such as software or firmware, or they may transmit information from the memory of the computing entities to external systems or devices such as servers, computers, or smartphones.

[0184] In some embodiments, two or more computing entities, such as 1910 and / or 1930, may establish connectivity using a network such as 1920 that utilizes any of the previously listed network protocols. In some embodiments, computing entities may use network interfaces such as 1910 and 1934 to communicate with each other, including by communicating data, content, information, and / or similar terms used herein interchangeably, which can be transmitted, received, operated, processed, displayed, and / or stored.

[0185] In some embodiments of the present invention, the entire system may be implemented and provided to end users and operators via the internet in a so-called cloud implementation. Local installation of software or hardware is not required, and end users and operators can access the system of the present invention directly via the internet using either a web browser or similar software on a client device, which may be a desktop, laptop, or mobile device. This eliminates the need for custom software installation on the client side, improves the flexibility of service delivery (software as a service), enhances user satisfaction, and makes the system easier to use. Various business models, profit models, and delivery mechanisms are envisioned for the present invention, and all are considered to fall within the scope of the invention.

[0186] Details of additional hardware and software implementations While exemplary processing systems are described above, embodiments of the subject matter and functional operation described herein can be implemented in computer software, firmware, or hardware, or one or more combinations thereof, including other types of digital electronic circuits, or structures disclosed herein and their structural equivalents.

[0187] The subject matter and embodiments of operation described herein can be implemented in computer software, firmware, or hardware, or one or more combinations thereof, including digital electronic circuits, or structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions, to be encoded in a computer storage medium for execution by an information / data processing device, or to control the operation of an information / data processing device. Alternatively, or in addition, program instructions can be encoded in artificially generated propagating signals, such as mechanically generated electrical, optical, or electromagnetic signals, and generated to encode information / data for transmission to a suitable receiving device for execution by an information / data processing device. The computer storage medium can be, or may be, a computer-readable storage device, a computer-readable storage board, a random or serial access memory or device, or one or more combinations thereof. Furthermore, although the computer storage medium is not a propagating signal, the computer storage medium can be a source or destination for computer program instructions encoded in artificially generated propagating signals. Computer storage media can also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices), or may be contained within them.

[0188] The operations described herein can be performed as operations performed by an information / data processing device on information / data stored in or received from one or more computer-readable storage devices.

[0189] The term "data processing unit" encompasses all types of devices, equipment, and machines for processing data, including, for example, programmable processors, computers, systems on a chip, or a combination thereof. Equipment may include special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, equipment may also include code that creates the execution environment for the computer program at hand, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or code comprising one or more of these. Equipment and execution environments can realize infrastructure for various different computing models, including web services, distributed computing, and grid computing infrastructure.

[0190] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and programs can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs may, but do not necessarily, correspond to files in a file system. Programs can be stored in a part of a file that holds other programs or information / data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program of interest, or in multiple collaborative files (e.g., files that store one or more modules, subprograms, or parts of code). Computer programs can be deployed to run on one computer or on multiple computers located in one place or distributed across multiple locations and interconnected by a communication network.

[0191] The processes and logical flows described herein are performed by one or more programmable processors that execute one or more computer programs, and can perform actions by operating on input information / data and producing outputs. Processors suitable for executing computer programs include, for example, both general-purpose processors and special-purpose processors, as well as one or more processors of either type of digital computer. Generally, processors receive instructions and information / data from read-only memory (ROM), random-access memory (RAM), or both. The basic elements of a computer are a processor for performing actions according to instructions, and one or more memory devices for storing instructions and data. Generally, a computer may have one or more mass storage devices for storing data, including, for example, magnetic, magneto-optical disks, or optical disks, or be operablely coupled to receive information / data from or transfer information / data to or from such mass storage devices. However, a computer may not have such devices. Suitable devices for storing computer program instructions and information / data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices including EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be complemented by or incorporated into dedicated logic circuits.

[0192] To provide user interaction, embodiments of the subject matter described herein can be implemented in a computer having a display device for displaying information / data to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal screen) monitor, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide user interaction. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and the input from the user can be acoustic, vocal, or haptic input. Furthermore, the computer can interact with the user by sending and receiving documents to and from the device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0193] Embodiments of the subject matter described herein can be implemented in a computing system including a backend component, e.g., an information / data server, or a middleware component, e.g., an application server, or a frontend component, e.g., a client computer having a graphical user interface or a web browser on which a user can interact with embodiments of the subject matter described herein, or in any combination of such backend, middleware, or frontend components. The components of the system can be interconnected by digital information / data communications of any form or medium, e.g., communication networks. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer networks).

[0194] A computing system can include a client and a server. The client and server are generally geographically distant from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server sends information / data (e.g., an HTML page) to the client device (for example, to display information / data to a user interacting with the client device and to receive user input from that user). Information / data generated on the client device (e.g., the results of user interactions) can be received by the server from the client device.

[0195] While this specification provides details of many specific embodiments, these should not be construed as limitations on the scope of any embodiment or the scope of what is claimed, but rather as descriptions of features specific to a particular embodiment. Certain features described herein in the context of a separate embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any preferred sub-combination. Furthermore, features that function in a particular combination, or even if initially claimed as such, may be described above, although one or more features from a claimed combination may be removed from the combination, and the claimed combination may cover sub-combinations or variations of sub-combinations.

[0196] Similarly, although the operations are depicted in a specific order in the diagrams, this should not be understood as requiring that such operations be performed in the specific order or sequence shown, or that all illustrated operations be performed, in order to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0197] Accordingly, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions described in the claims may be performed in a different order, and the desired results may still be achieved. Furthermore, the processes shown in the accompanying figures do not necessarily need to be in the specific order or sequence shown in the figures in order to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0198] In some embodiments of the present invention, the entire system can be implemented and provided to end users and operators via the internet in a so-called cloud implementation. Local installation of software or hardware is unnecessary, and end users and operators can access the system of the present invention directly via the internet using either a web browser or similar software on a client device, which may be a desktop, laptop, or mobile device. This eliminates the need for custom software installation on the client side, improving the flexibility of (software as a service) service delivery, increasing user satisfaction, and ease of use. Various business models, profit models, and delivery mechanisms of the present invention are envisioned and are all considered to be within the scope of the invention.

[0199] Generally, methods used to implement embodiments of the present invention can be implemented as part of an operating system, or as part of a specific application, component, program, object, module, or set of instructions referred to as a “computer program” or “computer code.” A computer program typically contains one or more instructions set at various points in time in the computer’s various memories and storage, and when read and executed by one or more processors of the computer, causes the computer to perform the actions necessary to execute elements including various aspects of the present invention. Furthermore, although the present invention has been described in the context of fully functional computers and computer systems, those skilled in the art will understand that various embodiments of the present invention can be distributed as various forms of program products, and that the present invention applies equally regardless of the specific type of machine or computer-readable medium used to actually affect distribution. Examples of computer-readable media include, but are not limited to, volatile and non-volatile (or non-temporary) memory devices, floppy disks and other removable disks, hard disk drives, optical discs, and other recordable media, as well as compact disc read-only memory (CD-ROM), digital multipurpose discs (DVDs), and digital and analog communication media.

[0200] conclusion Those skilled in the art will know that the use cases, structures, schematics, flowcharts, and steps may be presented in any order or in any combination, but the inventive concept of the present invention remains within the broader scope of the invention. All embodiments may be unique so that all active users and running application programs are housed in the server(s) implementing the method of the present invention, and the steps(s) of the method(s) may be shortened or lengthened, overlap with other activities, be postponed, be delayed, and / or continue after a time lag has occurred.

[0201] For the sake of simplicity, embodiments of the methods of this disclosure are depicted and described as a series of actions or steps. However, the actions or steps of this disclosure may occur in various orders and / or simultaneously, and in conjunction with other actions or steps not presented and described herein. Furthermore, not all illustrated actions or steps are required to carry out the methods of the disclosed subject matter. Moreover, those skilled in the art will understand and recognize that the methods may be alternatively represented as a series of interrelated states via a state diagram or events or their equivalents.

[0202] As used herein, the singular forms "a," "an," and "the" include plural references unless otherwise explicitly indicated in the context. Thus, for example, a reference to "cable" includes a single cable and a bundle of two or more different cables. Terms such as "comprise," "comprising," "includes," "including," "have," and "having," as used herein and in the claims, are open-ended and intended to be non-restrictive, meaning "including but not limited to."

[0203] The above description includes many specific details, such as specific structures, dimensions, processes, and parameters, in order to provide a complete understanding of the present invention. Certain features, structures, materials, or properties may be combined in any suitable manner in one or more embodiments. The words “example,” “exemplary,” and “illustrative” are used herein to mean examples, cases, or descriptions. No aspect or design described herein as an “example” or equivalent should necessarily be construed as preferable or advantageous to any other aspect or design. Rather, the use of the word “example” or equivalent is intended to present the concept in a concrete manner.

[0204] As used in this application, “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or evident from the context, “X contains A or B” is intended to mean a natural inclusive substitution. That is, if X contains A, X contains B, or X contains both A and B, then in any of the above cases, “X contains A or B” is satisfied.

[0205] Throughout this specification, any reference to “an embodiment,” “certain embodimens,” or “one embodiment” means that the specific features, structures, or characteristics described in relation to that embodiment are included in at least one embodiment. Therefore, the phrases “an embodiment,” “certain embodimens,” or “one embodiment” appearing throughout this specification do not necessarily all refer to the same embodiment.

[0206] As used herein, the term “about” with respect to a measured quantity refers to the normal variation of that measured quantity that can be expected by a person skilled in the art when performing a measurement and paying a level of care commensurate with the purpose of the measurement and the precision of the measuring instrument. For example, in some exemplary embodiments, the term “about” may include a range of + / - 10% of the given number, such that “about 10” includes 9 to 11. In other exemplary embodiments, the term “about” may include a range of + / - X% of the given number, where X is considered the normal variation in the above measurement by a person skilled in the art.

[0207] Features described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features described in the context of a single embodiment for the sake of brevity may be implemented separately or in any preferred sub-combination. The applicant hereby notifies that new claims may be formed for such features and / or combinations of such features during the examination of this application or any further application derived therefrom. The features of the temporary physical storage medium described may be incorporated into the corresponding methods and / or systems, used in the corresponding methods and / or systems, and vice versa.

[0208] While the present invention has been described with reference to certain exemplary embodiments, it will be apparent to those skilled in the art that various modifications and changes can be made to these embodiments without departing from the broader scope of the invention. Accordingly, the specification and drawings are to be considered illustrative rather than limiting. It will also be apparent to those skilled in the art that the above embodiments are specific examples of a single broader invention, which may have a broader scope than any of the single descriptions taught. Many modifications can be made herein without departing from the scope of the invention as defined by the claims.

Claims

1. A non-temporary physical storage medium for storing program code executable by a hardware processor, wherein when the hardware processor executes the program code, it causes the hardware processor to execute a computer implementation process for determining the best-matching scenario name label for an utterance during automated scenario detection in a customer-agent conversation, and the program code The acquisition stage includes a biencoder neural network to receive multiple scenarios, multiple scenario name labels, and multiple sample phrase lists, wherein each scenario in the multiple scenarios is associated with one of the multiple scenario name labels and one of the multiple sample phrase lists, and the biencoder neural network includes a masked and permutation pre-trained (MPNet) based model for language understanding, a pooling operation, a cosine similarity operation, multiple encoder stacks, and a multilayer perceptron (MLP). The acquisition stage involves encoding each sample phrase in the list of multiple sample phrases into a phrase encoding and generating a list of multiple phrase encodings, The acquisition stage generates multiple scenario encodings, wherein each of the multiple scenario encodings is associated with one of the multiple scenarios, associated with one of the multiple scenario name labels, associated with one of the multiple scenario name labels, associated with one of the list of phrase encodings, and each of the multiple scenario encodings is determined by normalizing the centroid of the list of phrase encodings associated with one of the multiple scenarios. The list of the multiple scenario encodings, the multiple scenario name labels, and the multiple phrase encodings is stored in a database. The acquisition stage involves receiving the utterance, The acquisition stage involves encoding the conversational context vector of the utterance, The acquisition stage generates a plurality of similarity scores for the conversation context vector of the utterance, wherein each of the plurality of similarity scores is associated with a given scenario encoding among a plurality of scenario encodings stored in the database. The acquisition stage involves selecting a given scenario encoding from among the multiple scenarios that has the highest similarity score, thereby determining the best-matching scenario encoding from among the multiple scenario encodings. The acquisition stage generates a plurality of sequence pairs, wherein the first component of each sequence pair in the plurality of sequence pairs is the utterance, and the second component of each sequence pair in the plurality of sequence pairs is a given phrase encoding from a given list of phrase encodings associated with the best-match scenario encoding. The method involves generating a plurality of similarity probabilities by a reordering stage including a cross-encoder neural network, wherein each similarity probability in the plurality of similarity probabilities is associated with an ordered pair among the plurality of ordered pairs, and the cross-encoder neural network includes a large language model (LLM) based on a bidirectional encoder representation (BERT) language model from a linear layer configured to accept CLS (Classification) token embeddings, a sigmoid operator, a transformer configured to accept sentences, a classification head, a plurality of encoder stacks, and a multilayer perceptron (MLP). The reordering stage determines whether at least one of the multiple similarity probabilities exceeds a predetermined threshold. If at least one of the plurality of similarity probabilities exceeds the predetermined threshold, the reordering stage assigns the best-matching scenario name label from among the plurality of scenario name labels associated with the best-matching scenario encoding to the utterance. If none of the aforementioned similarity probabilities exceed the predetermined threshold, the reordering stage assigns the unintentional scenario name label to the utterance. A non-temporary physical storage medium containing code to perform the following actions.

2. The non-temporary physical storage medium according to claim 1, wherein the biencoder neural network further includes input encoding, output encoding state, layer normalization, and multi-head attention.

3. The non-temporary physical storage medium according to claim 1, wherein the cross-encoder neural network further includes input encoding, output encoding state, layer normalization, and multi-head attention.

4. The non-temporary physical storage medium according to claim 1, wherein encoding a sample phrase includes generating a vector of real numbers.

5. The non-temporary physical storage medium according to claim 1, wherein the plurality of similarity scores include a plurality of cosine similarity scores.

6. The aforementioned program code is: Triggering conversation tags based on the best-match scenario name label and multiple configuration options, wherein the conversation tags include text strings. A non-temporary physical storage medium according to claim 1, further comprising code that performs the following.

7. The non-temporary physical storage medium according to claim 6, wherein the plurality of setting options include speaker identity, and the program code that triggers the conversation tag is further based on the identity of the speaker of the utterance.

8. The non-temporary physical storage medium according to claim 7, wherein the program code for triggering the conversation tag is further based on whether a customer's sentence follows a sequence of agent's sentences.

9. The non-temporary physical storage medium according to claim 6, wherein the plurality of setting options include speaker behavior, and the program code that triggers the conversation tag is further based on whether the speaker of the utterance referred to a particular phrase.

10. The non-temporary physical storage medium according to claim 6, wherein the plurality of setting options include timing, and the program code that triggers the conversation tag is further based on whether the utterance occurred within a predetermined period after the start of the conversation.

11. A non-temporary physical storage medium for storing program code executable by a hardware processor, wherein when the hardware processor executes the program code, it causes the hardware processor to execute a computer implementation process for determining the best-matching scenario name label for an utterance during automated scenario detection in a customer-agent conversation, and the program code The acquisition stage includes receiving the utterance, wherein the biencoder neural network includes a masked and replaceable pre-trained (MPNet) based model for language understanding, a pooling operation, a cosine similarity operation, a multiple encoder stack, and a multilayer perceptron (MLP). The acquisition stage involves encoding the conversational context vector of the utterance, The acquisition stage generates a plurality of similarity scores for the conversation context vector of the utterance, wherein each of the plurality of similarity scores is associated with one scenario encoding from among a plurality of scenario encodings, associated with one scenario name label from among a plurality of scenario name labels, and associated with one list of phrase encodings from among a list of a plurality of phrase encodings. The acquisition stage involves selecting a given scenario encoding from among the multiple scenario encodings that has the highest similarity score, thereby determining the best-matching scenario encoding from among the multiple scenario encodings. The acquisition stage generates a plurality of sequence pairs, wherein the first component of each sequence pair in the plurality of sequence pairs is the utterance, and the second component of each sequence pair in the plurality of sequence pairs is a given phrase encoding from a given list of phrase encodings associated with the best-match scenario encoding. The method involves generating a plurality of similarity probabilities by a reordering stage including a cross-encoder neural network, wherein each similarity probability in the plurality of similarity probabilities is associated with an ordered pair among the plurality of ordered pairs, and the cross-encoder neural network includes a large language model (LLM) based on a bidirectional encoder representation (BERT) language model from a linear layer configured to accept CLS (Classification) token embeddings, a sigmoid operator, a transformer configured to accept sentences, a classification head, a plurality of encoder stacks, and a multilayer perceptron (MLP). The reordering stage determines whether at least one of the multiple similarity probabilities exceeds a predetermined threshold. If at least one of the plurality of similarity probabilities exceeds the predetermined threshold, the reordering stage assigns the best-matching scenario name label from among the plurality of scenario name labels associated with the best-matching scenario encoding to the utterance. If none of the aforementioned similarity probabilities exceed the predetermined threshold, the reordering stage assigns the unintentional scenario name label to the utterance. A non-temporary physical storage medium containing code to perform the following actions.

12. A computer implementation method for determining the best-matching scenario name label for an utterance during automated scenario detection in a customer-agent conversation, wherein the computer implementation method is: The acquisition stage includes receiving the utterance, wherein the biencoder neural network includes a masked and replaceable pre-trained (MPNet) based model for language understanding, a pooling operation, a cosine similarity operation, a multiple encoder stack, and a multilayer perceptron (MLP). The acquisition stage involves encoding the conversational context vector of the utterance, The acquisition stage generates a plurality of similarity scores for the conversation context vector of the utterance, wherein each of the plurality of similarity scores is associated with one scenario encoding from among a plurality of scenario encodings, associated with one scenario name label from among a plurality of scenario name labels, and associated with one list of phrase encodings from among a list of a plurality of phrase encodings. The acquisition stage involves selecting a given scenario encoding from among the multiple scenario encodings that has the highest similarity score, thereby determining the best-matching scenario encoding from among the multiple scenario encodings. The acquisition stage generates a plurality of sequence pairs, wherein the first component of each sequence pair in the plurality of sequence pairs is the utterance, and the second component of each sequence pair in the plurality of sequence pairs is a given phrase encoding from a given list of phrase encodings associated with the best-match scenario encoding. The method involves generating a plurality of similarity probabilities by a reordering stage including a cross-encoder neural network, wherein each similarity probability in the plurality of similarity probabilities is associated with an ordered pair among the plurality of ordered pairs, and the cross-encoder neural network includes a large language model (LLM) based on a bidirectional encoder representation (BERT) language model from a linear layer configured to accept CLS (Classification) token embeddings, a sigmoid operator, a transformer configured to accept sentences, a classification head, a plurality of encoder stacks, and a multilayer perceptron (MLP). The reordering stage determines whether at least one of the multiple similarity probabilities exceeds a predetermined threshold. If at least one of the plurality of similarity probabilities exceeds the predetermined threshold, the reordering stage assigns the best-matching scenario name label from among the plurality of scenario name labels associated with the best-matching scenario encoding to the utterance. If none of the aforementioned similarity probabilities exceed the predetermined threshold, the reordering stage assigns the unintentional scenario name label to the utterance. Computer implementation methods, including those mentioned above.

13. The computer implementation method according to claim 12, wherein the biencoder neural network further includes input encoding, output encoding state, layer normalization, and multi-head attention.

14. The computer implementation method according to claim 12, wherein the cross-encoder neural network further includes input encoding, output encoding state, layer normalization, and multi-head attention.

15. The computer implementation method according to claim 12, wherein the plurality of similarity scores include a plurality of cosine similarity scores.

16. Triggering conversation tags based on the best-match scenario name label and multiple configuration options, wherein the conversation tags include a text string. The computer implementation method according to claim 12, further comprising:

17. The computer implementation method according to claim 16, wherein the plurality of setting options include speaker identity, and triggering the conversation tag is further based on the identity of the speaker of the utterance and whether a customer sentence follows the sequence of agent sentences.

18. The computer implementation method according to claim 16, wherein the plurality of setting options include speaker behavior, and triggering the conversation tag is further based on whether the speaker of the utterance referred to a particular phrase.

19. The computer implementation method according to claim 16, wherein the plurality of setting options include timing, and triggering the conversation tag is further based on whether the utterance occurred within a predetermined period after the start of the conversation.

Images