Systems and techniques for processing long text for pre-trained language models

Abstract

In some aspects, a computing device can receive a set of utterances at a data processing system, train or infer using a named entity recognizer, and assign a label to each token piece from the set of utterances. The computing device can determine a length of each utterance in the set, split the utterance into multiple overlapping chunks of token pieces when the utterance length exceeds a predetermined threshold of token pieces, assign a label along with a confidence score to each token piece in the chunk, determine a final label and associated confidence score for each chunk of token pieces by combining two confidence scores, determine a final annotated label for the utterance based on the combination of at least two confidence scores, and store the final annotated label in a memory.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. patent application Ser. No. 17 / 750,240, entitled "SYSTEM AND TECHNIQUES FOR HANDLING LONG TEXT FOR PRE-TRAINED LANGUAGE MODELS," filed May 20, 2022, which claims priority to commonly owned U.S. provisional patent application Ser. No. 63 / 282,146, entitled "SYSTEM AND TECHNIQUES FOR HANDLING LONG TEXT FOR PRE-TRAINED LANGUAGE MODELS," filed November 22, 2021, and incorporated by reference in its entirety and for all purposes.

[0002] FIELD OF THEINVENTION The present disclosure relates generally to the Chabot system, and more particularly to techniques for processing long texts of pre-trained language models for the Chabot system in natural language processing. [Background technology]

[0003] background Many users around the world rely on instant messaging and chat platforms to get instant responses. Organizations often use these instant messaging and chat platforms to have live conversations with customers (or end users). However, it can be very costly for organizations to employ service representatives to engage in live communication with customers or end users. Chatbots or bots have started to be developed to simulate conversations with end users, especially over the internet. End users can communicate with the bots through messaging apps that they already have installed and use. Intelligent bots generally leverage artificial intelligence (AI) and can communicate more intelligently and contextually in live conversations, which can enable a more natural conversation between the bot and the end user, improving the conversation experience. Instead of end users learning a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots can understand the end user's intent based on the user's utterances in natural language and respond accordingly.

[0004] Pre-trained language models used in natural language processing can be designed to support a maximum text length. For example, various models can limit the maximum text length to 512 token pieces (e.g., 512 subwords). In previous solutions, Chabot entries that exceeded the maximum text length were truncated to less than the maximum text length. This solution can result in lost information from the truncated data, reducing the accuracy of the results. This can lead to poor Chabot performance and a poor customer experience. Summary of the Invention

[0005] overview The technology disclosed herein generally relates to catboats. More specifically, but not by way of limitation, the technology disclosed herein relates to processing long text for pre-trained language models. In various aspects, the pre-trained language models typically support a predetermined maximum number of token pieces. The long text can be divided into overlapping chunks using a predefined chunk size and number of overlapping token pieces. The chunks can be tagged separately by named entity recognition (NER). Two scores of annotated labels for each overlapping token piece (one from the first chunk and one from the second chunk) can be combined to determine the final label for each token piece in the long text. This can reduce training and inference time while maintaining high performance of the chatbot.

[0006] In some aspects, a method includes, at a data processing system, receiving a set of utterances, training or interfacing with a named entity recognizer to assign a label to each token piece from one or more utterances, determining a length of the set of utterances, and if the length of the set of utterances exceeds a predetermined threshold of token pieces (i.e., chunk size), splitting the set of utterances into multiple overlapping chunks of token pieces, assigning a label along with a confidence score to each token piece in the chunk, determining a final label and associated confidence score for the overlapping token pieces by combining two confidence scores, the confidence score from the first chunk and the confidence score from the second chunk, determining a final annotated label for the original text input, and storing the label in a memory.

[0007] In some aspects, each chunk and its corresponding sequence of labels is treated as a separate example during training and inference.

[0008] In some aspects, partitioning the set of utterances of size N is partitioning the set of utterances of size N into (NL) / (KL) overlapping chunks, where K is the chunk size and L is the overlap size (K>L).

[0009] In some aspects, the determination of the overall score and label of a token piece is based on the confidence score from the first selected chunk of the multiple chunks.

[0010] In some aspects, the determination of the overall score for a token piece is based on the location of the token piece within the overlapping text: a first confidence score from a first chunk if it is in the first half of the overlapping text, and a second confidence score from a second chunk if it is in the second half of the overlapping text.

[0011] In some aspects, the overall score determination is based on the maximum confidence scores of multiple chunks.

[0012] In some aspects, the predetermined number of token pieces (i.e., chunk size) is 512 tokens and the predetermined number of duplicate tokens is 128. In some aspects, the predetermined number of token pieces is 32 and the predetermined number of duplicate tokens is 8.

[0013] In various aspects, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0014] In various aspects, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0015] The techniques described above and below can be implemented in many ways and in many contexts. As described in more detail below, some example implementations and contexts are provided with reference to the following figures. However, the following implementations and contexts are only a few of the many possible implementations and contexts. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a simplified block diagram of an environment incorporating an exemplary embodiment of a Chabot system, according to an aspect. [Diagram 2] FIG. 2 is a simplified block diagram of a computing system implementing a master bot according to an aspect. [Diagram 3] FIG. 1 is a simplified block diagram of a computing system implementing a skillbot in accordance with an aspect. [Figure 4] FIG. 1 illustrates a system for processing long texts for a pre-trained language model. [Diagram 5] FIG. 1 illustrates the conversion of an example utterance into multiple chunks. [Figure 6] FIG. 1 illustrates a technique for integrating predictions. [Figure 7] FIG. 1 is a flow diagram of a technique for processing long texts for pre-trained language models. [Figure 8] FIG. 1 illustrates a simplified diagram of a distributed system for implementing certain aspects. [Figure 9] A simplified block diagram of one or more components of a system environment in which services provided by one or more components of an embodiment system may be provided as a cloud service in accordance with an aspect. [Figure 10] FIG. 1 illustrates an exemplary computer system that can be used to implement certain aspects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain aspects. However, it will be apparent that various aspects may be practiced without these specific details. The figures and descriptions are not limiting. As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0018] Bots and Analytics A bot (also called a skill, chatbot, chatterbot, or talkbot) is a computer program that can conduct a conversation with an end user. A bot can respond to natural language messages (e.g., questions or comments) typically through a messaging application using natural language messages. An enterprise may use one or more bot systems to communicate with end users through messaging applications. The messaging application, which may be called a channel, may be an end user's preferred messaging application that the end user already has installed and is familiar with. Thus, an end user does not need to download and install a new application to chat with a bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (e.g., Facebook® Messenger, Facebook® WhatsApp®, WeChat®, Line, Kik®, Telegram®, Talk, Skype®, Slack®, or short message service (SMS), virtual private assistants (e.g., Amazon® Dot, Echo, or Show, Google Home®, Apple HomePod®, mobile app or web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat capabilities, or voice-based input (e.g., devices or apps with an interface that uses Siri®, Cortana®, Google® Voice, or other voice input for interaction).

[0019] In some examples, the bot system may be associated with a Uniform Resource Identifier (URI). The URI may use a string to identify the bot system. The URI may be used as a webhook for one or more messaging application systems. The URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The bot system may be designed to receive a message (e.g., a HyperText Transfer Protocol (HTTP) post-call message) from the messaging application system. An HTTP post-call message may be sent from the messaging application system to the URI. In some aspects, the message may differ from an HTTP post-call message. For example, the bot system may receive a message via SMS. Although the description herein may refer to a communication received by the bot system as a message, it should be understood that the message may be an HTTP post-call message, an SMS message, or any other type of communication between the two systems.

[0020] End users can interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), just like human-to-human interactions. In some cases, the interaction may involve the end user saying "hello" to the bot, which responds with "hello" and asks the end user how it can help. In some cases, the interaction may be a transactional interaction with a banking bot, e.g., transferring money from one account to another, an informational interaction with a human resources bot, e.g., checking vacation balances, or an interaction with a retail bot, e.g., discussing the return of a purchased item or asking for technical support.

[0021] In some aspects, the bot system can intelligently handle end user interactions without interaction with an administrator or developer of the bot system. For example, an end user may send one or more messages to the bot system to achieve a desired goal. The messages may include certain content, such as text, emojis, voice, images, videos, or other message delivery methods. In some aspects, the bot system can convert the content into a standardized format (e.g., a Representational State Transfer (REST) ​​call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system can also prompt the end user for additional input parameters or request other additional information. In some aspects, the bot system can also initiate communication with the end user rather than passively responding to the end user's utterances. Various techniques are described herein for identifying explicit invocations of the bot system and determining inputs to the invoked bot system. In some aspects, the explicit invocation analysis is performed by the master bot based on detection of an invocation name in the utterance. In response to detection of an invocation name, the utterance can be refined for input to a skill bot associated with the invocation name.

[0022] A conversation with a bot can follow a particular conversation flow that includes multiple states. The flow can define what happens next based on the input. In some aspects, a bot system can be implemented using a state machine that includes user-defined states (e.g., end user intent) and actions to perform in the states or from state to state. A conversation may take different paths based on the end user's input, which can affect the flow decisions made by the bot. For example, at each state, based on the end user's input or utterance, the bot can determine the end user's intent and determine the appropriate action to perform next. As used herein and in the context of utterances, the term "intent" refers to the intent of the user who provided the utterance. For example, a user may intend to converse with a bot to order a pizza, and thus the user's intent may be expressed through the utterance "order a pizza". The user's intent can be directed to a particular task that the user wants the bot to perform on their behalf. Thus, an utterance can be an expression of a question, command, request, etc. that reflects the user's intent. An intent can include a goal that the end user wants to achieve.

[0023] In the context of Chabot's configuration, the term "intent" is used herein to refer to configuration information for mapping a user's utterance to a specific task / action or category of task / action that Chabot can perform. To distinguish between an utterance intent (i.e., a user's intent) and a Chabot's intent, the latter may be referred to herein as a "bot's intent." A bot's intent may include a set of one or more utterances associated with the intent. For example, an intent to order pizza may include various permutations of utterances expressing a desire to order pizza. These related utterances may be used to train Chabot's intent classifier, which may then determine whether an input utterance from a user matches the intent of ordering pizza. A bot's intent may be associated with one or more dialog flows for initiating a conversation with a user at a state. For example, the first message of an intent to order pizza may be the question, "What kind of pizza do you like?" In addition to the related utterances, a bot's intent may further include named entities associated with the intent. For example, an intent to order a pizza may include variables or parameters that are used to perform the task of ordering a pizza, e.g., topping 1, topping 2, type of pizza, size of pizza, amount of pizza, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0024] FIG. 1 is a simplified block diagram of an environment 100 incorporating a Chabot system according to an embodiment. The environment 100 comprises a digital assistant builder platform (DABP) 102 that allows users of the DABP 102 to create and deploy digital assistants or Chabot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or Chabot systems. For example, as shown in FIG. 1, a user 104 representing a particular business can use the DABP 102 to create and deploy a digital assistant 106 for users of the particular business. For example, a bank can use the DABP 102 to create one or more digital assistants for use by customers of the bank. The same DABP 102 platform can be used by multiple businesses to create digital assistants. As another example, a restaurant (e.g., a pizza place) owner can use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0025] For purposes of this disclosure, a "digital assistant" is an entity that assists a user of the digital assistant in accomplishing various tasks through natural language conversation. A digital assistant can be implemented using only software (e.g., a digital assistant is a digital entity implemented using programs, codes, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or implemented in a variety of physical systems or devices, such as a computer, a mobile phone, a watch, an appliance, a vehicle, etc. A digital assistant is sometimes referred to as a Chabot system. Thus, for purposes of this disclosure, the terms digital assistant and Chabot system are interchangeable.

[0026] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform a variety of tasks via natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and obtain responses 112 from the digital assistant 106. A conversation can include one or more of the inputs 110 and responses 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the tasks requested by the user and respond to the user with an appropriate response.

[0027] User input is generally in a natural language format and is referred to as an utterance. User utterance 110 can be in text format, such as when a user inputs a sentence, a question, a fragment of text, or a single word and provides it as input to the digital assistant 106. In some aspects, user utterance 110 can be in voice input or audio format, such as when a user says or speaks something that is provided as input to the digital assistant 106. The utterance is typically in a language that the user 108 speaks. For example, the utterance can be in English or another language. If the utterance is in audio format, the voice input is converted into a text format utterance in that particular language, and the text utterance is processed by the digital assistant 106. Various voice-to-text processing techniques can be used to convert the voice or voice input into a text utterance, which is then processed by the digital assistant 106. In some aspects, the voice-to-text conversion may be performed by the digital assistant 106 itself.

[0028] The utterance may be a text or audio utterance and may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing on the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, including identifying one or more intents and one or more entities corresponding to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or behaviors depending on the understood meaning or intent. For purposes of this disclosure, it is assumed that the utterance is a text utterance provided directly by a user 108 of the digital assistant 106 or is the result of converting an input audio utterance into text format. However, this is not intended to be limiting or restrictive in any way.

[0029] For example, a user 108 input may request a pizza order by providing an utterance such as "I want to order a pizza." Upon receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and perform an appropriate action. The appropriate action may include, for example, responding to the user with a question requesting user input about the type of pizza the user wants to order, the size of the pizza, the pizza toppings, etc. The responses provided by the digital assistant 106 may also be in natural language form, typically in the same language as the input utterance. As part of generating these responses, the digital assistant 106 may perform natural language generation (NLG). In the case of a user ordering a pizza, through a conversation between the user and the digital assistant 106, the digital assistant may guide the user to provide all the information required to order the pizza, and then have the pizza ordered at the end of the conversation. The digital assistant 106 may end the conversation by outputting information to the user indicating that a pizza is ordered.

[0030] At a conceptual level, the digital assistant 106 performs various processing in response to utterances received from a user. In some aspects, this processing includes a series or pipeline of processing steps, such as understanding the meaning of the input utterance (also called natural language understanding (NLU)), determining an action to be performed in response to the utterance, appropriately triggering the execution of the action therein, generating a response to be output to the user in response to the user's utterance, outputting the response to the user, etc. NLU processing can include analyzing the received input utterance to understand the structure and meaning of the utterance, refining and reshaping the utterance to develop a more understandable form (e.g., logical form) or structure of the utterance. Generating the response can include the use of NLG technology.

[0031] NLU processing performed by a digital assistant such as digital assistant 106 may include various NLP-related processing such as sentence parsing (e.g., tokenization, lemmatization, identifying part-of-speech tags for a sentence, identifying named entities within a sentence, generating a dependency tree representing the sentence structure, splitting the sentence into clauses, analyzing the individual clauses, resolving anaphora, performing chunking, etc.). In some aspects, the NLU processing or parts thereof are performed by the digital assistant 106 itself. In some other aspects, the digital assistant 106 may use other resources to perform parts of the NLU processing. For example, the syntax and structure of an input spoken sentence may be identified by processing the sentence using a parser, a part-of-speech tagger, and / or a named entity recognizer. In one implementation, for English, a parser, a part-of-speech tagger, and a named entity recognizer such as those provided by the Stanford Natural Language Processing (NLP) Group are used to analyze the structure and syntax of the sentence. These are provided as part of the Stanford CoreNLP toolkit.

[0032] Although various examples provided in this disclosure show speech in English, this is meant as an example only. In an aspect, the digital assistant 106 can also process speech in languages ​​other than English. The digital assistant 106 can provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for different languages. These subsystems can be implemented as pluggable devices that can be invoked using service calls from the NLU core server. This makes NLU processing flexible and extensible for each language, including allowing different processing orders. Language packs can be provided for individual languages, and the language packs can register a list of subsystems that can be provided by the NLU core server.

[0033] A digital assistant, such as the digital assistant 106 shown in FIG. 1, may be available or accessible to its user 108 through a variety of different channels, including, but not limited to, through an application, through a social media platform, through various messaging services or applications, through other applications or channels, etc. A single digital assistant may be configured with multiple channels to run on different services or be accessed simultaneously.

[0034] A digital assistant or Chabot system generally includes or is associated with one or more skills. In an aspect, these skills are individual catboats (called skillbots) that are configured to interact with a user and perform a specific type of task, such as tracking inventory, submitting a time card, creating an expense report, ordering food, checking a bank account, making a reservation, purchasing a widget, etc. For example, in the embodiment shown in FIG. 1, the digital assistant 106 or Chabot system includes a first skillbot 116-1, a second skillbot 116-2, etc. For purposes of this disclosure, the terms "skill" and "skills" are used synonymously with the terms "skill bot" and "skill bots," respectively.

[0035] Each skill associated with a digital assistant helps a user of the digital assistant complete a task through a conversation with the user, which may include a combination of text or voice input provided by the user and responses provided by the skill bot. These responses may take the form of text or voice messages to the user and / or use simple user interface elements (e.g., selection lists) presented to the user for the user to select from.

[0036] There are various ways in which skills or skillbots can be associated with or added to a digital assistant. In some cases, skillbots can be developed by companies and then added to a digital assistant using DABP 102. In other examples, skillbots can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet other examples, DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills directed to a wide range of tasks. Skills offered through the skill store can also expose various cloud services. To add a skill to a digital assistant being created using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and indicate that the selected skill is to be added to the digital assistant being created using DABP 102. Skills from the skill store can be added to a digital assistant as is or in a modified form (e.g., a user of DABP102 can select and clone a particular skill bot provided by the skill store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP102).

[0037] A variety of different architectures can be used to implement a digital assistant or chabot system. For example, in one aspect, a digital assistant created and deployed using DABP 102 may be implemented using a masterbot / child (or sub)bot paradigm or architecture. According to this paradigm, a digital assistant is implemented as a masterbot that interacts with one or more child bots, which are skillbots. For example, in the embodiment shown in FIG. 1, the digital assistant 106 includes a masterbot 114 and a first skillbot 116-1, a second skillbot 116-2, etc., which are child bots of the masterbot 114. In one aspect, the digital assistant 106 itself is considered to function as a masterbot.

[0038] A digital assistant implemented according to the master-child bot architecture allows a user of the digital assistant to interact with multiple skills through a unified user interface, i.e., a master bot. When a user operates the digital assistant, the user input is received by the master bot. The master bot then performs processing to determine the meaning of the utterance of the user input. The master bot then determines whether the task requested by the user in the utterance can be handled by the master bot itself. If not, the master bot selects an appropriate skill bot to handle the user's request and routes the conversation to the selected skill bot. This allows a user to converse with the digital assistant through a common single interface and also provides the ability to use multiple skill bots configured to perform specific tasks. For example, in the case of a digital assistant developed for an enterprise, the master bot of the digital assistant can be coordinated with skill bots with specific functions, such as a CRM bot to perform functions related to customer relationship management (CRM), an ERP bot to perform functions related to enterprise resource planning (ERP), an HCM bot to perform functions related to human capital management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common master bot interface, and multiple skill bots are provided behind the scenes to handle the user's requests.

[0039] In an aspect, in a masterbot / childbot infrastructure, the masterbot is configured to know a list of available skillbots. The masterbot has access to metadata identifying the various skillbots available and, for each skillbot, the capabilities of the skillbot, including tasks that can be performed by the skillbot. Upon receiving a user request in the form of an utterance, the masterbot is configured to identify or predict, among a plurality of available skillbots, a particular skillbot that can best serve or process the user request. The masterbot then routes the utterance (or a portion of the utterance) to that particular skillbot for further processing. Thus, control flows from the masterbot to the skillbot. The masterbot can support multiple input and output channels. In an aspect, the routing can be performed utilizing processing performed by one or more available skillbots. For example, as described below, a skillbot can be trained to infer the intent of an utterance and determine whether the inferred intent matches an intent configured in the skillbot. Thus, the routing performed by the masterbot can include communicating to the masterbot an indication of whether the skillbot is configured with an intent suitable for processing the utterance.

[0040] 1 illustrates a digital assistant 106 including a masterbot 114, a first skillbot 116-1, a second skillbot 116-2, and a third skillbot 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored in a computer-readable medium and executable by one or more processors), solely in hardware, or in an implementation using a combination of software and hardware.

[0041] DABP 102 provides infrastructure and various services and features that enable a user of DABP 102 to create a digital assistant including one or more skill bots that are associated with the digital assistant. In some cases, a skill bot can be created by duplicating an existing skill bot, for example, by duplicating a skill bot provided by a skill store. As previously indicated, DABP 102 provides a skill store or skill catalog that provides multiple skill bots for performing various tasks. A user of DABP 102 can clone a skill bot from the skill store. If necessary, modifications and customizations can be made to the cloned skill bot. In other examples, a user of DABP 102 creates a skill bot from scratch using tools and services provided by DABP 102. As previously indicated, a skill store or skill catalog provided by DABP 102 can provide multiple skill bots for performing various tasks.

[0042] In one aspect, at a high level, creating or customizing a skillbot includes the following steps:

[0043] (1) Configure the settings for a new skill bot (2) Configure one or more intents for the skill bot (3) the composition of one or more entities for one or more intents (4) Skill Bot Training (5) Creating a dialogue flow for the skill bot (6) Add custom components to your skill bot as needed (7) Testing and Deploying Skill Bots Each of the above steps is briefly described below.

[0044] (1) Configuring Settings for a New Skill Bot - Various settings can be configured for a skill bot. For example, a skill bot designer can specify one or more invocation names for the skill bot they create. These invocation names can then be used by users of the digital assistant to explicitly invoke the skill bot. For example, a user can enter an invocation name in a user utterance to explicitly invoke the corresponding skill bot.

[0045] (2) Configuring one or more intents and associated example utterances for a skill bot - A skill bot designer specifies one or more intents (also called bot intents) for the skill bot to be created. The skill bot is then trained based on these specified intents. These intents represent categories or classes for which the skill bot is trained to infer input utterances. Upon receiving an utterance, the skill bot being trained infers the intent of the utterance. The inferred intent is selected from a set of predefined intents used to train the skill bot. The skill bot then performs an appropriate action in response to the utterance based on the inferred intent for the utterance. In some cases, the intents of the skill bot represent tasks that the skill bot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skill bot being trained for banking, the intents specified for the skill bot may include "CheckBalance", "TransferMoney", "DepositCheck", etc.

[0046] For each intent defined for a skillbot, the skillbot designer can also provide one or more example utterances that represent and explain that intent. These example utterances are intended to represent utterances that a user can input to the skillbot for that intent. For example, for the CheckBalance intent, example utterances might include "What's the balance in my savings account?", "How much is in my checking account?", "How much money is in my account?", etc. Thus, various permutations of typical user utterances can be specified as example utterances for an intent.

[0047] The intents and associated example utterances are used as training data to train the skill bot. A variety of different training techniques may be used. This training results in a predictive model configured to receive an utterance as input and output an intent that is inferred for the utterance by the predictive model. In some cases, the input utterance is provided to an intent analysis engine, which is configured to predict or infer an intent for the input utterance using the trained model. The skill bot can perform one or more actions based on the inferred intent.

[0048] (3) Configuring an entity for one or more intents of a skill bot - In some cases, additional context may be required to enable the skill bot to respond appropriately to a user utterance. For example, there may be situations where user input utterances resolve to the same intent in a skill bot. For instance, in the above example, the utterances "What is the balance in my savings account?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests that ask for different things. To disambiguate such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​"checking" and "saving" allows the skill bot to parse the user request and respond appropriately. In the above example, the utterances resolve to the same intent, but the values ​​associated with the AccountType entity are different for the two utterances. This may enable the skill bot to take different actions for the two utterances even though they are resolving to the same intent. One or more entities can be specified for an intent configured for a skill bot. Thus, entities are used to add context to the intent itself. Entities help to more completely describe the intent, enabling the skill bot to complete the user request.

[0049] In an aspect, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by the skill bot designer. Built-in entities are generic entities that can be used by various bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, periodic period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by the skill bot designer that allows various banking transactions by checking user input against keywords such as check, savings, credit card, etc.

[0050] (4) Training the Skillbot - The skillbot is configured to receive user input in the form of utterances, parse or process the received input, and identify or select an intent associated with the received user input. As indicated above, the skillbot needs to be trained for this. In an embodiment, the skillbot is trained based on intents configured for the skillbot and example utterances associated with those intents (collectively, training data), so that the skillbot can resolve user input utterances to one of its configured intents. In an embodiment, the skillbot uses a predictive model that is trained using the training data to enable the skillbot to identify what the user is saying (or, in some cases, trying to say). DABP 102 provides various training techniques that the skillbot designer can use to train the skillbot, such as various machine learning based training techniques, rule-based training techniques, and / or combinations thereof. In an embodiment, a portion of the training data (e.g., 80%) is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once training is complete, the trained model (also referred to as the trained skillbot) can be used to process and respond to user utterances. In some cases, a user utterance may be a question that requires only one answer and no further conversation. To address such situations, you can define a Q&A (Question and Answer) intent for your skill bot. This allows your skill bot to output a response to a user request without updating the dialog definition. A Q&A intent is created in a similar way to a regular intent. The dialog flow for a Q&A intent may differ from that of a regular intent.

[0051] (5) Creating a Dialog Flow for a Skill Bot -- The dialog flow specified for a skill bot describes how the skill bot reacts as different intents of the skill bot are resolved depending on the user input received. The dialog flow defines the behavior or actions that the skill bot performs, such as how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. The dialog flow is like a flowchart that the skill bot follows. Skill bot designers specify the dialog flow using a language such as Markdown language. In one aspect, a version of YAML called OBotML can be used to specify the dialog flow for a skill bot. The dialog flow definition for a skill bot serves as a model of the conversation itself, allowing the skill bot designer to choreograph the interaction between the skill bot and the user that the skill bot serves.

[0052] In one embodiment, a skill bot's dialog flow definition includes three sections:

[0053] (a) Context Section (b) Default transition section (c) Status section Context Section - Skill bot designers can define variables that will be used in the conversation flow in the context section. Other variables that can be named in the context section include but are not limited to variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot to know and preserve user preferences, etc.

[0054] Default Transitions Section - Transitions for a skill bot can be defined in the dialog flow states section or in the default transitions section. Transitions defined in the default transitions section act as fallbacks and are triggered when there is no corresponding transition defined within a state or when the conditions required to trigger a state transition are not met. The default transitions section allows you to define routing that enables your skill bot to gracefully handle unexpected user actions.

[0055] State Section - A dialog flow and its associated behavior are defined as a set of temporary states that govern the logic within the dialog flow. Each state node in a dialog flow definition names a component that provides the functionality required at that point in the dialog. Thus, states are built around components. States contain component-specific properties and define transitions to other states that are triggered after the component is executed.

[0056] Special case scenarios can be handled using the states section. For example, you may want to offer a user the option to temporarily leave a first skill they are using to do something in a second skill within the digital assistant. For example, if a user is engaged in a conversation with a shopping skill (e.g., the user is making some choices about a purchase), the user may want to jump to a banking skill (e.g., the user may want to ensure they have enough money for the purchase), and then return to the shopping skill to complete the user's order. To address this, you can configure an action in the first skill to initiate an interaction with a second, different skill within the same digital assistant, and then return to the original flow.

[0057] (6) Adding Custom Components to a Skillbot - As described above, a state specified in a skillbot's dialog flow names a component that corresponds to that state and provides the required functionality. A component enables a skillbot to perform a function. In an aspect, DABP 102 provides a set of pre-configured components to perform a wide range of functions. A skillbot designer can select one or more of these pre-configured components and associate them with a state in the skillbot's dialog flow. A skillbot designer can also create custom or new components using tools provided by DABP 102 and associate the custom components with one or more states in the skillbot's dialog flow.

[0058] (7) Testing and Deploying Skillbots - DABP102 provides several features that enable skillbot designers to test the skillbots they are developing, after which they can be deployed and included in a digital assistant.

[0059] While the above discussion describes how to create a skill bot, similar techniques can also be used to create a digital assistant (or master bot). At the master bot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the master bot) can handle without invoking a skill bot associated with the digital assistant. Examples of system intents defined for a master bot are: (1) Exit: applies when a user signals that they want to end the digital assistant's current conversation or context. (2) Help: applies when a user asks for help or direction. (3) UnresolvedIntent: applies to user input that does not match well with the exit and help intents. The digital assistant also stores information about one or more skill bots associated with the digital assistant. This information allows the master bot to select a specific skill bot to process an utterance.

[0060] At the MasterBot or Digital Assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and associated conversation. The digital assistant determines this using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance should be routed to a specific skill for processing, should be handled by the digital assistant or MasterBot itself according to a built-in system intent, or should be treated as a separate state in the current conversation flow.

[0061] In an aspect, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skill bot corresponding to the invocation name. In such a scenario, the digital assistant may route the user input to the explicitly invoked skill bot for further processing. In the absence of a specific or explicit invocation, in an aspect, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and the skill bot associated with the digital assistant. The calculated score for the skill bot or system intent indicates the likelihood that the user input represents a task that the skill bot is configured to perform, or the likelihood that the user input represents a system intent. A system intent or skill bot with an associated calculated confidence score exceeding a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing the user input utterance. In an aspect, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model of each skill) and a confidence score is determined for each intent. In general, intents with confidence scores above a threshold (e.g., 70%) are treated as candidate intents. If a particular skill bot is selected, the user's utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0062] FIG. 2 is a simplified block diagram of a Masterbot (MB) system 200 according to an embodiment. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multi-intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skillbot invoker 240, and a data store 250. The MB system 200 shown in FIG. 2 is only one example of an arrangement of components in a Masterbot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more subsystems, or may have a different configuration or arrangement of the subsystems.

[0063] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance via a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in a variety of ways, such as audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 can convert the speech to text using a speech-to-text converter (not shown) that inserts punctuation marks (e.g., commas, semicolons, periods, etc.) into the resulting text.

[0064] The language detector 212 detects the language of the utterance 202 based on the text of the utterance 202. The way in which the utterance 202 is processed is language dependent, since each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of the utterance.

[0065] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language parser 214 may also tokenize the linguistic units of the utterance 202 (e.g., to convert each word into a separate token) and lemmatize the words. Lemmats are the main form of a set of words represented in a dictionary (e.g., "run" is a lemma for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 may perform include chunking of compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". The language parser 214 may also identify relationships between words in the utterance 202. For example, in some aspects the language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., particular nouns) are direct objects, which parts of the utterance are prepositions, etc. The results of the processing performed by the language parser 214 form the extracted information 205 and are provided as inputs to the MIS 220 along with the utterance 202 itself. The preprocessing system 210 may include a named entity recognizer 216 that can be used to recognize an utterance 202 or portions thereof.

[0066] As indicated above, the utterance 202 may include multiple sentences. For purposes of detecting multiple intents and explicit invocations, the utterance 202 may be treated as a single unit even if it contains multiple sentences. However, in an embodiment, preprocessing may be performed, for example by the preprocessing subsystem 210, to identify a single sentence among multiple sentences for multiple intent and explicit invocation analysis. In general, the results generated by the MIS 220 and the EIS 230 are substantially the same regardless of whether the utterance 202 is processed at the level of individual sentences or as a single unit containing multiple sentences.

[0067] The MIS 220 determines whether the utterance 202 expresses multiple intents. Although the MIS 220 can detect the presence of multiple intents in the utterance 202, the process performed by the MIS 220 does not include determining whether the intent of the utterance 202 matches any intent configured for the bot. Instead, the process for determining whether the intent of the utterance 202 matches the intent of the bot may be performed by the intent classifier 242 of the MB system 200 or by the intent classifier of the skill bot (e.g., as shown in the embodiment of FIG. 3). The process performed by the MIS 220 assumes that there is a bot (e.g., a specific skill bot or the master bot itself) that can process the utterance 202. Thus, the process performed by the MIS 220 does not require knowledge of the bots in the Chabot system (e.g., the ID of the skill bot registered with the master bot) or knowledge of the intents configured for a specific bot.

[0068] To determine that an utterance 202 includes multiple intents, the MIS 220 applies one or more rules from a rule set 252 in the data store 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and may include a sentence pattern that indicates the presence of multiple intents. For example, the sentence pattern may include a coordinating conjunction that joins two parts of a sentence (e.g., a conjunction), both parts corresponding to separate intents. If the utterance 202 matches the sentence pattern, it can be inferred that the utterance 202 represents multiple intents. Note that an utterance that includes multiple intents does not necessarily have different intents (e.g., intents directed to different bots or intents directed to different intents within the same bot). Instead, the utterance may include separate instances of the same intent. For example, "order pizza using payment account X, then order pizza using payment account Y."

[0069] As part of determining that the utterance 202 represents multiple intents, the MIS 220 also determines which portions of the utterance 202 are associated with each intent. For each intent expressed in the utterance that includes multiple intents, the MIS 220 constructs a new utterance for separate processing in place of the original utterance, e.g., utterance “B” 206 and utterance “C” 208, as shown in FIG. 2. Thus, the original utterance 202 may be split into two or more separate utterances that are processed one at a time. The MIS 220 determines which of the two or more utterances should be processed first using the extracted information 205 and / or from an analysis of the utterance 202 itself. For example, the MIS 220 may determine that the utterance 202 includes an indicator word that indicates that a particular intent should be processed first. The newly formed utterance corresponding to this particular intent (e.g., one of the utterances 206 or utterance 208) will be sent first for further processing by the EIS 230. After the conversation caused by the first utterance has ended (or has been temporarily interrupted), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) can be sent to EIS 230 for processing.

[0070] The EIS 230 determines whether the received utterance (e.g., utterance 206 or utterance 208) includes a call name of the skillbot. In an embodiment, each skillbot in the Chabot system is assigned a unique call name that distinguishes the skillbot from other skillbots in the Chabot system. A list of call names can be maintained as part of the skillbot information 254 in the data store 250. If the utterance contains words that match the call name, the utterance is considered to be an explicit call. If the bot is not explicitly called, the utterance received by the EIS 230 is considered an implicit calling utterance 234 and is input to the masterbot's intent classifier (e.g., intent classifier 242) to determine which bot to use to process the utterance. In some cases, the intent classifier 242 will determine that the masterbot should process an implicit calling utterance. In other examples, the intent classifier 242 will determine which skillbot to route the utterance to for processing.

[0071] The explicit call functionality provided by the EIS 230 has several advantages. It can reduce the amount of processing that the masterbot needs to perform. For example, when there is an explicit call, the masterbot may not need to perform an intent classification analysis (e.g., using the intent classifier 242) or may need to perform a reduced intent classification analysis to select a skillbot. Thus, the explicit call analysis may enable the selection of a particular skillbot without relying on an intent classification analysis.

[0072] There may also be situations where functionality overlaps between multiple skillbots. This can occur, for example, when the intents handled by two skillbots overlap or are very close to each other. In such situations, it may be difficult for the masterbot to identify which of multiple skillbots to select based on intent classification analysis alone. In such scenarios, an explicit invocation would make clear the specific skillbot to be used.

[0073] In addition to determining that the utterance is an explicit call, EIS 230 is responsible for determining whether any portion of the utterance should be used as input to the explicitly invoked skillbot. In particular, EIS 230 may determine whether any portion of the utterance is not relevant to the call. EIS 230 may perform this determination through analysis of the utterance and / or analysis of extracted information 205. Instead of sending the entire utterance received by EIS 230, EIS 230 may send the portion of the utterance that is not associated with the call to the invoked skillbot. In some cases, the input to the invoked skillbot is formed by simply removing the portion of the utterance that is associated with the call. For example, "I would like to order a pizza using PizzaBot" can be shortened to "I would like to order a pizza" because "using PizzaBot" is relevant to the call of PizzaBot but not the processing performed by PizzaBot. In some cases, EIS 230 may reformat the portion sent to the invoked bot to form, for example, a complete sentence. Thus, EIS 230 not only determines that there is an explicit call, but also determines what to send to the skillbot if there is an explicit call. In some cases, there may be no text to input to the bot being invoked. For example, if the utterance was "pizzabot," the EIS 230 may determine that the pizzabot is being invoked, but there is no text to be processed by the pizzabot. In such a scenario, the EIS 230 may indicate to the skillbot invoker 240 that there is nothing to send.

[0074] The skillbot invoker 240 invokes a skillbot in a variety of ways. For example, the skillbot invoker 240 can invoke the bot in response to receiving an indication 235 that a particular skillbot is selected as a result of an explicit invocation. The indication 235 can be sent by the EIS 230 along with the input of the explicitly invoked skillbot. In this scenario, the skillbot invoker 240 hands over control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input by treating the input from the EIS 230 as a standalone utterance. For example, the response can be to perform a particular action or to start a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0075] Another way that the skillbot invoker 240 can invoke a skillbot is through an implicit invocation using the intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform. The intent classifier 242 is trained with different classes, one class for each skillbot. For example, each time a new skillbot is registered with the masterbot, the intent classifier 242 can be trained using a list of example utterances associated with the new skillbot to determine the likelihood that a particular utterance represents a task that the new skillbot can perform. The parameters (e.g., a set of values ​​for the parameters of the machine learning model) generated as a result of this training can be stored as part of the skillbot information 254.

[0076] In an embodiment, the intent classifier 242 is implemented using a machine learning model, as described in more detail herein. Training the machine learning model may include inputting at least a subset of utterances from example utterances associated with various skill bots, and generating, as an output of the machine learning model, an inference about which bot is the correct bot to process a particular training utterance. For each training utterance, an indication of the correct bot to use for the training utterance may be provided as ground truth information. The operation of the machine learning model may then be adapted (e.g., through backpropagation) to minimize the difference between the generated inference and the ground truth information.

[0077] In an embodiment, the intent classifier 242 determines a confidence score for each skill bot registered with the master bot, indicating the likelihood that the skill bot can process the utterance (e.g., the implicit invocation utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each configured system-level intent (e.g., help, quit). If a particular confidence score meets one or more conditions, the skill bot invoker 240 invokes the bot associated with the particular confidence score. For example, a confidence score threshold may need to be met. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skill bot. In some embodiments, in addition to meeting the confidence score threshold, the confidence score must exceed the next highest confidence score by a certain win rate. Imposing such a condition allows routing to a particular skill bot when the confidence scores of multiple skill bots each exceed the confidence score threshold.

[0078] After identifying the bot based on the evaluation of the confidence score, the skillbot invoker 240 hands over the processing to the identified bot. In case of system intent, the identified bot becomes the master bot. Otherwise, the identified bot is the skillbot. Furthermore, the skillbot invoker 240 decides what to provide as input 247 to the identified bot. As indicated above, in case of an explicit invocation, the input 247 may be based on a part of the utterance that is not associated with the invocation, or there may be no input 247 (e.g., an empty string). In case of an implicit invocation, the input 247 may be the entire utterance.

[0079] The data store 250 comprises one or more computing devices that store data used by various subsystems of the masterbot system 200. As described above, the data store 250 includes rules 252 and skillbot information 254. The rules 252 include, for example, rules for determining by the MIS 220 when an utterance expresses multiple intents and how to split an utterance expressing multiple intents. The rules 252 further include rules for determining by the EIS 230 which part of an utterance that explicitly invokes a skillbot is sent to the skillbot. The skillbot information 254 includes the invocation names of the skillbots in the Chabot system, for example, a list of the invocation names of all skillbots registered to a particular masterbot. The skillbot information 254 can also include information used by the intent classifier 242 to determine a confidence score for each skillbot in the Chabot system, for example, parameters of a machine learning model.

[0080] 3 is a simplified block diagram of a Skillbot system 300 according to an embodiment. The Skillbot system 300 is a computing system that can be implemented in software only, hardware only, or a combination of hardware and software. In an embodiment, such as the embodiment shown in FIG. 1, the Skillbot system 300 can be used to implement one or more Skillbots within a digital assistant.

[0081] The Skillbot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to the MIS 220 of FIG. 2 and provides similar functionality, including being operable to determine using rules 352 in a data store 350: (1) whether an utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each intent of the multiple intents. In an embodiment, the rules applied by the MIS 310 to detect multiple intents and split the utterance are the same as the rules applied by the MIS 220. The MIS 310 receives the utterance 302 and the information to be extracted 304. The information to be extracted 304 is similar to the information to be extracted 205 of FIG. 1 and can be generated using the language parser 214 or a language parser local to the Skillbot system 300.

[0082] The intent classifier 320 can be trained in a manner similar to the intent classifier 242 described above in connection with the embodiment of FIG. 2 and in further detail herein. For example, in one aspect, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of the example utterances associated with the particular skill bot as training utterances. The ground truth for each training utterance becomes the intent of the particular bot associated with the training utterance.

[0083] The utterance 302 can be received directly from a user or provided via a masterbot. For example, if the utterance 302 is provided via a masterbot as a result of processing via the MIS 220 and the EIS 230 in the embodiment shown in FIG. 2, the MIS 310 can be bypassed to avoid repeating the processing already performed by the MIS 220. However, if the utterance 302 is received directly from a user, for example during a conversation that occurs after routing to a skillbot, the MIS 310 can process the utterance 302 to determine whether the utterance 302 represents multiple intents. If so, the MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, for example, utterance "D" 306 and utterance "E" 308. If the utterance 302 does not represent multiple intents, the MIS 310 forwards the utterance 302 to the intent classifier 320 for intent classification without splitting the utterance 302.

[0084] The intent classifier 320 is configured to match the received utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As explained above, a skillbot can be configured with one or more intents, with each intent including at least one example utterance associated with the intent and used to train the classifier. In the embodiment of FIG. 2, the intent classifier 242 of the masterbot system 200 is trained to determine a confidence score for each individual skillbot and a confidence score for the system intent. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skillbot system 300. The classification performed by the intent classifier 242 is at the bot level, whereas the classification performed by the intent classifier 320 is at the intent level and is therefore more granular. The intent classifier 320 has access to intent information 354. For each intent associated with the skillbot system 300, the intent information 354 represents and illustrates the meaning of the intent and typically includes a list of utterances associated with tasks that can be performed by the intent. The intent information 354 may further include parameters generated as a result of training on this utterance list.

[0085] The conversation manager 330 receives as an output of the intent classifier 320 an indication 322 of a particular intent identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some cases, the intent classifier 320 may not be able to determine a match. For example, if the utterance is directed to a system intent or to an intent of a different skill bot, the confidence score calculated by the intent classifier 320 may fall below a confidence score threshold. When this occurs, the skill bot system 300 may refer the utterance to a master bot for processing, e.g., routing to another skill bot. However, if the intent classifier 320 is successful in identifying the intent within the skill bot, the conversation manager 330 will initiate a conversation with the user.

[0086] A conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine may include a default starting state (e.g., when the intent is invoked without additional input) and one or more additional states, each state having associated therewith an action to be performed by the skill bot (e.g., performing a purchase transaction) and / or a dialog to be presented to the user (e.g., questions, responses). Thus, the conversation manager 330 may determine an action / dialog 335 upon receiving an instruction 322 identifying the intent, and may determine additional actions or dialogs depending on subsequent utterances received during the conversation.

[0087] The data store 350 comprises one or more computing devices that store data used by various subsystems of the Skillbot system 300. As shown in Figure 3, the data store 350 includes rules 352 and intent information 354. In an embodiment, the data store 350 can be integrated into a masterbot or digital assistant data store, such as data store 250 of Figure 2.

[0088] Chabot can perform the task of named entity recognition (NER). NER may refer to the task of finding spans of text that constitute proper noun entities and tagging the type of entity. People, places, organizations, and geopolitical entities are four commonly used named entities. However, the term "named entity" may also refer to dates, times, numbers, currencies, etc. Below is an example of the output of a NER system: [Person David Smith] paid [Currency Number 95.00] to [Seller Palm Court Restaurant] in [Location New York] on [Date September 12th].

[0089] NER is an important NLP task that requires text analysis to understand relative meaning and sentiment. For example, knowing if a named entity like "Sydney" is the name of a place, a person, or a university is important for many natural language understanding tasks. Some of the important tasks that benefit from NER are discussed below. In question answering, NER can be used to identify the range of text that constitutes the answer. For example, based on the above example, the answer to the following question would be "$95." How much did David Smith pay at the restaurant? In aspect-based sentiment analysis, NER can identify entities as a first step to know customer sentiment towards them. For example, in the following example sentence, there can be both positive and negative opinions about the entities "Nikon camera" and "Canon camera."

[0090] Nikon cameras are great, better than Canon cameras. NER is one of the techniques used in digital assistants. NER can be used in digital assistants to search and classify user's words into predefined categories such as PER (which can be an abbreviation for PERSON), DATE, and TIME. These types of information can be used by digital assistants to process user's requests. For example, to generate an expense report from the following sentence, digital assistants can use NER model to identify SFO, $10, and May 21 as MER (which can be an abbreviation for MERCHANT), CUR (which can be an abbreviation for CURRENCY), and DATE entities. LOC is a category identifier that is an abbreviation for LOCATION.

[0091] [ MER SFO] will charge [ CUR $10] was charged.

[0092] NER is not an easy task due to the difficulties posed by segmentation and type ambiguity. Segmentation ambiguity arises from the complexity of finding entities and their boundaries in a sentence. For example, multiple words in a sentence may represent a single entity. As shown below, "New York Times" is a single entity (with MER tag) consisting of three words.

[0093] [ MER New York Times] to $20 this year. Furthermore, the same word may refer to multiple entity types, and NER must resolve this type of ambiguity depending on the context: for example, "Sydney" can refer to a person or a place in the following sentence:

[0094] Late Monday afternoon, PER I got a text message from Sydney. [ LOC The best view in Sydney.

[0095] Understanding classifiers can be helpful in understanding how NER algorithms work. A classifier can be a system that learns a function that, given a set of inputs, determines a class or label among a predefined finite set of categories. As a common example of a text classifier, sentiment analysis can be used to identify the sentiment and direction of opinions expressed in a piece of text (e.g., a product review) as positive, negative, or neutral.

[0096] In comparison, sequence labeling classifies (i.e., assigns a label to) every word in a text, while NER as a sequence labeling task can involve identifying and labeling subsequences of words. To do this, the Begins, Inside, Outside (BIO) labeling scheme can be a method to identify subsequences using a sequence labeler. The labels capture both the boundaries and the types of the named entities. In this technique, a token that starts a range of interest may be tagged with the label B, tokens that occur within the range are tagged with the label I, and tokens that are outside the range of interest are tagged with the label O. There may be only one O tag, but there may be different B and I tags for each named entity class (e.g., I-DATE and I-MER). The following example shows the BIO encoding of two adjacent named entities of the same type (i.e., [ MER Plaza Hotel's] and [ MER The authors show how to define the boundary between the two (Palm Court Restaurant). Table 1 shows an example of a classification of utterances.

[0097] [Table 1]

[0098] The intelligent assistant NER can incorporate the following cutting-edge technologies:

[0099] Deep learning refers to neural networks composed of multiple processing layers that automatically learn different levels of data representation. Each layer learns to transform the input data into a slightly more abstract and complex representation. This allows for feature learning, eliminating the need for a wealth of hand-crafted features. This makes deep neural networks a suitable tool for solving complex problems providing sufficient data. Deep learning has significantly improved the state of the art in natural language processing, including NER tasks, by discovering complex structures in large datasets and integrating long-range information in the input.

[0100] Pretraining refers to the process of training a network on a large external dataset, such as Wikipedia or CommonCrawl (an open repository of data freely provided by crawling the web). The intuition behind pretraining is that once a network has been trained on a sufficiently large and general dataset, it will effectively act as a general-purpose model of the text world. With a pretrained network, we can leverage the knowledge gained in solving one problem (e.g., language modeling) to initialize the backbone of a new network for solving a different but related problem (e.g., text classification) without starting from scratch (also known as transfer learning).

[0101] Sequence labeling models (e.g., NER) can lead to inconsistencies between adjacent labels. For example, in the following sentence, the model labels "Sydney" as B-PER and "Harbor" as I-LOC, but this is not a valid sequence because a label boundary cannot start with an I tag. Therefore, the correct sequence would be B-LOC and I-LOC. For this reason, conditional random fields (CRFs) are used to enforce that adjacent labels are consistent. Table 3 shows example labeling of parts of a sentence.

[0102] [Table 2]

[0103] To that end, CRF learns tag-tag weights during training and avoids generating impossible BIO tag sequences by assigning highly negative weights (e.g., X values ​​in Table 3 below) to those sequences.

[0104] [Table 3]

[0105] Deep learning models with a large number of parameters can be prone to overfitting to the training data. This causes the model to learn the noise in the training data, which leads to poor performance when evaluating the model on new data. Dropout can be a computationally inexpensive and highly effective technique to address this problem. The key idea is to randomly ignore or remove some layer output devices from the model during training to prevent the devices from co-adapting too much (i.e., overfitting).

[0106] Deep learning models require different constraints and capabilities to generalize well to different data patterns. These constraints are controlled by several measures called hyperparameters that must be tuned so that the model can optimally solve the problem. Hyperparameter tuning refers to the process of finding the optimal set of hyperparameters for a learning algorithm.

[0107] The disclosed NER system may include several model improvements, training improvements, and data improvements.

[0108] For model improvement, the disclosed NER system can combine context and gazetteer features. The disclosed NER model can be a hybrid model that combines context features with gazetteer features. The novel method combines context features with external knowledge resources called gazetteers to improve the model performance. Gazetteers can be lists of named entities such as organizations, countries, cities, people names, etc., that are matched against the unstructured text to provide additional capabilities to the model. For example:

[0109] [Table 4]

[0110] By leveraging external knowledge, the disclosed NER models are less dependent on annotated data, which can be very costly and labor-intensive to collect.

[0111] The disclosed NER system can include fixed CRF tag-tag transitions on small training data sets. With large training data sets, the CRF learns that inconsistent tag pairs such as OI-LOCI-PER do not appear in the training data, and therefore assigns very negative weights to these tag-to-tag transitions. However, when the training data set is small, the CRF may learn a model that generates these inconsistent tag-to-tag transitions. Therefore, the disclosed NER system can introduce a novel technique to stop the CRF from finding inconsistent tag-to-tag transitions by setting the weights of inconsistent transitions to very negative values ​​and modifying the corresponding weights after training. This technique ensures that consistent tag-to-tag transitions are generated even when training data is sparse.

[0112] To improve training, the disclosed NER system can include selective dropout. A new technique called selective dropout allows for a higher dropout rate to be applied to entity tokens compared to non-entity tokens. This allows the model to focus on contextual information during training, making the model more reliable and robust to different values ​​for each entity type. Example: (The blue parts are entity values.) Training example: Please deposit $100 into my savings account.

[0113] Test example: Could you please transfer £250 to my bank account? The disclosed NER system can include extensive hyperparameter tuning. The hypertuning framework can cover a wide range of values, allowing the disclosed NER system to perform extensive hyperparameter tuning and identify optimal hyperparameter choices for achieving high quality results.

[0114] The disclosed NER system can include data improvements. One of the data improvements can include a duplicate chunking mechanism. A limitation of existing state-of-the-art models is that memory and computation requirements grow as the square of the length of the input sequence. Given the limitations of commonly available hardware, current pre-trained language models can only process input sequences of up to 512 tokens. To alleviate this limitation, the disclosed system can use a mechanism to split long text into duplicate chunks, and each chunk and its corresponding label is treated as an individual example during training. For evaluation / inference, predictions from chunks of the same input text need to be merged. This mechanism allows the model to process larger sequences, significantly reducing training and inference times while maintaining high performance.

[0115] 4 shows a system 400 for processing long text for a pre-trained language model. An utterance 402 can be received by a length determination engine 410. The length determination engine can determine whether the length exceeds a predetermined threshold of tokens. A token can be a word, part of a word, or punctuation.

[0116] If the length exceeds a predetermined threshold of token pieces, the chunking engine 420 can split the utterance 402 into multiple overlapping chunks. For example, the chunking engine 420 can split the utterance 402 into chunk A 422, chunk B 424, and chunk C 426. Although three chunks are shown, the disclosed technology is not so limited and is applicable to any number of chunks. For example, as few as two chunks can be processed depending on the capabilities of the processing system.

[0117] The named entity recognizer 430 can determine a classifier or label for each chunk. For example, the named entity recognizer 430 can determine label A 432 for chunk A 422, label B 434 for chunk B 424, and label C 436 for chunk C 426. As described above, the named entity recognizer 430 can assign a label to each token in a chunk. The named entity recognizer can also assign a label to each chunk of multiple chunks. The score engine 440 can receive label A 432 for chunk A 422, label B 434 for chunk B 424, and label C 436 for chunk C 426. Each label can be assigned a chunk score. The score engine 440 can calculate a final label and associated confidence score for each chunk of the overlapping chunks of the token piece by combining two confidence scores, the confidence score from the first chunk and the confidence score from the second chunk. The score engine 440 can determine a final annotated label for the set of utterances based on a combination of at least two confidence scores. The score engine 440 can store the final annotated scores 442 in a memory.

[0118] FIG. 5 illustrates converting an example utterance 502 into chunks. The example utterance 502 can be divided into chunks, each chunk being a certain number of words. As shown in FIG. 5, the example utterance 502 can be divided into segments of a predetermined length that includes a number of words. In some aspects, the predetermined length can be 32 words. As an example, the utterance 502 can be divided into chunk A 504, which can include indexes 1 through 10 for an example chunk size of 10 tokens. Chunk B can include indexes 6 through 15 for an example chunk size of 10 tokens. As shown in FIG. 4, the named entity recognizer 430 can process smaller chunks more efficiently. For example, it can be faster for the entity recognizer 430 to process a 10-word segment than it can process the entire example utterance 502.

[0119] The named entity recognizer 430 may analyze the utterance 502 and determine a tag or label for each token. For example, the first token (index 1) of the utterance 502 is the word "My" and the named entity recognizer 430 may assign the token a tag of "O", meaning that the token is out of scope. Similarly, the named entity recognizer 430 may assign the tokens "name", "is", "living", "in", "and", "working", and "for" a tag of "O". The named entity recognizer 430 may assign the token "Davis" at index 4 a tag of "B-PER" meaning the start of a person's name. The named entity recognizer 430 may assign the token "Brisbane" at index 8 a tag of "B-LOC" meaning the start of a place. The named entity recognizer 430 may assign the token "Australia" at index 10 a tag of "B-LOC" meaning the start of a place. The named entity recognizer 430 may assign the token "oracle" at index 14 a tag of "B-ORG" signifying the beginning of an organization. The named entity recognizer 430 may assign the token "corp" at index 15 a tag within the organization name range.

[0120] 5 shows a chunk size of 10 tokens and an overlap size of 5 chunks. For example, a first overlap portion 508 can include the latter half of chunk A 504. A second overlap portion 510 can include the first half of chunk B 506.

[0121] 5 further illustrates a technique for combining predictions from chunks. For example, while analyzing chunk A, named entity recognizer 430 may properly label “Brisbane” at index 8, but miss “Australia” at index 10. While analyzing chunk B, named entity recognizer 430 may properly label “Australia” at index 10, but miss “Brisbane” at index 8. The outputs of the analyses from chunk A and chunk B are combined, resulting in “Brisbane” and “Australia” being properly labeled.

[0122] Figure 6 illustrates a technique for combining predictions. The score engine 440 shown in Figure 4 can determine a confidence score for each label from the named entity recognizer 430. For example, the score engine 440 can determine confidence scores for indices 6, 7, 8, and 9. The confidence scores indicate a confidence of 0.9 for the assigned label, with 90% confidence that the label is correct. For index 10 of chunk A, the score engine 440 can determine a confidence score of 0.5 for the assigned label "O," with 50% confidence that the label is correct.

[0123] For chunk B, the score engine 440 may determine confidence scores for indices 6, 7, 9, and 10, indicating a confidence score of 0.9 for the assigned label and a 90% confidence that the label is correct. For index 8 of chunk B, the score engine 440 may determine a confidence score of 0.5 for the assigned level "O", indicating a 50% confidence that the label is correct.

[0124] There are several possible outcomes for the combined output. For example, output strategies can include "half," "first," "second," and "maximum." For the "half" strategy, the score engine 440 can use 1 / 2 the prediction from the first chunk and 1 / 2 the prediction from the second chunk. For the "first" strategy, the score engine 440 can use the prediction from the first chunk. For the "second" strategy, the score engine 440 can use the prediction from the second chunk. For the "maximum" strategy, the score engine 440 can determine a prediction based on the maximum confidence score.

[0125] As shown in Figure 6, the label at index 8 of "Brisbane" and the label at index 10 of "Australia" are both accurate, so the "Half" strategy can provide accurate results. The "First" strategy correctly identifies the label at index 8 of "Brisbane" but mislabels the token piece of "Australia". The "Second" strategy misidentifies the label at index 8 of "Brisbane" but correctly labels the token piece of "Australia". The "Max" strategy can provide accurate results, because the label at index 8 of "Brisbane" and the label at index 10 of "Australia" are both accurate.

[0126] In some aspects, the predetermined length may be 32 token pieces. As shown in FIG. 6, each chunk may be divided into multiple elements. For example, each chunk may provide an overlap between chunks (which may be abbreviated as l). The amount of overlap between chunks may be predetermined. For example, for a chunk of 32 token pieces, the overlap may be 16 token pieces (subwords).

[0127] FIG. 7 is a flowchart of an example process 700 associated with systems and techniques for processing long text for pre-trained language models. In some implementations, one or more process blocks of FIG. 7 may be performed by a computing device (e.g., computing device 1000). In some implementations, one or more process blocks of FIG. 7 may be performed by another device, or a group of devices separate from or including the computing device. Additionally or alternatively, one or more process blocks of FIG. 7 may be performed by one or more components of device 1000, such as processing subsystem 1004, storage subsystem 1018, I / O subsystem 1008, communication subsystem 1024, and / or bus subsystem 1002.

[0128] As shown in FIG. 7, process 700 can include receiving a set of utterances at a data processing system and training or interfacing with a named entity recognizer to assign a label to each token piece from the set of utterances (block 710). For example, a computing device can receive a set of utterances at a data processing system and training or interfacing with a named entity recognizer to assign a label to each token piece from the set of utterances as described above. In various embodiments, the set of utterances can be received through user input via Chabot using a keyboard. In various embodiments, the set of utterances can be received from a user audibly via Chabot using a microphone. The system can convert the verbal utterances into text input.

[0129] As further shown in FIG. 7, process 700 may include determining a length of the set of utterances (block 720). For example, a computing device may determine the length of the set of utterances as described above. Process 700 may determine the length of the set of utterances by dividing the utterance into multiple token pieces. A token piece may be a word, a part of a word, or a punctuation mark. A complex word may be broken down into one or more token pieces. The length may be the number of token pieces in the utterance.

[0130] As further shown in FIG. 7, process 700 may include when the length of the set of utterances exceeds a predetermined threshold of token pieces, and may include splitting the set of utterances into multiple overlapping chunks of token pieces (block 730). The length of the utterance may be compared to a predetermined threshold. The predetermined threshold may be varied as needed. In various aspects, the predetermined threshold may be 512 token pieces. One of ordinary skill in the art will appreciate that other thresholds are within the scope of the present disclosure.

[0131] As further shown in FIG. 7, the process 700 can include assigning a label along with a confidence score to each token piece in the chunk (block 740). As described above, a label can be assigned to each token piece in the utterance. The named entity recognizer 430 can identify and label subsequences of words. In various embodiments, a start, interior, and exterior (BIO) label scheme can be used. The BIO label scheme can identify subsequences, where the labels capture both the boundaries and the types of the named entities. In this technique, tokens that start a range of interest can be tagged with a label B, tokens that occur within the range are tagged with a label I, and tokens that are outside the range are tagged with a label O. There may be only one O tag, but there can be different B and I tags for each named entity class (e.g., I-DATE and I-MERCHANT). Each label can be assigned a confidence score that indicates the confidence that the label is correct.

[0132] As further shown in FIG. 7, process 700 may include determining a final label and associated confidence score for each chunk of the overlapping chunks of the token piece by combining two confidence scores, the confidence score from the first chunk and the confidence score from the second chunk (block 750).

[0133] As further shown in FIG. 7, process 700 may include determining a final annotated label for the set of utterances based on a combination of at least two confidence scores (block 760).

[0134] As further shown in FIG. 7, the process 700 may include storing the final annotated label in memory (block 770). The process 700 may utilize several different strategies for determining the final annotated label. For example, the output strategies may include "half", "first", "second", and "maximum". For the "half" strategy, the score engine 440 may use 1 / 2 prediction from the first chunk and 1 / 2 prediction from the second chunk. For the "first" strategy, the score engine 440 may use a prediction from the first chunk. For the "second" strategy, the score engine 440 may use a prediction from the second chunk. For the "maximum" strategy, the score engine 440 may determine a prediction based on a maximum confidence score.

[0135] Process 700 may include additional implementations, such as any combination of single implementations or implementations described below and / or in conjunction with one or more other processes described elsewhere herein.

[0136] In the first implementation, each chunk and its corresponding sequence of labels are treated as a separate example during training.

[0137] In the second implementation, either alone or in combination with the first implementation, an utterance of length N is split into (NL) / (KL) overlapping chunks, where K is the chunk size and L is the overlap size.

[0138] In a third implementation, alone or in combination with one or more of the first and second implementations, an overall score and final annotated label for a token piece are determined based at least in part on the confidence score from a first selected chunk of the multiple overlapping chunks.

[0139] In a fourth implementation, alone or in combination with one or more of the first through third implementations, determining an overall score for a token piece is based on the position of the token piece within the token piece overlap chunk, and the overall score is a first confidence score from the first chunk if the first chunk is in the first half of the token piece overlap chunk, and a second confidence score from the second chunk if the second chunk is in the second half of the token piece overlap chunk.

[0140] In a fifth implementation, alone or in combination with one or more of the first to fourth implementations, the determination of the overall score is based on the maximum confidence score of multiple overlapping chunks.

[0141] In a sixth implementation, alone or in combination with one or more of the first through fifth implementations, the predetermined threshold of token pieces is 512 token pieces and the predetermined number of duplicate token pieces is 128 token pieces.

[0142] Although Figure 7 illustrates example blocks of process 700, in some implementations process 700 may include more, fewer, different, or differently arranged blocks than those illustrated in Figure 7. Additionally or alternatively, two or more blocks of process 700 may be performed in parallel.

[0143] 8 shows a simplified diagram of a distributed system 800 for implementing an embodiment. In the illustrated embodiment, the distributed system 800 includes one or more client computing devices 802, 804, 806, and 808 coupled to a server 812 via one or more communication networks 810. The client computing devices 802, 804, 806, and 808 can be configured to run one or more applications.

[0144] In various aspects, the server 812 may be adapted to run one or more services or software applications that enable techniques for processing long texts of pre-trained language models.

[0145] In an aspect, server 812 may also provide other services or software applications, which may include non-virtual and virtual environments. In some aspects, these services may be provided as web-based or cloud services, such as under a software-as-a-service (SaaS) model, to users of client computing devices 802, 804, 806, and / or 808. Users operating client computing devices 802, 804, 806, and / or 808 may in turn utilize one or more client applications to interact with server 812 and utilize the services provided by these components.

[0146] In the configuration shown in Figure 8, server 812 may include one or more components 818, 820, and 822 that implement the functions performed by server 812. These components may include software components that may be executed by one or more processors, hardware components, or combinations thereof. It should be understood that a variety of different system configurations are possible that may differ from distributed system 800. Thus, the embodiment shown in Figure 8 is an example of a distributed system for implementing the system of the embodiments and is not intended to be limiting.

[0147] A user may use client computing devices 802, 804, 806, and / or 808 for techniques for processing long texts of pre-trained language models in accordance with the teachings of this disclosure. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user via the interface. Although only four client computing devices are shown in FIG. 8, any number of client computing devices may be supported.

[0148] Client devices may include various types of computing systems, such as portable handheld devices, general purpose computers such as personal computers or laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems such as Google Chrome™ OS), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android™, BlackBerry®, Palm OS®). Portable handheld devices may include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices. The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® game consoles with or without Kinect® gesture input devices, Sony Play Station® systems, various gaming systems offered by Nintendo®, etc.), etc. The client devices may run a variety of different applications, such as various Internet-related apps, communication applications (e.g., email applications, short message service (SMS) applications), etc., and may use a variety of communication protocols.

[0149] Network 810 may be any type of network familiar to those skilled in the art and may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), Apple Talk, etc. By way of example only, network 810 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite), Bluetooth, and / or other wireless protocols), and / or any combination of these and / or other networks.

[0150] The servers 812 may be comprised of one or more general purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable arrangements and / or combinations. The servers 812 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage that may be virtualized to maintain the server's virtual storage. In various aspects, the servers 812 may be adapted to run one or more services or software applications that provide the functionality described in the preceding disclosure.

[0151] The computing system of server 812 may run one or more operating systems, including any of those mentioned above, as well as any commercially available server operating system. Server 812 may also run any of a variety of additional server applications and / or mid-tier applications, including a HyperText Transport Protocol (HTTP) server, a File Transfer Protocol (FTP) server, a Common Gateway Interface (CGI) server, a JAVA server, a database server, and the like. Exemplary database servers include, but are not limited to, those commercially available from Oracle, Microsoft, Sybase, IBM (International Business Machines), and the like.

[0152] In some implementations, the server 812 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of the client computing devices 802, 804, 806, and 808. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third party information sources and continuous data streams, which may include real-time events associated with sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like. The server 812 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of the client computing devices 802, 804, 806, and 808.

[0153] The distributed system 800 may also include one or more data repositories 814, 816. These data repositories may be used in some embodiments to store data and other information. For example, one or more of the data repositories 814, 816 may be used to store information about techniques for processing long texts of pre-trained language models (e.g., intent scores, overall scores). The data repositories 814, 816 may reside in a variety of locations. For example, the data repository used by the server 812 may be local to the server 812, may be remote from the server 812, and may communicate with the server 812 via a network-based or dedicated connection. The data repositories 814, 816 may be of different types. In some embodiments, the data repository used by the server 812 may be a database, such as, for example, a relational database, such as a database provided by Oracle Corporation® or other vendors. One or more of these databases may be adapted to enable storage, updating, and retrieval of data in the database in response to commands in the Structured Query Language (SQL) format.

[0154] In an aspect, one or more of the data repositories 814, 816 may be used by an application to store application data. The data repositories used by an application may be of various types, such as, for example, a key / value store repository, an object store repository, or a general storage repository supported by a file system.

[0155] In an aspect, the techniques for processing long text for pre-trained language model functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 6 is a simplified block diagram of a cloud-based system environment in which various text processing related services may be provided as cloud services according to an aspect. In the embodiment shown in FIG. 6, a cloud infrastructure system 602 may provide one or more cloud services that may be requested by users using one or more client computing devices 604, 606, and 608. The cloud infrastructure system 602 may comprise one or more computers and / or servers, which may include those described above for server 812. The computers in the cloud infrastructure system 602 may be organized as general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0156] The network 610 can facilitate communication and exchange of data between the clients 604, 606, and 608 and the cloud infrastructure system 602. The network 610 can include one or more networks. The networks can be of the same type or different types. The network 610 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0157] The embodiment shown in Figure 9 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some other aspects, cloud infrastructure system 902 may have more or fewer components than those shown in Figure 9, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 9 shows three client computing devices, in alternative aspects any number of client computing devices may be supported.

[0158] The term cloud services is generally used to refer to services provided to users on demand by a service provider's system (e.g., cloud infrastructure system 902) over a communication network such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premise servers and systems. The cloud service provider's system is managed by the cloud service provider. Thus, customers can use cloud services provided by the cloud service provider without separately purchasing licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system can host applications, and users can order and use the applications on demand over the network 910 (e.g., the Internet) without purchasing infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation of Redwood Shores, California.

[0159] In an aspect, cloud infrastructure system 902 may provide one or more cloud services using different models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, and others including hybrid service models. Cloud infrastructure system 902 may include a set of applications, middleware, databases, and other resources that enable the delivery of various cloud services.

[0160] In the SaaS model, an application or software may be provided to a customer as a service over a communications network such as the Internet without the customer having to purchase the underlying application hardware or software. For example, the SaaS model may be used to provide customers with access to on-demand applications hosted by the cloud infrastructure system 902. Examples of SaaS services offered by Oracle Corporation include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, etc.

[0161] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware and networking resources) as a cloud service to customers to provide elastic computing and storage capabilities. A variety of IaaS services are offered by Oracle Corporation.

[0162] The PaaS model is generally used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without the customer having to procure, build, or maintain such resources. Examples of PaaS services provided by Oracle Corporation include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0163] Cloud services are generally provided on an on-demand self-service basis, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 902 via a subscription order. Cloud infrastructure system 902 then performs processing to provide the services requested in the customer's subscription order. Cloud infrastructure system 902 may be configured to provide one or more cloud services.

[0164] Cloud infrastructure system 902 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 902 can be owned by a third-party cloud service provider and cloud services are provided to public customers, which can be individuals or businesses. In certain other aspects, under a private cloud model, cloud infrastructure system 902 can be operated within an organization (e.g., within a corporate organization) and services can be provided to customers within the organization. For example, the customers can be various departments of a company, such as human resources, payroll, or individuals within a company. In certain other aspects, in a community cloud model, cloud infrastructure system 902 and the services provided can be shared by multiple organizations within an associated community. Various other models, such as hybrids of the above models, can also be used.

[0165] Client computing devices 904, 906, and 908 may be of different types (e.g., devices 802, 804, 806, and 808 shown in FIG. 8) and may be capable of running one or more client applications. A user may use a client device to interact with cloud infrastructure system 902, such as to request a service provided by cloud infrastructure system 902. For example, a user may use a client device to request a chatbot service described in this disclosure.

[0166] In some aspects, the processing performed by cloud infrastructure system 902 to provide the Chabot service may include big data analytics. This analysis may include using, analyzing, and manipulating large data sets to detect and visualize various trends, behaviors, relationships, and the like in the data. This analysis may be performed by one or more processors and may involve parallel processing of the data, running simulations using the data, and the like. For example, big data analytics may be performed by cloud infrastructure system 902 to determine the intent of an utterance. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0167] 9, cloud infrastructure system 902 may include infrastructure resources 930 utilized to facilitate the provision of various cloud services provided by cloud infrastructure system 902. Infrastructure resources 930 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0168] In an aspect, to facilitate efficient provisioning of these resources to support various cloud services offered by cloud infrastructure system 902 for various customers, resources may be bundled into resource sets or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In an aspect, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, a second set of pods may include a different combination of resources than the pods in the first set and may be provisioned for Java services, etc. For some services, resources allocated to provision a service may be shared between services.

[0169] The cloud infrastructure system 902 may itself use services 932 internally that are shared by different components of the cloud infrastructure system 902 and that facilitate provisioning of services by the cloud infrastructure system 902. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services enabling cloud support, email services, notification services, file transfer services, etc.

[0170] Cloud infrastructure system 902 may comprise multiple subsystems. These subsystems may be implemented in software or hardware, or a combination thereof. As shown in FIG. 9, the subsystems may include a user interface subsystem 912 that allows a user or customer of cloud infrastructure system 902 to interact with cloud infrastructure system 902. User interface subsystem 912 may include a variety of different interfaces, such as a web interface 914, an online store interface 916, and the cloud services offered by cloud infrastructure system 902 are advertised and available for purchase by consumer and other interfaces 918. For example, a customer may use a client device to request one or more services offered by cloud infrastructure system 902 (service request 934) using one or more of interfaces 914, 916, and 918. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 902, and place a subscription order for one or more services offered by cloud infrastructure system 902 to which the customer wishes to subscribe. The service request may include information identifying the customer and the one or more services to which the customer wishes to subscribe. For example, a customer may place a subscription order for Chabot-related services provided by cloud infrastructure system 902. As part of the order, the customer may provide information identifying an input (e.g., an utterance).

[0171] In certain aspects, such as the embodiment depicted in FIG. 9, the cloud infrastructure system 902 may include an order management subsystem (OMS) 920 configured to process new orders. As part of this process, the OMS 920 creates an account for the customer if one has not already been created, receives billing and / or accounting information from the customer that will be used to bill the customer for providing the requested services to the customer, verifies the customer information, and, once verified, books the order for the customer. Various workflows may be configured to coordinate and prepare the order for provisioning (if one has not already been created).

[0172] Upon proper validation, the OMS 920 may invoke an order provisioning subsystem (OPS) 924 configured to provision resources for the order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring the resources to facilitate the services requested by the customer's order. The manner in which resources are provisioned for the order and the type of resources provisioned may vary depending on the type of cloud service ordered by the customer. For example, according to one workflow, the OPS 924 may be configured to determine the specific cloud service being requested and identify the number of pods that may be pre-configured for that specific cloud service. The number of pods allocated for the order may vary depending on the size / amount / level / scope of the service being requested. For example, the number of pods allocated may be determined based on the number of users supported by the service, the duration for which the service is being requested, etc. The pods allocated may then be customized for the specific requesting customer to provide the requested service.

[0173] The cloud infrastructure system 902 may send a response or notification 944 to the requesting customer to indicate when the requested service is available for use. In some cases, information (e.g., a link) may be sent to the customer that enables the customer to begin using and taking advantage of the requested service.

[0174] Cloud infrastructure system 902 may provide services to multiple customers. For each customer, cloud infrastructure system 902 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 902 may also collect usage statistics regarding the customers' use of the subscription services. For example, statistics may be collected such as the amount of storage used, the amount of data transferred, the number of users, system uptime and downtime, etc. This usage information may be used to bill the customer. Billing may occur, for example, on a monthly cycle.

[0175] Cloud infrastructure system 902 can provide services to multiple customers in parallel. Cloud infrastructure system 902 can store information for these customers, possibly including proprietary information. In an aspect, cloud infrastructure system 902 comprises an identity management subsystem (IMS) 928 configured to manage customer information and provide separation of management information such that information related to one customer cannot be accessed by another customer. IMS 928 can be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, and services for managing customer identities and roles and related functions.

[0176] FIG. 10 illustrates an exemplary computer system 1000 that can be used to implement certain aspects. For example, in some aspects, the computer system 1000 can be used to implement the system 400 for processing long texts in a pre-trained language model, as shown in FIG. 4, or any of the various servers and computer systems described above. As shown in FIG. 10, the computer system 1000 includes various subsystems, including a processing subsystem 1004 that communicates with many other subsystems via a bus subsystem 1002. These other subsystems can include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. The storage subsystem 1018 can include a non-transitory computer-readable storage medium, including a storage medium 1022 and a system memory 1010.

[0177] Bus subsystem 1002 provides a mechanism that allows the various components and subsystems of computer system 1000 to communicate with each other as intended. Although bus subsystem 1002 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1002 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, and the like. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. It may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, and the like.

[0178] The processing subsystem 1004 controls the operation of the computer system 1000 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 1000 may be organized into one or more processing units 1032, 1034, etc. The processing units may include one or more processors, one or more cores from the same or different processors, combinations of cores and processors, or other combinations of cores and processors. In some aspects, the processing subsystem 1004 may include one or more dedicated co-processors, such as a graphics processor, digital signal processor (DSP), etc. In some aspects, some or all of the processing units of the processing subsystem 1004 may be implemented using customized circuitry, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0179] In some embodiments, the processing units in the processing subsystem 1004 can execute instructions stored in the system memory 1010 or the computer readable storage medium 1022. In various embodiments, the processing units can execute various program or code instructions and maintain multiple simultaneously executing programs or processes. At any time, some or all of the program code being executed can be located in the system memory 1010 and / or the computer readable storage medium 1022, potentially including one or more storage devices. Through appropriate programming, the processing subsystem 1004 can provide the various functions described above. If the computer system 1000 is running one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0180] In an aspect, a processing acceleration unit 1006 may be optionally provided to accelerate the overall processing performed by the computer system 1000, to perform customized processing, or to offload portions of the processing performed by the processing subsystem 1004.

[0181] The I / O subsystem 1008 may include devices and mechanisms for inputting information to the computer system 1000 and / or outputting information from or through the computer system 1000. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information to the computer system 800. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors that allow a user to control and operate the input device, Microsoft Xbox® 360 game controllers, devices that provide an interface for receiving input using gestures and voice commands, and the like. User interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector, that detects a user's eye movements (e.g., "blinking" when taking a photo or selecting a menu) and translates the eye gestures as input to the input device (e.g., Google Glass®). Additionally, user interface input devices may include voice recognition sensing devices that allow a user to interact with a voice recognition system (e.g., the Siri® navigator) through voice commands.

[0182] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, eye-tracking devices, etc. Additionally, user interface input devices may include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, etc.

[0183] In general, use of the term output device is intended to include all possible types of devices and mechanisms for outputting information from computer system 1000 to a user or to another computer. User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. Display subsystems may be flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs) or plasma displays, projection devices, touch screens, etc. For example, user interface output devices include, but are not limited to, a variety of display devices that visually convey textual, graphical, and audio / video information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, modems, etc.

[0184] The storage subsystem 1018 provides a repository or data store for storing information and data used by the computer system 1000. The storage subsystem 1018 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data structures that provide the functionality of some aspects. The storage subsystem 1018 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1004, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 1004. The storage subsystem 1018 can also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0185] The storage subsystem 1018 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 10, the storage subsystem 1018 includes a system memory 1010 and a computer-readable storage medium 1022. The system memory 1010 may include multiple memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution, and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1000, such as during startup, may typically be stored in a ROM. The RAM typically contains data and / or program modules currently being operated and executed by the processing subsystem 1004. In some implementations, the system memory 1010 may include multiple different types of memories, such as static random access memory (SRAM), dynamic random access memory (DRAM), and the like.

[0186] 10, system memory 1010 may load executing application programs 1012, which may include various applications, such as a web browser, a mid-tier application, a relational database management system (RDBMS), program data 1014, and an operating system 1016. By way of example, operating system 1016 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems, such as various versions of iOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0187] The computer-readable storage medium 1022 can store programming and data structures that provide functionality of some aspects. The computer-readable medium 1022 can provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1000. When executed by the processing subsystem 1004, software (programs, code modules, instructions) that provide the functionality described above can be stored in the storage subsystem 1018. As an example, the computer-readable storage medium 1022 can include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a digital video disk (DVD), an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 1022 can include, but is not limited to, a Zip® drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, and the like. The computer readable storage medium 1022 may also include solid state drives (SSDs) based on non-volatile memory such as flash memory based SSDs, enterprise flash drives, SSDs based on volatile memory such as solid state RAM, dynamic RAM, static RAM, dynamic random access memory (DRAM) based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory based SSDs.

[0188] In an aspect, the storage subsystem 1018 may also include a computer readable storage medium reader 1020, which may be further connected to a computer readable storage medium 1022. The reader 1020 may be configured to receive and read data from a memory device, such as a disk, a flash drive, or the like.

[0189] In an embodiment, computer system 1000 may support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, computer system 1000 may provide support for running one or more virtual machines. In an embodiment, computer system 1000 may execute a program, such as a hypervisor, that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computing (e.g., processors, cores), I / O, and network resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same or different from the operating systems run by other virtual machines executed by computer system 1000. Thus, multiple operating systems may potentially be run by computer system 1000 simultaneously.

[0190] The communications subsystem 1024 provides an interface to other computer systems and networks. The communications subsystem 1024 serves as an interface for sending and receiving data from the computer system 1000 to and from other systems. For example, the communications subsystem 1024 allows the computer system 1000 to establish a communications channel with one or more client devices over the Internet to send and receive information from the client devices. For example, the communications subsystem may be used to send a response to a user regarding Chabot's query.

[0191] The communications subsystem 1024 can support both wired and / or wireless communications protocols. For example, in an aspect, the communications subsystem 1024 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution)), Wi-Fi (IEEE 802.XX family of standards, other mobile communications technologies, or a combination thereof), global positioning system (GPS) receiver components, and / or other components. In some aspects, the communications subsystem 1024 can provide wired network connections (e.g., Ethernet) in addition to or instead of a wireless interface.

[0192] The communications subsystem 1024 can send and receive data in a variety of formats. For example, in some aspects, in addition to other formats, the communications subsystem 1024 can receive incoming communications in the form of structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. For example, the communications subsystem 1024 may be configured to receive (or send) data feeds 1026 in real time from users of social media networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, rich site summary (RSS) feeds, and / or real-time updates from one or more third party information sources.

[0193] In an aspect, the communications subsystem 1024 is configured to receive data in the form of a continuous data stream, which may include an event stream 1028 of real-time events and / or event updates 1030 that may have no explicit end and may be continuous or unlimited in nature. Examples of applications that generate continuous data may include sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0194] The communications subsystem 1024 may also be configured to communicate data from the computer system 1000 to other computer systems or networks. Data may be communicated in a variety of forms, such as structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1000.

[0195] The computer system 1000 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a personal digital assistant (PDA)), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the ever-changing nature of computers and networks, the description of the computer system 1000 shown in FIG. 10 is intended only as a specific example. Many other configurations are possible having more or fewer components than the system shown in FIG. 10. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will appreciate other methods and / or techniques for implementing various aspects.

[0196] While specific aspects have been described, various modifications, variations, alternative configurations, and equivalents are possible. The embodiments are not limited to operating in one specific data processing environment, but may freely operate in multiple data processing environments. Moreover, while certain aspects have been described using a particular sequence of transactions and steps, those skilled in the art will appreciate that this is not intended to be limiting. Although some flow charts describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may include additional steps not included in the figures. Various features and aspects of the above aspects may be used individually or in combination.

[0197] Additionally, while certain aspects have been described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are possible. Certain aspects may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0198] Where a device, system, component, or module is described as being configured to perform a certain operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor) to perform the operation, such as by executing computer instructions or code, or a processor or core that is programmed to execute code or instructions stored in a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0199] Specific details are given in the present disclosure to provide a thorough understanding of the aspects. However, the aspects may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary details to avoid obscuring the aspects. This description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of other aspects. Rather, the description of the aspects above provides those skilled in the art with an effective description for implementing various aspects. Various changes may be made in the function and arrangement of elements.

[0200] Accordingly, the specification and drawings should be regarded in an illustrative, rather than a restrictive, sense. However, it will be apparent that additions, subtractions, deletions, and other modifications and alterations may be made without departing from the broader spirit and scope of the appended claims. Thus, while specific embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are intended to be within the scope of the following claims.

Claims

1. receiving, at a data processing system, a set of utterances and training or interfacing with a named entity recognizer to assign a label to each token piece from said set of utterances; determining a length of the set of utterances; If the length of the set of utterances exceeds a predetermined threshold of token pieces. Dividing the set of utterances into a plurality of overlapping chunks of token pieces; assigning a label along with a confidence score to each token piece in the chunk; determining a final label and associated confidence score for each chunk of the plurality of overlapping chunks of token pieces by combining two confidence scores, the confidence score from the first chunk and the confidence score from the second chunk; determining a final annotated label for the set of utterances based on the combination of at least the two confidence scores; storing the final annotated labels in a memory.

2. The method of claim 1 , wherein each chunk and its corresponding sequence of labels is treated as a separate example during training.

3. 2. The method of claim 1 , wherein the dividing the set of size N utterances divides the set of size N utterances into (N L ) / (K L ) overlapping chunks, where K is a chunk size and L is an overlap size.

4. The method of claim 1 , wherein the determining an overall score and final annotated label for a token piece is based at least in part on a confidence score from an initially selected chunk of the plurality of overlapping chunks.

5. 2. The method of claim 1, wherein the determination of an overall score for a token piece is based on the position of the token piece in the multiple overlapping chunks of a token piece, and the overall score is a first confidence score from a first chunk if the first chunk is in the first half of the multiple overlapping chunks of a token piece, and a second confidence score from a second chunk if the second chunk is in the second half of the multiple overlapping chunks of a token piece.

6. The method of claim 1 , wherein the determination of an overall score is based on a maximum confidence score of the multiple overlapping chunks.

7. 2. The method of claim 1, wherein the predetermined threshold of token pieces is 512 tokens and the predetermined number of duplicate token pieces is 128 tokens.

8. one or more processors; a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions, when executed by the one or more processors, causing the one or more processors to: receiving, at a data processing system, a set of utterances and training or interfacing with a named entity recognizer to assign a label to each token piece from said set of utterances; determining a length of the set of utterances; if the length of the set of utterances exceeds a predetermined threshold of token pieces; Dividing the set of utterances into a plurality of overlapping chunks of token pieces; assigning a label along with a confidence score to each token piece in the chunk; determining a final label and associated confidence score for each chunk of the overlapping chunks of token pieces by combining two confidence scores, the confidence score from the first chunk and the confidence score from the second chunk; determining a final annotated label for the set of utterances based on the combination of at least the two confidence scores; storing the final annotated labels in a memory.

9. The system of claim 8 , wherein each chunk and its corresponding sequence of labels is treated as a separate example during training.

10. 10. The system of claim 8 or 9, wherein the dividing the set of size N utterances divides the set of size N utterances into (N L ) / (K L ) overlapping chunks, where K is a chunk size and L is an overlap size.

11. 10. The system of claim 8 or 9, wherein the determining of the overall score and final annotated label for a token piece is based at least in part on a confidence score from an initially selected chunk of the plurality of overlapping chunks.

12. 10. The system of claim 8 or 9, wherein the determination of an overall score for a token piece is based on the position of the token piece in the overlapping chunk of token pieces, and the overall score is a first confidence score from a first chunk if the first chunk is in the first half of the overlapping chunk of token pieces, and a second confidence score from a second chunk if the second chunk is in the second half of the overlapping chunk of token pieces.

13. The system of claim 8 or 9, wherein the determination of an overall score is based on a maximum confidence score of the multiple overlapping chunks.

14. 10. The system of claim 8 or 9, wherein the predetermined threshold of token pieces is 512 tokens and the predetermined number of duplicate token pieces is 128 tokens.

15. A program that causes one or more processors to execute the method according to any one of claims 1 to 7.

Images