Systems and techniques for handling long text for pre-trained language models

Abstract

In some aspects, a computing device can receive, at a data processing system, a set of utterances for training a named entity recognizer or for inference with a named entity recognizer to assign a label to each token segment in the set of utterances. The computing device can determine a length of each utterance in the set and, when the length of the utterance exceeds a predetermined threshold of token segments: divide the utterance into a plurality of overlapping token segment chunks; assign a label and a confidence score to each token segment in the token chunk; determine a final label and an associated confidence score for each token segment chunk by merging two confidence scores; determine a final annotated label for the utterance based at least on merging the two confidence scores; and store the final annotated label in a memory.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 750,240, filed May 20, 2022, entitled “System and Technology for Handling Long Text for Pre-Trained Language Models,” which claims priority to co-owned U.S. Provisional Patent Application No. 63 / 282,146, filed November 22, 2021, entitled “System and Technology for Handling Long Text for Pre-Trained Language Models,” the entire contents of which are incorporated herein by reference for various purposes. Technical Field

[0003] This disclosure generally relates to the Chabot system, and more specifically, to techniques for handling long texts in natural language processing using pre-trained language models for the Chabot system. Background Technology

[0004] To achieve immediate responses, many users around the world utilize instant messaging or chat platforms. Organizations frequently use these platforms to engage in real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time can be very expensive for organizations. Chatbots or bots have begun to be developed to simulate conversations with end users, especially via the Internet. End users can interact with bots through messaging applications they have already installed and are using. Intelligent bots (typically powered by artificial intelligence (AI)) can communicate more intelligently and context-sensitively in real-time conversations, thus allowing for more natural conversations between the bot and the end user to improve the conversational experience. Unlike end users who learn a fixed set of keywords or commands to which the bot knows how to respond, intelligent bots can understand the end user's intent based on natural language utterances and respond accordingly.

[0005] Pre-trained language models using natural language processing can be designed to support a maximum text length. For example, for various models, the maximum text length can be limited to 512 wordpiece segments (e.g., 512 subwords). In previous solutions, Chabot entries that exceed the maximum text length would be truncated to be less than the maximum text length. This solution can result in loss of information in the truncated data and inaccurate results. This results in degraded performance of the chatbot and poor customer experience. SUMMARY

[0006] The technology disclosed herein relates generally to chatbots. More specifically and without limitation, the technology disclosed herein relates to techniques for handling long text for pre-trained language models. In various aspects, the pre-trained language models generally support a predetermined maximum number of wordpiece segments. The long text can be divided into overlapping word pieces that have a predefined word piece size and have a predefined number of overlapping wordpiece segments. The word pieces can be individually labeled by named entity recognition (NER). Two scores of the labeled tag for each overlapping wordpiece segment (one score from a first word piece and another score from a second word piece) can be combined to determine a final label for each wordpiece segment in the long text. This can reduce training and inference time while maintaining high performance of the Chabot.

[0007] In some aspects, a method includes receiving, at a data processing system, a set of utterances for training or interacting with a named entity recognizer to assign a label to each wordpiece in one or more utterances; determining a length of the set of utterances; when the length of the set of utterances exceeds a predetermined threshold of wordpiece segments (i.e., a word piece size): dividing the set of utterances into a plurality of overlapping wordpiece word pieces; assigning a label and a confidence score to each wordpiece in the word pieces; determining a final label and an associated confidence score for an overlapping wordpiece by merging two confidence scores, one of the two confidence scores from a first word piece and the other from a second word piece; determining a final labeled tag for the original text input; and storing the label in a memory.

[0008] In some aspects, each word piece and its corresponding sequence of labels are treated as separate examples at training and inference.

[0009] In some aspects, dividing the set of utterances of size N resolves into (N-L) / (K-L) overlapping word pieces. Here, K is the word piece size and L is the overlap size (K>L).

[0010] In some aspects, determining the overall score and label for the wordpiece is based on a confidence score from a first selected word piece of the plurality of word pieces.

[0011] In some aspects, determining the overall score for the token segment is based on a position of the token segment in the overlapping text. If the token segment is in a first half of the overlapping text, the overlap score is the first confidence score from the first wordpiece, and if the token segment is in a second half of the overlapping text, the overlap score is the second confidence score from the second wordpiece.

[0012] In some aspects, determining the overall score is based on a maximum confidence score of the plurality of wordpieces.

[0013] In some aspects, the predetermined number of token segments (i.e., wordpiece size) is 512 token segments, and the predetermined number of overlapping tokens is 128. In some aspects, the predetermined number of token segments is 32, and the predetermined number of overlapping tokens is 8.

[0014] In various aspects, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions which, when executed by the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0015] In various aspects, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and including instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0016] The techniques described above and below can be implemented in a number of ways and in various contexts. Several example implementations and contexts are provided below in greater detail, with reference to the following drawings. However, the following implementations and contexts are merely some of the many. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a simplified block diagram of an environment incorporating example embodiments of Chabot systems in accordance with certain aspects.

[0018] Figure 2 is a simplified block diagram of a computing system implementing a master robot in accordance with certain aspects.

[0019] Figure 3 is a simplified block diagram of a computing system implementing a skill robot in accordance with certain aspects.

[0020] Figure 4 A system for handling long text for a pre-trained language model is shown.

[0021] Figure 5 Converting an example utterance into a plurality of wordpieces is shown.

[0022] Figure 6 Techniques for merge prediction are shown.

[0023] Figure 7 is a flow diagram of techniques for handling long text for pre-trained language models.

[0024] Figure 8 depicted is a simplified diagram of a distributed system for implementing certain aspects.

[0025] Figure 9 is a simplified block diagram of one or more components of a system environment in accordance with certain aspects, through which services provided by one or more components of an embodiment system can be offered as a cloud service.

[0026] Figure 10 depicted is an example computer system that can be used for implementing certain aspects. DETAILED DESCRIPTION

[0027] In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain aspects. However, it will be apparent that various aspects can be practiced without these specific details. The drawings and description are not intended to be restrictive. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0028] Robots and analysis systems

[0029] Robots (also referred to as skills, Chabots, conversational robots, or conversational bots) are computer programs that can perform a conversation with an end user. A robot can typically respond to a natural language message (e.g., a question or a comment) by using a messaging application that uses natural language messages. An enterprise can use one or more robot systems to communicate with end users through a messaging application. The messaging application (which can be referred to as a channel) can be an end user preferred messaging application that the end user already has installed and is familiar with. Thus, to chat with a robot system, an end user does not need to download and install a new application. The messaging application can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or Short Message Service (SMS)), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), mobile or web app extensions that give local or hybrid / responsive mobile apps chat functionality or web application extensions that extend chat functionality or voice-based input (such as devices or apps with interfaces using Siri, Cortana, Google Voice, or other voice input for interaction).

[0030] In some examples, a robot system can be associated with a uniform resource identifier (URI). The URI can use a string of characters to identify the robot system. The URI can be used as a web hook for one or more messaging application systems. The URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from the messaging application system. The HTTP post call messages can be directed to the URI from the messaging application system. In some aspects, the messages can be different from the HTTP post call messages. For example, the robot system can receive messages from an SMS. While the discussion herein can refer to the communications received by the robot system as messages, it should be understood that the messages can be HTTP post call messages, SMS messages, or any other type of communication between the two systems.

[0031] End users can interact with a robotic system through conversational interactions (sometimes referred to as conversational user interfaces (UIs)), just as they would with another human. In some cases, the interaction can include an end user saying "Hello" to the robot and the robot responding with "Hi" and asking how the end user can be helped. In some cases, the interaction can also be a transactional interaction, such as with a banking bot, like transferring money from one account to another; an informational interaction, such as with an HR bot, like querying a vacation balance; or an interaction with a retail bot, like discussing returning a purchased item or seeking technical support.

[0032] In some aspects, a robotic system can intelligently handle end user interactions without interaction with a developer or administrator of the robotic system. For example, an end user can send one or more messages to a robotic system in order to achieve a desired goal. The messages can include some content, such as text, emojis, audio, images, video, or other methods of conveying a message. In some aspects, the robotic system can convert the content into a standardized form (e.g., a Representational State Transfer (REST) call with appropriate parameters for an enterprise service) and generate a natural language response. The robotic system can also prompt the end user for additional input parameters or request other additional information. In some aspects, the robotic system can also initiate a conversation with the end user, rather than passively responding to end user utterances. Described herein are various techniques for identifying explicit invocations of a robotic system and determining inputs for the invoked robotic system. In certain aspects, explicit invocation analysis is performed by a host robot based on detecting an invocation name in an utterance. In response to detecting the invocation name, the utterance can be refined for input to a skill robot associated with the invocation name.

[0033] A conversation with a robot can follow a particular conversation flow that includes a plurality of states. The flow can define what happens next based on input. In some aspects, a robot system can be implemented using a state machine that includes user-defined states (e.g., end user intents) and actions to take in or between states. The conversation can take different paths based on end user input, which can influence decisions made by the robot for the flow. For example, at each state, based on end user input or utterance, the robot can determine the intent of the end user in order to determine the appropriate next action to take. As used herein and in the context of an utterance, the term "intent" refers to the intent of the user providing the utterance. For example, a user can intend to have the robot engage in a conversation for ordering a pizza, such that the intent of the user can be expressed by the utterance "Order pizza." The user intent can relate to a particular task that the user wants the Chabot to perform on behalf of the user. Thus, the utterance can be expressed as a question, command, request, etc. that reflects the intent of the user. The intent can include a goal that the end user wants to accomplish.

[0034] In the context of a configuration of a Chabot, the term "intent" as used herein refers to configuration information for mapping utterances of a user to a particular task / action or a particular kind of task / action that the Chabot can perform. To distinguish the intent of an utterance (i.e., the user intent) from the intent of a Chabot, the latter is sometimes referred to herein as a "robot intent." A robot intent can include a set of one or more utterances associated with the intent. For example, an intent to order a pizza can have various permutations of utterances that express the desire to place an order to purchase a pizza. These associated utterances can be used to train an intent classifier of the Chabot to enable the intent classifier to subsequently determine whether an input utterance from a user matches the order a pizza intent. A robot intent can be associated with one or more dialog flows for initiating a conversation with a user in a certain state. For example, a first message for the order a pizza intent can be the question "What kind of pizza would you like?" In addition to the associated utterances, a robot intent can further include named entities related to the intent. For example, the order a pizza intent can include variables or parameters for performing the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, number of pizzas, etc. The values of the entities are typically obtained through a conversation with the user.

[0035] Figure 1is a simplified block diagram of an environment 100 incorporating a Chabot system according to certain aspects. The environment 100 includes a digital assistant builder platform (DABP) 102 that enables users of the DABP 102 to create and deploy digital assistants or Chabot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or Chabot systems. For example, as shown in Figure 1 the DABP 102 can be used by a bank to create one or more digital assistants for use by customers of the bank. Multiple enterprises can use the same DABP 102 platform to create digital assistants. As another example, an owner of a restaurant (e.g., a pizza restaurant) can use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order a pizza).

[0036] For purposes of the present disclosure, a “digital assistant” is an entity that helps users of the digital assistant accomplish various tasks through natural language conversations. A digital assistant can be implemented using only software (e.g., the digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as Chabot systems. Thus, for purposes of the present disclosure, the terms “digital assistant” and “Chabot system” are interchangeable.

[0037] Digital assistants (such as the digital assistant 106 built using the DABP 102) can be used to perform various tasks via natural language based conversations between the digital assistant and its users 108. As part of the conversations, users can provide one or more user inputs 110 to the digital assistant 106 and obtain returned responses 112 from the digital assistant 106. A conversation can include one or more of the inputs 110 and responses 112. Via these conversations, users can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the tasks requested by the user and respond to the user with appropriate responses.

[0038] User input is typically in the form of natural language and is referred to as an utterance. The user utterance 110 can be in textual form, such as when the user types a sentence, question, piece of text, or even a single word and provides the text as input to the digital assistant 106. In some aspects, the user utterance 110 can be in the form of audio input or speech, such as when the user speaks or says something as input to the digital assistant 106. The utterance is typically in the form of a language spoken by the user 108. For example, the utterance can be in English or some other language. When the utterance is in the form of speech, the speech input is converted to a textual form of the utterance in that particular language and then processed by the digital assistant 106 as a textual utterance. Various speech-to-text processing techniques can be used to convert the speech or audio input to a textual utterance that is then processed by the digital assistant 106. In some aspects, the speech-to-text conversion can be done by the digital assistant 106 itself.

[0039] The utterance, which can be a textual utterance or a speech utterance, can be a piece, a sentence, multiple sentences, one or more words, one or more questions, a combination of the above types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 is configured to perform processing for understanding the meaning of the utterance that involves identifying one or more intents and one or more entities that correspond to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 can perform one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that these utterances are textual utterances that have been directly provided by the user 108 of the digital assistant 106 or are a result of converting input speech utterances to textual form. However, this is not intended to be limiting or constraining in any way.

[0040] For example, the input of the user 108 can request ordering of a pizza by providing an utterance such as "I want to order a pizza." Upon receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. For example, the appropriate action can involve responding to the user with questions that request user input for the type of pizza the user desires to order, the size of the pizza, any toppings for the pizza, etc. The responses provided by the digital assistant 106 can also be in natural language and typically in the same language as the input utterance. As part of generating these responses, the digital assistant 106 can perform natural language generation (NLG). For the user to order the pizza via a conversation between the user and the digital assistant 106, the digital assistant can guide the user to provide all the necessary information for the pizza order and then cause the pizza to be ordered at the end of the conversation. The digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0041] At a conceptual level, the digital assistant 106 performs various processing in response to utterances received from a user. In some aspects, the processing involves a series of processing steps or processing pipeline, including, for example, understanding the meaning of an input utterance (sometimes referred to as natural language understanding (NLU)), determining actions to perform in response to the utterance, causing the actions to be performed where appropriate, generating a response to be output to the user in response to the user's utterance, outputting the response to the user, etc. The NLU processing can include parsing the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better understandable form (e.g., logical form) or structure of the utterance. Generating a response can include using NLG techniques.

[0042] The NLU processing performed by a digital assistant, such as the digital assistant 106, can include various NLP-related processing, such as sentence parsing (e.g., tokenizing, lemmatizing, identifying part-of-speech tags of a sentence, identifying named entities in a sentence, generating dependency trees to represent sentence structure, dividing a sentence into clauses, analyzing individual clauses, resolving references, performing chunking, etc.), etc. In certain aspects, the NLU processing, or portions thereof, is performed by the digital assistant 106 itself. In some other aspects, the digital assistant 106 can use other resources to perform portions of the NLU processing. For example, the syntax and structure of an input utterance sentence can be identified by processing the sentence using a parser, a part-of-speech tagger, and / or a named entity recognizer. In one implementation, for the English language, a parser, a part-of-speech tagger, and a named entity recognizer provided by the Stanford Natural Language Processing (NLP) group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0043] While various examples provided in this disclosure illustrate utterances in the English language, this is meant only as an example. In certain aspects, the digital assistant 106 is also capable of handling utterances in languages other than English. The digital assistant 106 can provide subsystems (e.g., components implementing NLU functionality) that are configured to perform processing for different languages. These subsystems can be implemented as pluggable units that can be invoked from an NLU core server using service calls. This makes the NLU processing flexible and scalable for each language, including allowing different processing orders. Language packs can be provided for individual languages, where the language packs can register a list of subsystems that can provide services from the NLU core server.

[0044] Digital assistants (like the digital assistant 106 depicted in FIG. 1) can be made available or accessible to their users 108 through a variety of different channels, such as but not limited to via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. A single digital assistant can configure itself with several channels, such that a single digital assistant can run on and be accessible through different services at the same time. Figure 1

[0045] Digital assistants or Chabot systems typically include or are associated with one or more skills. In certain aspects, these skills are individual chatbots (referred to as skill bots) that are configured to interact with users and complete specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, querying bank accounts, making appointments, purchasing widgets, etc. For example, for the depicted embodiment, the digital assistant 106 or Chabot system includes a first skill bot 116-1, a second skill bot 116-2, etc. For the purposes of this disclosure, the terms "skill" and "skills" are used synonymously with the terms "skill bot" and "skill bots," respectively. Figure 1

[0046] Each skill associated with a digital assistant helps the user of the digital assistant complete a task through a conversation with the user, where the conversation can include a combination of textual or audio input provided by the user and responses provided by the skill bot. These responses can take the form of textual or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for the user to select.

[0047] ​​There are various ways in which skills or skill bots can be associated with or added to digital assistants. In some instances, a skill bot can be developed by an enterprise and then added to a digital assistant that uses DABP 102. In other instances, a skill bot can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In still other instances, DABP 102 provides an online digital store (referred to as a "skills store") that provides a plurality of skills that relate to a wide variety of tasks. Skills provided through the skills store can also expose various cloud services. To add a skill to a digital assistant generated using DABP 102, a user of DABP 102 can access the skills store via DABP 102, select a desired skill, and instruct that the selected skill be added to a digital assistant created using DABP 102. Skills from the skills store can be added to a digital assistant as-is or in modified form (e.g., a user of DABP 102 can select and copy a particular skill bot provided by the skills store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102).

[0048] A digital assistant or Chabot system can be implemented using a variety of different architectures. For example, in certain aspects, a digital assistant created and deployed using DABP 102 can be implemented using a master bot / secondary (or sub) bot paradigm or architecture. According to this paradigm, a digital assistant is implemented as a master bot that interacts with one or more secondary bots that are skill bots. For example, in the depicted embodiment, digital assistant 106 includes master bot 114 and first skill bot 116-1, second skill bot 116-2, etc. that are secondary bots to master bot 114. In certain aspects, digital assistant 106 itself is considered to function as a master bot. Figure 1

[0049] ​Digital assistants implemented in accordance with a primary-secondary robot architecture enable users of the digital assistant to interact with a variety of skills through a unified user interface (i.e., via the primary robot). When a user interacts with the digital assistant, the primary robot receives the user input. The primary robot then performs processing to determine the meaning of the user input utterance. The primary robot then determines whether the task requested by the user in the utterance can be handled by the primary robot itself, or else the primary robot selects an appropriate skill robot to handle the user request and routes the conversation to the selected skill robot. This enables users to have conversations with the digital assistant through a common single interface and still have the ability to use several skill robots that are configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the primary robot of the digital assistant can interface with skill robots having specific functionality, such as a CRM robot for performing functions related to customer relationship management (CRM), an ERP robot for performing functions related to enterprise resource planning (ERP), an HCM robot for performing functions related to human capital management (HCM), and so on. In this way, the end users or consumers of the digital assistant need only know how to access the digital assistant through the common primary robot interface, while multiple skill robots are provided in the background to handle user requests.

[0050] In certain aspects, in a primary-secondary robot infrastructure, the primary robot is configured to understand a list of available skill robots. The primary robot can access metadata that identifies various available skill robots, and for each skill robot, can access the capabilities of the skill robot, including the tasks that can be performed by the skill robot. After receiving a user request in the form of an utterance, the primary robot is configured to identify or predict, from the plurality of available skill robots, a particular skill robot that can best serve or handle the user request. The primary robot then routes the utterance (or a portion of the utterance) to the particular skill robot for further handling. Thus, control flows from the primary robot to the skill robot. The primary robot can support multiple input channels and output channels. In certain aspects, the routing can be performed with the aid of processing performed by one or more available skill robots. For example, as discussed below, a skill robot can be trained to infer an intent of an utterance and determine whether the inferred intent matches an intent configured for the skill robot. Thus, the routing performed by the primary robot can involve the skill robot communicating an indication to the primary robot that indicates whether the skill robot has been configured with an intent suitable for handling the utterance.

[0051] While Figure 1The embodiments in FIG. 1 illustrate that the digital assistant 106 includes a host robot 114 as well as a first skill robot 116-1, a second skill robot 116-2, and a third skill robot 116-3, but this is not intended to be limiting. The digital assistant can include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems can be implemented in software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), in hardware only, or in implementations that use a combination of software and hardware.

[0052] The DABP 102 provides the infrastructure that enables users of the DABP 102 to create digital assistants (including one or more skill robots associated with the digital assistant) as well as various services and features. In some instances, a skill robot can be created by cloning an existing skill robot, e.g., cloning a skill robot provided by a skill store. As previously noted, the DABP 102 provides a skill store or skill catalog that provides a plurality of skill robots for performing various tasks. A user of the DABP 102 can clone a skill robot from the skill store. The cloned skill robot can be modified or customized as needed. In some other instances, a user of the DABP 102 creates a skill robot from scratch using the tools and services provided by the DABP 102. As previously noted, the skill store or skill catalog provided by the DABP 102 can provide a plurality of skill robots for performing various tasks.

[0053] In certain aspects, at a high level, creating or customizing a skill robot involves the following steps:

[0054] (1) configuring settings for the new skill robot

[0055] (2) configuring one or more intents for the skill robot

[0056] (3) configuring one or more entities for the one or more intents

[0057] (4) training the skill robot

[0058] (5) creating a dialog flow for the skill robot

[0059] (6) adding custom components to the skill robot as needed

[0060] (7) testing and deploying the skill robot

[0061] Each of the above steps is briefly described below.

[0062] (1) Configure settings for the new skill robot - Various settings can be configured for the skill robot. For example, the skill robot designer can specify one or more invocation names for the skill robot being created. Then, the users of the digital assistant can use these invocation names to explicitly invoke the skill robot. For example, the user can input the invocation name in the user's utterance to explicitly invoke the corresponding skill robot.

[0063] (2) Configure one or more intents and associated example utterances for the skill robot - The skill robot designer specifies one or more intents (also referred to as robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent the categories or classifications that the skill robot is trained to infer for an input utterance. After receiving an utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a set of predefined intents used to train the skill robot. Then, the skill robot takes appropriate actions to respond to the utterance based on the inferred intent for the utterance. In some instances, the intents of the skill robot represent the tasks that the skill robot can perform for the users of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skill robot trained for banking, the intents specified for the skill robot can include "Check Balance", "Transfer Money", "Deposit Check", etc.

[0064] For each intent defined for the skill robot, the skill robot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent utterances that a user can input to the skill robot for the intent. For example, for the Check Balance intent, example utterances can include "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account", etc. Thus, various permutations of typical user utterances can be specified as example utterances for the intent.

[0065] These intents and their associated example utterances serve as training data for training the skill bot. Various different training techniques can be used. As a result of this training, a predictive model is generated that is configured to take an utterance as input and output an intent that the predictive model infers for the utterance. In some instances, an input utterance is provided to an intent analysis engine that is configured to use the trained model to predict or infer an intent of the input utterance. The skill bot can then take one or more actions based on the inferred intent.

[0066] (3) One or more intents for the skill bot are configured with entities - in some instances, additional context can be needed to enable the skill bot to properly respond to user utterances. For example, there can be instances where a user input utterance resolves to the same intent in the skill bot. For example, in the above example, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same query balance intent, but these utterances are different requests that request different things. To clarify this request, one or more entities are added to the intent. Using the example of a banking skill bot, an entity called AccountType (which defines values called "checking" and "savings") can enable the skill bot to parse the user request and properly respond. In the above example, while the utterances resolve to the same intent, the values associated with the AccountType entity for the two utterances are different. This enables the skill bot to perform potentially different actions for the two utterances, despite the two utterances resolving to the same intent. One or more entities can be specified for certain intents that are configured for the skill bot. Thus, entities serve to add context to the intent itself. Entities help more fully describe the intent and enable the skill bot to fulfill the user request.

[0067] In certain aspects, there are two types of entities: (a) built-in entities provided by the DABP 102; and (2) custom entities that can be specified by the skill bot designer. The built-in entities are generic entities that can be used with a variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, and the like. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by the skill bot designer to enable various banking transactions by examining keywords entered by the user, such as checking, savings, and credit cards, among others.

[0068] (4) Training the skill bot - The skill bot is configured to receive user input in the form of utterances, parse or otherwise process the received input and identify or select an intent that is relevant to the received user input. As indicated above, to do so, the skill bot must be trained. In certain aspects, the skill bot is trained based on intents configured for the skill bot and example utterances associated with the intents (collectively, training data) such that the skill bot can parse a user input utterance into one of its configured intents. In certain aspects, the skill bot uses a predictive model that is trained using the training data and allows the skill bot to discern what the user is saying (or in some cases, is trying to say). The DABP 102 provides a variety of different training techniques that can be used by the skill bot designer to train the skill bot, including various machine learning based training techniques, rule based training techniques, and / or combinations thereof. In certain aspects, a portion of the training data (e.g., 80%) is used to train the skill bot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes also referred to as the trained skill bot) can be used to handle and respond to user utterances. In certain cases, a user utterance can be a question that only requires a single answer and no further conversation. To handle such cases, a question and answer (Q&A) intent can be defined for the skill bot. This enables the skill bot to output a reply to the user request without having to update the dialog definition. The Q&A intent is created in a similar manner as a regular intent. The dialog flow for a Q&A intent can be different from the dialog flow for a regular intent.

[0069] (5) Creating a Dialog Flow for a Skill Bot - The dialog flow specified for a skill bot describes how the skill bot reacts when resolving different intents of the skill bot in response to received user inputs. The dialog flow defines the operations or actions that the skill bot will take, e.g., how the skill bot responds to user utterances, how the skill bot prompts the user for input, how the skill bot returns data. The dialog flow is like a flowchart that the skill bot follows. The skill bot designer uses a language, such as markdown language, to specify the dialog flow. In some aspects, a YAML version called OBotML can be used to specify the dialog flow for a skill bot. The dialog flow definition for a skill bot acts as a model for the conversation itself, which is a model that enables the skill bot designer to choreograph the interaction between the skill bot and the user that the skill bot serves.

[0070] In some aspects, the dialog flow definition for a skill bot contains the following three parts:

[0071] (a) Context Part

[0072] (b) Default Transitions Part

[0073] (c) States Part

[0074] Context Part - The skill bot designer can define variables used in the conversation flow in the context part. Other variables that can be named in the context part include, but are not limited to, variables for error handling, variables for built-in entities or custom entities, user variables that enable the skill bot to recognize and save user preferences, etc.

[0075] Default Transitions Part - Transitions for a skill bot can be defined in the dialog flow states part or in the default transitions part. Transitions defined in the default transitions part act as a fallback and are triggered when no applicable transition is defined within a state or the conditions required to trigger a state transition are not met. The default transitions part can be used to define routes that allow the skill bot to gracefully handle unexpected user actions.

[0076] States Part - The dialog flow and its related operations are defined as a sequence of temporary states that manage the logic within the dialog flow. Each state node within the dialog flow definition names a component that provides the functionality required at that point in the conversation. Thus, states are built around components. The state contains properties specific to the component and defines transitions to other states that are triggered after the component executes.

[0077] Special case scenarios can be handled using the state section. For example, you might sometimes want to provide a user with the option to do something in a second skill within the digital assistant for which the user is temporarily engaged with a first skill. For example, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some purchase selections), the user can want to jump to a banking skill (e.g., the user can want to make sure he / she has enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action in the first skill can be configured to initiate an interaction with a second, different skill in the same digital assistant and then return to the original flow.

[0078] (6) Adding custom components to the skill bot - As described above, the states specified in the dialog flow of a skill bot name the components that provide the functionality corresponding to the state. The components enable the skill bot to perform the functionality. In some aspects, the DABP 102 provides a set of preconfigured components for performing a wide variety of functionality. A skill bot designer can select one or more of these preconfigured components and associate them with a state in the dialog flow of the skill bot. The skill bot designer can also use the tools provided by the DABP 102 to create custom or new components and associate the custom components with one or more states in the dialog flow of the skill bot.

[0079] (7) Testing and deploying the skill bot - The DABP 102 provides several features that enable a skill bot designer to test the skill bot being developed. The skill bot can then be deployed and included in a digital assistant.

[0080] While the above description describes how to create a skill bot, similar techniques can also be used to create a digital assistant (or host bot). At the host bot or digital assistant level, the digital assistant can be configured with built-in system intents. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the host bot) can handle without invoking a skill bot associated with the digital assistant. Examples of system intents defined for the host bot include: (1) Exit: applicable when a user signals that the user wants to exit the current session or context in the digital assistant; (2) Help: applicable when a user requests help or directions; and (3) Unresolved Intent: applicable to user input that does not closely match the exit intent and the help intent. The digital assistant also stores information about one or more skill bots associated with the digital assistant. This information enables the host bot to select a particular skill bot for handling an utterance.

[0081] At the master robot or digital assistant level, when a user inputs a phrase or utterance to the digital assistant, the digital assistant is configured to perform processing for determining how to route the utterance and related conversation. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance is to be routed to a particular skill for handling, to be handled by the digital assistant or master robot itself according to a built-in system intent, or to be handled as a different state in the current conversation flow.

[0082] In certain aspects, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skill robot using its invocation name. If an invocation name is present in the user input, the invocation name is considered an explicit invocation of the skill robot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill robot for further handling. In certain aspects, if there is no particular invocation or explicit invocation, the digital assistant evaluates the received user input utterance and computes confidence scores for system intents and skill robots associated with the digital assistant. The scores computed for skill robots or system intents represent how likely the user input represents a task that the skill robot is configured to perform or represents a system intent. Any system intent or skill robot whose associated computed confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a particular system intent or skill robot from the identified candidates for further handling of the user input utterance. In certain aspects, after one or more skill robots are identified as candidates, the intents associated with those candidate skills are evaluated (according to each skill’s intent model) and a confidence score is determined for each intent. Any intent whose confidence score exceeds a threshold (e.g., 70%) is generally considered a candidate intent. If a particular skill robot is selected, the user utterance is routed to that skill robot for further processing. If a system intent is selected, one or more actions are performed by the master robot itself according to the selected system intent.

[0083] Figure 2 is a simplified block diagram of a master robot (MB) system 200 according to certain aspects. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multi-intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and a data store 250. Figure 2The depicted MB system 200 is merely an example of a component arrangement in a host robot. Those of ordinary skill in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 can have more or fewer systems or components than those shown, can combine two or more subsystems, or can have a different subsystem configuration or arrangement. Figure 2

[0084] The pre-processing subsystem 210 receives an utterance“A” 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in various ways including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 can use a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio to text.

[0085] The language detector 212 detects the language of the utterance 202 based on the text of the utterance 202. The way the utterance 202 is handled depends on the language, as each language has its own grammar and semantics. Differences between languages are taken into account when analyzing the syntax and structure of the utterance.

[0086] The language parser 214 performs a grammatical analysis of the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language parser 214 can also tokenize (e.g., convert each word to an individual token) and lemmatize the linguistic units of the utterance 202. A lemma is the principal form of a set of words as represented in a dictionary (e.g., “run” is the lemma for run, runs, ran, running, etc.). Other types of pre-processing that the language parser 214 can perform include chunking of compound expressions, e.g., combining “credit” and “card” into a single expression “credit card.” The language parser 214 can also identify relationships between words in the utterance 202. For example, in some aspects, the language parser 214 generates a dependency tree that indicates which part of the utterance (e.g., a particular noun) is a direct object, which part of the utterance is a preposition, etc. The result of the processing performed by the language parser 214 forms extracted information 205 and is provided as input to the MIS 220 along with the utterance 202 itself. The pre-processing system 210 can include a named entity recognizer 216 that can be used to identify certain utterances 202 or portions thereof.​

[0087] As indicated above, the utterance 202 can include more than one sentence. For the purposes of detecting multiple intents and explicit invocation, the utterance 202 can be treated as a single unit, even if it includes multiple sentences. However, in certain aspects, preprocessing can be performed, e.g., by the preprocessing subsystem 210, to identify individual sentences in the multiple sentences for multiple intent analysis and explicit invocation analysis. Generally, the results produced by the MIS 220 and the EIS 230 are substantially the same whether the utterance 202 is processed at the level of individual sentences or as a single unit including multiple sentences.

[0088] The MIS 220 determines whether the utterance 202 represents multiple intents. Although the MIS 220 can detect that there are multiple intents in the utterance 202, the processing performed by the MIS 220 does not involve determining whether the intents of the utterance 202 match any of the intents that have been configured for the robots. Rather, the processing for determining whether the intents of the utterance 202 match the intents of the robots can be performed by the intent classifier 242 of the MB system 200 or the intent classifier of the skill robot (e.g., as shown in the embodiments of FIGS. 2A and 2B). The processing performed by the MIS 220 assumes that there is a robot (e.g., a particular skill robot or the host robot itself) that can handle the utterance 202. Thus, the processing performed by the MIS 220 does not need to know which robots are in the Chabot system (e.g., the identities of the skill robots registered with the host robot) nor does it need to know what intents have been configured for a particular robot. Figure 3

[0089] To determine that the utterance 202 includes multiple intents, the MIS 220 applies one or more rules of a set of rules 252 in the data store 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and can include sentence patterns that indicate the presence of multiple intents. For example, a sentence pattern can include a coordinating conjunction that connects two parts of a sentence (e.g., a conjunction), where the two parts correspond to different intents. If the utterance 202 matches the sentence pattern, then it can be inferred that the utterance 202 represents multiple intents. It should be noted that an utterance with multiple intents does not necessarily have different intents (e.g., intents that involve different robots or different intents within the same robot). Rather, the utterance can have different instances of the same intent, e.g., “Place a pizza order using payment account X, then place a pizza order using payment account Y.”

[0090] ​As part of determining that discourse 202 represents multiple intentions, MIS 220 also determines which parts of discourse 202 are associated with each intention. MIS 220 constructs a new discourse for each intention represented in a discourse containing multiple intentions, replacing the original discourse for individual processing, for example, such as... Figure 2 The original utterance 202 can be broken down into two or more separate utterances, one at a time, as depicted in the diagram. The MIS 220 uses the extracted information 205 and / or analysis of the utterance 202 itself to determine which of the two or more utterances should be processed first. For example, the MIS 220 may determine that utterance 202 contains markers indicating a specific intention that should be processed first. The newly formed utterance corresponding to that specific intention (e.g., one of utterances 206 or 208) will be sent first for further processing by the EIS 230. After the session triggered by the first utterance has ended (or been temporarily paused), the next highest priority utterance (e.g., another utterance of utterances 206 or 208) can then be sent to the EIS 230 for processing.

[0091] EIS 230 determines whether a received utterance (e.g., utterance 206 or utterance 208) contains a call name for a skill robot. In some respects, each skill robot in the Chabot system is assigned a unique call name that distinguishes it from other skill robots in the Chabot system. A list of call names may be stored in data storage 250 as part of skill robot information 254. When a utterance contains a word that matches a call name, the utterance is considered an explicit call. If a robot is not explicitly called, the utterance received by EIS 230 is considered a non-explicit call utterance 234 and is fed into the master robot's intent classifier (e.g., intent classifier 242) to determine which robot to use to handle the utterance. In some instances, intent classifier 242 will determine that the master robot should handle non-explicit call utterances. In other instances, intent classifier 242 will determine which skill robot to route the utterance to for handling.

[0092] The explicit invocation functionality provided by EIS230 offers several advantages. It reduces the amount of processing the master robot must perform. For example, when explicit invocation is present, the master robot may not need to perform any intent classification analysis (e.g., using intent classifier 242), or may need to perform a simplified intent classification analysis to select the skill robot. Therefore, explicit invocation analysis enables the selection of a specific skill robot without resorting to intent classification analysis.

[0093] Moreover, there can be cases of functional overlap between multiple skill robots. This can occur, for example, if the intents handled by two skill robots overlap or are very close to each other. In this case, it can be difficult for the host robot to identify which of the multiple skill robots to select based on intent classification analysis alone. In this scenario, the explicit invocation makes it unambiguous as to which particular skill robot to use.

[0094] In addition to determining that the utterance is an explicit invocation, the EIS 230 is also responsible for determining whether any portion of the utterance should be used as input to the skill robot that was explicitly invoked. Specifically, the EIS 230 can determine whether a portion of the utterance is irrelevant to the invocation. The EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. The EIS 230 can send the portion of the utterance that is irrelevant to the invoked skill robot instead of sending the entire utterance as received by the EIS 230. In some instances, the input to the invoked skill robot is simply formed by deleting any portion of the utterance that is associated with the invocation. For example, “I want to order pizza using Pizza Bot” can be shortened to “I want to order pizza” because “using Pizza Bot” is relevant to invoking the Pizza Bot, but is irrelevant to any processing to be performed by the Pizza Bot. In some instances, the EIS 230 can reformat the portion to be sent to the invoked robot, e.g., to form a complete sentence. Thus, the EIS 230 not only determines that there is an explicit invocation, but also determines what to send to the skill robot when there is an explicit invocation. In some instances, there can be no text that can be input to the invoked robot. For example, if the utterance is “Pizza Bot,” the EIS 230 can determine that the Pizza Bot is being invoked, but there is no text to be processed by the Pizza Bot. In this scenario, the EIS 230 can indicate to the skill robot invoker 240 that there is nothing to send.

[0095] The skill robot invoker 240 invokes skill robots in various ways. For example, the skill robot invoker 240 can invoke a robot in response to receiving an indication 235 that a particular skill robot has been selected as a result of an explicit invocation. The indication 235 can be sent by the EIS 230 along with the input for the explicitly invoked skill robot. In this scenario, the skill robot invoker 240 hands over control of the conversation to the explicitly invoked skill robot. The explicitly invoked skill robot will determine an appropriate response to the input from the EIS 230 by treating the input as a standalone utterance. For example, the response can be to perform a particular action or to begin a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0096] Another way in which the skill robot invoker 240 can invoke skill robots is through implicit invocation using the intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform. The intent classifier 242 is trained on different classifications, one for each skill robot. For example, whenever a new skill robot is registered with the host robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. The parameters that result from this training (e.g., a set of parameter values for a machine learning model) can be stored as part of the skill robot information 254.

[0097] In certain aspects, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training of the machine learning model can involve at least inputting a subset of utterances from example utterances associated with various skill robots to generate, as output of the machine learning model, an inference as to which robot is the correct robot for handling any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the difference between the generated inference and the ground truth information.

[0098] In certain aspects, the intent classifier 242 determines, for each skill robot registered with the host robot, a confidence score indicating the likelihood that the skill robot can handle the utterance (e.g., the non-explicit invocation utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each system-level intent (e.g., help, exit) that has been configured. If a particular confidence score satisfies one or more conditions, the skill robot invoker 240 will invoke the robot associated with the particular confidence score. For example, it can be required that a threshold confidence score value be satisfied. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skill robot. In some aspects, in addition to satisfying a threshold confidence score value, the confidence score must also exceed the next highest confidence score by a certain margin. Such a condition will enable routing to a particular skill robot when multiple skill robots have confidence scores that all exceed the threshold confidence score value.

[0099] After a robot is identified based on the evaluation of the confidence scores, the skill robot invoker 240 hands off processing to the identified robot. In the case of a system intent, the identified robot is the host robot. Otherwise, the identified robot is a skill robot. Further, the skill robot invoker 240 will determine what to provide as input 247 to the identified robot. As explained above, in the case of an explicit invocation, the input 247 can be based on the portion of the utterance that is not associated with the invocation, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invocation, the input 247 can be the entire utterance.

[0100] The data store 250 includes one or more computing devices that store data used by the various subsystems of the host robot system 200. As explained above, the data store 250 includes the rules 252 and the skill robot information 254. The rules 252 include, for example, rules for determining by the MIS 220 when an utterance represents multiple intents and how to split an utterance that represents multiple intents. The rules 252 further include rules for determining by the EIS 230 which portions of an utterance that display invokes a skill robot to send to the skill robot. The skill robot information 254 includes invocation names of skill robots in the Chabot system, e.g., a list of invocation names of all skill robots registered with a particular host robot. The skill robot information 254 can also include information used by the intent classifier 242 to determine a confidence score for each skill robot in the Chabot system, e.g., parameters of a machine learning model.

[0101] Figure 3This is a simplified block diagram of a skill robot system 300 based on certain aspects. The skill robot system 300 is a computing system that can be implemented solely in software, solely in hardware, or a combination of hardware and software. In some aspects, such as... Figure 1 In the depicted embodiments, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0102] The skill robot system 300 includes a MIS 310, an intent classifier 320, and a session manager 330. The MIS 310 is similar to... Figure 2 The MIS 220 provides similar functionality, including the ability to operatively use rule 352 in data storage 350 to determine: (1) whether a utterance represents multiple intentions, and if so, (2) how to split the utterance into separate utterances for each of the multiple intentions. In some respects, the rules applied by the MIS 310 for detecting multiple intentions and for splitting utterances are the same as those applied by the MIS 220. The MIS 310 receives utterance 302 and extracted information 304. The extracted information 304 is similar to... Figure 1 The information extracted from 205 can be generated using a language parser 214 or a language parser native to the skill robot system 300.

[0103] The intent classifier 320 can be combined with the above. Figure 2 The intent classifier 242 discussed in the embodiments is trained in a similar manner and is described in further detail herein. For example, in some aspects, the intent classifier 320 is implemented using a machine learning model. For a specific skill robot, the machine learning model of the intent classifier 320 is trained using at least a subset of example utterances associated with that specific skill robot as training utterances. The underlying fact of each training utterance will be the specific robot intent associated with the training utterance.

[0104] Phras 302 can be received directly from the user or provided through the main bot. When phrasing 302 is provided through the main bot, for example, as via... Figure 2As a result of the processing by MIS 220 and EIS 230 in the depicted embodiment, MIS 310 can be bypassed to avoid duplicating processing already performed by MIS 220. However, if an utterance 302 is received directly from a user, e.g., during a conversation that occurs after routing to the skill bot, then MIS 310 can process the utterance 302 to determine whether the utterance 302 represents multiple intents. If so, then MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, e.g., utterance "D" 306 and utterance "E" 308. If the utterance 302 does not represent multiple intents, then MIS 310 forwards the utterance 302 to intent classifier 320 for intent classification without splitting the utterance 302.

[0105] Intent classifier 320 is configured to match a received utterance, e.g., utterance 306 or 308, to an intent associated with skill bot system 300. As explained above, a skill bot can be configured with one or more intents, each intent including at least one example utterance associated with the intent and used to train the classifier. In the depicted embodiment, intent classifier 320 is trained to determine a confidence score for each intent associated with skill bot system 300. Figure 2 In the depicted embodiment, intent classifier 242 of host bot system 200 is trained to determine confidence scores for individual skill bots and for system intents. Similarly, intent classifier 320 can be trained to determine a confidence score for each intent associated with skill bot system 300. The classification performed by intent classifier 242 is at the bot level, while the classification performed by intent classifier 320 is at the intent level and is thus more granular. Intent classifier 320 can access intent information 354. For each intent associated with skill bot system 300, intent information 354 includes a list of utterances that represent the intent and illustrate the meaning of the intent and are generally associated with tasks that can be performed by the intent. Intent information 354 can further include parameters that result from training on the list of utterances.

[0106] Session manager 330 receives as output from intent classifier 320 an indication 322 of a particular intent that is identified by intent classifier 320 as the best match to the utterance input to intent classifier 320. In some instances, intent classifier 320 is unable to determine any match. For example, if the utterance relates to a system intent or an intent of a different skill bot, then the confidence score computed by intent classifier 320 can be below a threshold confidence score value. When this occurs, skill bot system 300 can submit the utterance to the host bot for disposition, e.g., to route to a different skill bot. However, if intent classifier 320 successfully identifies an intent within the skill bot, then session manager 330 will initiate a conversation with the user.

[0107] The session initiated by the session manager 330 is a session specific to the intent identified by the intent classifier 320. For example, the session manager 330 can be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine can include a default start state (e.g., when the intent is invoked without any additional input) and one or more additional states, where each state is associated with an action to be performed by the skill bot (e.g., execute a purchase transaction) and / or a dialog (e.g., questions, responses) to be presented to the user. Thus, the session manager 330 can determine the action / dialog 335 upon receiving the indication 322 that the intent was identified, and can determine additional actions or dialogs in response to subsequent utterances received during the session.

[0108] The data store 350 includes one or more computing devices that store data used by the various subsystems of the skill bot system 300. As Figure 3 depicted, the data store 350 includes rules 352 and intent information 354. In certain aspects, the data store 350 can be integrated into the data store of the host robot or digital assistant, such as the data store 250 in Figure 2 FIG. 1.

[0109] Chabot can perform the task of named entity recognition (NER). NER can refer to the task of finding spans of text that constitute a proper name entity and labeling the entity type. Person, location, organization, and geopolitical entity are four common types of named entities. However, the term "named entity" can refer to things like dates, times, numbers, currency, etc. Here is an example of the output of our NER system:

[0110] [PERSON David Smith] paid [CURRENCY $95] on [DATE Sep 12th] at the [MERCHANT Palm Court Restaurant] in [LOCATION New York].

[0111] NER is a key NLP task that requires text analysis to understand relevant semantics and sentiment. For example, knowing that a named entity like "Sydney" is a place, a person, or a name of a university is important for many natural language understanding tasks. Some important tasks that benefit from NER are described as follows. In question answering, NER can be used to identify the text span that constitutes the answer. For example, based on the above example, the answer to the following question is "$95." How much did David Smith pay at the restaurant? In aspect-based sentiment analysis, NER can identify entities as an initial step to understanding the sentiment of a customer towards an entity. For example, in the following sentence, there can be both positive and negative opinions about the "Nikon camera" entity and the "Canon camera" entity in the following example statement.

[0112] The Nikon camera is amazing; it is better than the Canon camera.

[0113] NER can be one of the technologies used in digital assistants. NER can be used in digital assistants to locate the user's speech and classify it into pre-defined categories like PER (which can be an abbreviation for PERSON), date and time. The digital assistant can use this type of information to process the user's request. For example, to generate a cost report from the following sentence, the digital assistant can use a NER model to identify SFO, $10, and May 21st as MER (which can be an abbreviation for MERCHANT) entities, CUR (which can be an abbreviation for CURRENCY) entities, and DATE entities. LOC can be an abbreviation for the class identifier LOCATION.

[0114] [MER SFO] charged [CUR $10] for parking on [DATE May 21st].

[0115] NER is not a simple task due to the difficulties caused by segmentation ambiguity and type ambiguity. Segmentation ambiguity stems from the complexity of finding entities and their boundaries within a sentence. For example, multiple words in a sentence may represent a single entity. As shown below, "New York Times" is a single entity composed of 3 words (with MER tags):

[0116] Change this year's transaction amount for [MER New York Times] to $20.

[0117] Furthermore, the same word can refer to multiple entity types, and NER needs to resolve this type of ambiguity based on the context. For example, "Sydney" can refer to a person or place in the following sentence.

[0118] Late on Monday afternoon, [PER Sydney] texted us.

[0119] This is the best view in [LOC Sydney].

[0120] Understanding classifiers can help in understanding how NER algorithms work. A classifier can be a system that learns to determine a category or label from a predefined, finite set of categories given a set of inputs. As a common example of a text classifier, sentiment analysis enables the identification of the sentiment and tendency of an opinion expressed in a sentence (e.g., a product review) as positive, negative, or neutral.

[0121] In contrast, sequence labeling classifies each word in a text (i.e., assigns a label to each word); and NER as a sequence labeling task can include recognizing and labeling sub-sequences of words. To this end, the Begin (Begins), Inside (Inside), and Outside (Outside) (BIO) tagging scheme can be a way to use a sequence labeler to recognize sub-sequences, where the labels capture both the boundaries and the types of named entities. In this technique, any word unit that begins an interesting span can be labeled with the label B, any word unit that occurs within a span is labeled with the label I, and any word unit that is outside of any interesting span is labeled as O. While there can be only one O label, there can be different B and I labels for each named entity class (e.g., I-DATE and I-MER). The following example shows how BIO encoding defines the boundaries between two adjacent named entities of the same type (i.e., [MER Plaza Hotel's] and [MER Palm Court Resturant]). Table 1 illustrates an example classification of an utterance.

[0122] Text: BIO tags: David B-PER Smith I-PER paid O $95 B-CUR on O Sep B-DATE 12th I-DATE at O the O Plaza B-MER Hotel′s I-MER Palm B-MER Court I-MER Restaurant I-MER in O New B-LOC York I-LOC

[0123] Table 1

[0124] Smart assistant NER can incorporate the following state-of-the-art techniques.

[0125] Deep learning refers to neural networks composed of multiple processing layers that automatically learn representations at different levels of abstraction. Each layer learns to transform its input data into a slightly more abstract and complex representation, so that each layer implements feature learning and eliminates the need for rich handcrafted features. For this reason, deep neural networks can be the right tool for solving complex problems provided with enough data. Deep learning discovers complex structures in large datasets and incorporates longer-range information in the input, significantly improving the state-of-the-art in natural language processing, including NER tasks.

[0126] Pre-training refers to the process of training a network on large external datasets such as Wikipedia and Common Crawl (an open repository of freely available data obtained by crawling the web). The intuition behind pre-training is that if a network is trained on a dataset that is large enough and general enough, the network will effectively serve as a universal model of the world of text. Using a pre-trained network, one can initialize the backbone of a new network with the knowledge gained in solving one problem (e.g., language modeling) to solve a different but related problem (e.g., text classification) without starting from scratch (a.k.a. transfer learning).

[0127] Sequence labeling models (e.g., NER) can result in inconsistencies between adjacent labels. For example, in the following sentence, the model labels “Sydney” as B-PER and “Harbour” as I-LOC, which is not a valid sequence because a label boundary cannot start with an I tag. Thus, the correct sequence would be B-LOC and I-LOC. For this reason, a conditional random field (CRF) is used to guarantee that adjacent labels are consistent. Table 3 illustrates an exemplary labeling of a portion of a sentence.

[0128] Text: I walked over the Sydney Harbour Bridge BIO tags: O O O O B-LOC I-LOC I-LOC

[0129] Table 2

[0130] To this end, the CRF learns tag-tag weights at training time and avoids generating impossible BIO label sequences by assigning very negative weights to these sequences (e.g., the X values in Table 3 below).

[0131] O B-LOC B-PER I-LOC I-PER O X X B-LOC X B-PER X I-LOC X I-PER X

[0132] Table 3

[0133] Deep learning models with a large number of parameters can easily overfit to the training data. This has the effect that the model learns the noise in the training data, which leads to poor performance when the model is evaluated on new data. In addressing this problem, dropout can be a computationally inexpensive and remarkably effective technique. The key idea is to randomly ignore or drop some of the layer output units in the model during training to prevent the units from over-adapting to each other (i.e., overfitting).

[0134] Deep learning models require different constraints and capabilities to generalize well to different patterns of data. These constraints are controlled by a number of measures called hyperparameters, which must be tuned so that the model can solve the problem in the best way. Hyperparameter tuning refers to the process of finding a set of optimal hyperparameters for a learning algorithm.

[0135] The disclosed NER system can include several model improvements, training improvements, and data improvements.

[0136] For model improvements, the disclosed NER system can incorporate contextual and gazetteer features. The disclosed NER model can be a hybrid model that incorporates contextual and gazetteer features. The new approach incorporates contextual features and an external knowledge resource called a gazetteer to improve model performance. A gazetteer can be a list of named entities such as organizations, countries, cities, and people names that are matched to unstructured text in order to provide additional features to the model. For example:

[0137] Value Entity Type Baton Rouge Business Burgerville Business Ramen Street Business Slim Chickens Business Ribs & Burgers Business Pizza Hut Business Max & Erma's Business

[0138] Table 4

[0139] By leveraging external knowledge, the disclosed NER model reduces the dependence on collecting annotated data, which can be extremely expensive and laborious.

[0140] The disclosed NER system can include fixed CRF tag-tag transitions on small training datasets. On large training datasets, the CRF learns inconsistent tag pairs like O I-LOC I-PER that never occur in the training data and thus assigns very negative weights for these tag-tag transitions. However, on small training datasets, the CRF sometimes learns models that generate these inconsistent tag-tag transitions. Thus, by modifying the corresponding weights after training by setting the weights of inconsistent transitions to very negative values, the disclosed NER system can introduce a new technique to prevent the CRF from finding inconsistent tag-tag transitions. This technique guarantees consistent tag-tag transitions even with less training data.

[0141] To improve training, the disclosed NER system can include selective dropout. A new technique called selective dropout can apply a higher dropout rate to entity tokens than to non-entity tokens. This can force the model to focus on contextual information during training and make the model more reliable and robust for different values of each entity type. For example: (The blue part is the entity value.)

[0142] Training example: Please transfer $100 to my saving account.

[0143] Test example: Would you please transfer to my banking account?

[0144] The disclosed NER system can include extensive hyperparameter tuning. The hyper tuning framework can encompass a wide range of values. This enables the disclosed NER system to run extensive hyperparameter tuning to identify the best choices of hyperparameters, resulting in high-quality results.

[0145] The disclosed NER system can include data improvement. One data improvement can include an overlapping chunking mechanism. A limitation of prior art models is that their memory and compute requirements grow quadratically with the length of the input sequence. Given the limitations of universally available hardware, current pre-trained language models can only process input sequences with up to 512 wordpieces. To mitigate this limitation, the disclosed system can use a mechanism for decomposing long text into overlapping word chunks, where each word chunk and its corresponding label are treated as a separate example at training time. For evaluation / inference, the predictions from word chunks of the same input text must be merged. This mechanism enables the model to serve larger sequences and significantly reduce training and inference time while maintaining high performance.

[0146] Figure 4 A system 400 for handling long text for a pre-trained language model is illustrated. A length determination engine 410 can receive an utterance 402. The length determination engine can determine whether the length exceeds a predetermined threshold of wordpieces. A wordpiece can be a word, a portion of a word, or a punctuation mark.

[0147] If the length exceeds the predetermined threshold of wordpiece segments, a chunking engine 420 can decompose the utterance into a plurality of overlapping word chunks. For example, the chunking engine 420 can divide the utterance 402 into word chunk A 422, word chunk B 424, and word chunk C 426. While three word chunks are illustrated, the disclosed technology is not so limited and is applicable to any number of word chunks. For example, as few as two word chunks can be processed depending on the capabilities of the processing system.

[0148] A named entity recognizer 430 can determine a classifier or label for each word chunk. For example, the named entity recognizer 430 can determine label A 432 for word chunk A 422, label B 434 for word chunk B 424, and label C 436 for word chunk C 426. As described above, the named entity recognizer 430 can assign a label to each wordpiece in a word chunk. The named entity recognizer can also assign a label to each word chunk in a plurality of word chunks. A scoring engine 440 can receive label A 432 for word chunk A 422, label B 434 for word chunk B 424, and label C 436 for word chunk C 426. Each label can have an accompanying word chunk score. The scoring engine 440 can compute a final label and associated confidence score for each word chunk in an overlapping wordpiece segment word chunk by merging two confidence scores, one from a first word chunk and the other from a second word chunk. The scoring engine 440 can determine a final annotated label for the set of utterances based on at least merging the two confidence scores. The scoring engine 440 can store the final annotated score 442 in memory.

[0149] Figure 5FIGURE illustrates converting an example utterance 502 into multiple word chunks. The example utterance 502 can be divided into multiple word chunks, where each word chunk is a particular number of words. As shown, Figure 5 The example utterance 502 can be divided into segments of a predetermined length that include multiple words. In some aspects, the predetermined length can be 32 words. As an example, the utterance 502 can be divided into word chunk A 504, which can include indices 1-10 for an example word chunk size of 10 word units. Word chunk B can include indices 6-15 for an example word chunk size of 10 word units. As shown, Figure 4 The named entity recognizer 430 can process smaller word chunks more efficiently. For example, the entity recognizer 430 processes the 10-word segment faster than processing the entire example utterance 502.

[0150] The named entity recognizer 430 can analyze the utterance 502 and determine a label or tag for each word unit. For example, the first word unit of the utterance 502 is the word “My” at index 1, and the named entity recognizer 430 can assign the label “O” for that word unit, which means that the word unit is out of scope. Similarly, the named entity recognizer 430 can assign the labels “O” for the word units “name,” “is,” “,” “living,” “in,” “and,” “working,” “for.” The named entity recognizer 430 can assign the label “B-PER” for the word unit “Davis” at index 4, which means that it is the beginning of a person name. The named entity recognizer 430 can assign the label “B-LOC” for the word unit “Brisbane” at index 8, which means that it is the beginning of a location. The named entity recognizer 430 can assign the label “B-LOC” for the word unit “Australia” at index 10, which means that it is the beginning of a location. The named entity recognizer 430 can assign the label “B-ORG” for the word unit “Oracle” at index 14, which means that it is the beginning of an organization. The named entity recognizer 430 can assign the label for the word unit “Corp” at index 15 that indicates within an organization name span.

[0151] As an example, Figure 5 FIGURE illustrates a word chunk size of 10 word units, where the overlap size is 5 word chunks. For example, the first overlap portion 508 can include the second half of the word chunk A 504. The second overlap portion 510 can include the first half of the word chunk B 506.

[0152] Figure 5Further techniques for merging predictions from word chunks are illustrated. For example, during analysis of word chunk A, the named entity recognizer 430 can correctly label "Brisbane" at index 8, but miss "Australia" at index 10. During analysis of word chunk B, the named entity recognizer 430 can correctly label "Australia" at index 10, but miss "Brisbane" at index 8. When merging the analysis outputs from word chunk A and word chunk B, however, both "Brisbane" and "Australia" are correctly labeled.

[0153] Figure 6 Techniques for merging predictions are illustrated. As shown, the score engine 440 can determine a confidence score for each label from the named entity recognizer 430. For example, the score engine 440 can determine a confidence score for indices 6, 7, 8, and 9 that indicates a confidence of 0.9 in the assigned label, which represents a 90% confidence that the label is correct. For index 10 of word chunk A, the score engine 440 can determine a confidence score of 0.5 for the assigned label "0," which represents a 50% confidence that the label is correct. Figure 4

[0154] For word chunk B, the score engine 440 can determine a confidence score for indices 6, 7, 9, and 10 that indicates a confidence of 0.9 in the assigned label, which represents a 90% confidence that the label is correct. For index 8 of word chunk B, the score engine 440 can determine a confidence score of 0.5 for the assigned label "0," which represents a 50% confidence that the label is correct.

[0155] For the merged output, there can be several possible outcomes. For example, the output strategy can include "half," "first," "second," and "max." For the "half" strategy, the score engine 440 can use 1 / 2 of the prediction from the first word chunk and 1 / 2 of the prediction from the second word chunk. For the "first" strategy, the score engine 440 can use the prediction from the first word chunk. For the "second" strategy, the score engine 440 can use the prediction from the second word chunk. For the "max" strategy, the score engine 440 can decide the prediction based on the maximum confidence score.

[0156] As shown, the score engine 440 can determine a confidence score for each label from the named entity recognizer 430. For example, the score engine 440 can determine a confidence score for indices 6, 7, 8, and 9 that indicates a confidence of 0.9 in the assigned label, which represents a 90% confidence that the label is correct. For index 10 of word chunk A, the score engine 440 can determine a confidence score of 0.5 for the assigned label "0," which represents a 50% confidence that the label is correct. Figure 6 ​As shown, the “half” strategy can provide accurate results because the label for index 8 of “Brisbane” and the label for index 10 of “Australia” are both accurate. The “first” strategy correctly identifies the label for index 8 of “Brisbane” but incorrectly labels the token segment of “Australia.” The “second” strategy incorrectly identifies the label for index 8 of “Brisbane” but correctly labels the token segment of “Australia.” The “max” strategy can provide accurate results because the label for index 8 of “Brisbane” and the label for index 10 of “Australia” are both accurate.

[0157] In some aspects, the predetermined length can be 32 token segments. As shown, Figure 6 As shown, each word chunk can be broken down into a plurality of elements. For example, each word chunk can provide an overlap between word chunks (which can be abbreviated as l). The amount of overlap between word chunks can be predetermined. In an example, for a word chunk of 32 token segments, the overlap can be 16 token segments (subwords).

[0158] Figure 7 is a flow diagram of an example process 700 associated with systems and techniques for handling long text for pre-trained language models. In some implementations, Figure 7 One or more process blocks of the process 700 can be performed by a computing device (e.g., the computing device 1000). In some implementations, Figure 7 One or more process blocks of the process 700 can be performed by another device or a group of devices separate from or including the computing device. Additionally or alternatively, Figure 7 One or more process blocks of the process 700 can be performed by one or more components of the device 1000, such as the processing subsystem 1004, the storage subsystem 1018, the I / O subsystem 1008, the communication subsystem 1024, and / or the bus subsystem 1002.

[0159] As shown, Figure 7 The process 700 can include receiving, at a data processing system, a set of utterances for training or interacting with a named entity recognizer to assign a label to each token segment in the set of utterances (block 710). For example, as described above, a computing device can receive, at a data processing system, a set of utterances for training or interacting with a named entity recognizer to assign a label to each token segment in the set of utterances. In aspects, the set of utterances can be received using a keyboard through user input via a Chabot. In aspects, the set of utterances can be received using a microphone through user audibly input via a Chabot. The system can transcribe the spoken utterances into text entries.

[0160] As Figure 7 Further, process 700 can include determining a length of the set of utterances (block 720). For example, as described above, the computing device can determine a length of the set of utterances. Process 700 can determine the length of the set of utterances by dividing the utterances into a plurality of token segments. A token segment can be a word, a portion of a word, or a punctuation mark. Complex words can be broken into one or more token segments. The length can be a number of token segments in the utterance.

[0161] As Figure 7 Further, process 700 can include dividing the set of utterances into a plurality of overlapping token segment chunks when the length of the set of utterances exceeds a predetermined threshold of token segments (block 730). The length of the utterances can be compared to the predetermined threshold. The predetermined threshold can vary as desired. In various aspects, the predetermined threshold can be 512 token segments. Those of skill in the art will appreciate that other thresholds will be within the scope of the present disclosure.

[0162] As Figure 7 Further, process 700 can include assigning a label and a confidence score to each token segment in the chunk (block 740). As described above, each token segment in the utterance can be assigned a label. The named entity recognizer 430 can identify and label subsequences of words. In various aspects, a Begin, Inside, and Outside (BIO) tagging scheme can be used. The BIO tagging scheme can identify subsequences in which the label captures both the boundaries and the type of the named entity. In this technique, any token that begins an interesting span can be labeled with the tag B, tokens that occur within a span are labeled with the tag I, and any tokens outside of any interesting span are labeled as O. While there can be only one O label, there can be different B and I labels for each named entity class (e.g., I-DATE and I-MERCHANT). Each label can be assigned a confidence score that indicates a confidence that the label is correct.

[0163] As Figure 7 Further, process 700 can include determining a final label and an associated confidence score for each of the overlapping token segment chunks by merging two confidence scores, one of which is from a first chunk and the other of which is from a second chunk (block 750).

[0164] As Figure 7 Further, process 700 can include determining a final annotated label for the set of utterances based at least on merging the two confidence scores (block 760).

[0165] As Figure 7Further, process 700 can include storing the final annotated label in memory (block 770). Process 700 can utilize several different strategies to determine the final annotated label. For example, output strategies can include "half," "first," "second," and "max." For the "half" strategy, scoring engine 440 can use ½ of the prediction from the first word chunk and ½ of the prediction from the second word chunk. For the "first" strategy, scoring engine 440 can use the prediction from the first word chunk. For the "second" strategy, scoring engine 440 can use the prediction from the second word chunk. For the "max" strategy, scoring engine 440 can decide the prediction based on the maximum confidence score.

[0166] Process 700 can include additional implementations, such as any single implementation or any combination of the implementations described below and / or in connection with one or more other processes described elsewhere herein.

[0167] In a first implementation, each word chunk and its corresponding sequence of labels is treated as a separate example at training time.

[0168] In a second implementation, whether alone or in combination with the first implementation, a length-N utterance is decomposed into K overlapping word chunks, where K is a word chunk size and L is an overlap size.

[0169] In a third implementation, whether alone or in combination with one or more of the first implementation and the second implementation, a total score and a final annotated label for a token segment are determined based at least in part on a confidence score from a first selected word chunk of the plurality of overlapping word chunks.

[0170] In a fourth implementation, whether alone or in combination with one or more of the first implementation through the third implementation, the determination of the total score for the token segment is based on a position of the token segment in the overlapping token segment word chunk, where if the token segment is in a first half of the overlapping token segment word chunk, the total score is a first confidence score from a first word chunk, and if the token segment is in a second half of the overlapping token segment word chunk, the total score is a second confidence score from a second word chunk.

[0171] In a fifth implementation, whether alone or in combination with one or more of the first implementation through the fourth implementation, the determination of the total score is based on a maximum confidence score of the plurality of overlapping word chunks.

[0172] ​In the sixth embodiment, whether alone or in combination with one or more of the first to fifth embodiments, the predetermined threshold for word segments is 512 word segments, and the predetermined number of overlapping word segments is 128 word segments.

[0173] although Figure 7 An example block of process 700 is shown, but in some implementations, process 700 may include... Figure 7 The boxes depicted are compared to boxes with different arrangements, additional boxes, fewer boxes, or different boxes. Alternatively, two or more boxes in process 700 can be executed in parallel.

[0174] Figure 8 A simplified diagram of a distributed system 800 for implementing embodiments is depicted. In the illustrated embodiment, the distributed system 800 includes one or more client computing devices 802, 804, 806, and 808 coupled to a server 812 via one or more communication networks 810. The client computing devices 802, 804, 806, and 808 can be configured to execute one or more applications.

[0175] In various respects, server 812 can be adapted to run one or more services or software applications that implement techniques for processing long texts for pre-trained language models.

[0176] In some respects, server 812 may also provide other services or software applications, including both non-virtual and virtual environments. In some respects, these services may be provided as web-based services or cloud services (such as under a Software as a Service (SaaS) model) to users of client computing devices 802, 804, 806, and / or 808. Users operating client computing devices 802, 804, 806, and / or 808 may then use one or more client applications to interact with server 812 to utilize the services provided by these components.

[0177] exist Figure 8 In the depicted configuration, server 812 may include one or more components 818, 820, and 822 that implement the functions performed by server 812. These components may include software components that can be executed by one or more processors, hardware components, or a combination thereof. It should be understood that various different system configurations, possibly different from distributed system 800, are possible. Therefore, Figure 8 The illustrated embodiment is an example of a distributed system for implementing the system of the embodiment and is not intended to be limiting.

[0178] Users can use client computing devices 802, 804, 806, and / or 808 to process long texts for pre-trained language models according to the teachings of this disclosure. The client device can provide an interface that enables users to interact with the client device. The client device can also output information to the user via this interface. Although Figure 8 It describes only four client computing devices, but can support any number of client computing devices.

[0179] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft). Apple Or a UNIX-like operating system, Linux, or a Linux-like operating system (such as Google Chrome). TM OS), including various mobile operating systems (e.g., Microsoft Windows). Windows Android TM , Palm Portable handheld devices can include cellular phones, smartphones (e.g., ), tablet computers (e.g., Wearable devices include Google's personal digital assistants (PDAs), personal digital assistants (PDAs), and more. Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, internet-enabled gaming devices (e.g., with or without internet access). Microsoft gesture input devices Game console, Sony System, by Various game systems and others are provided. Client devices can run a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and can use various communication protocols.

[0180] One or more networks 810 can be any type of network familiar to those skilled in the art that supports data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), and IPX (Internet Packet Switching). Etc. By way of example only, network(s) 810 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., according to the IEEE 1002.11 protocol suite), etc. (and / or any network operated by any of the other wireless protocols) and / or any combination of these networks and / or other networks.

[0181] Server 812 may consist of: one or more general-purpose computers, dedicated server computers (including PC (personal computer) servers by way of example), Servers (including servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination thereof. Server 812 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as one or more elastic pools of logical storage devices that can be virtualized to maintain virtual storage devices for the server. In various aspects, server 812 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0182] The computing system in server 812 can run one or more operating systems, including any of the operating systems discussed above, as well as any commercially available server operating system. Server 812 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, etc. Servers, database servers, etc. Exemplary database servers include, but are not limited to, those that can be accessed from... Those commercially available database servers, such as those from IBM.

[0183] In some implementations, server 812 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client computing devices 802, 804, 806, and 808. As an example, data feeds and / or event updates may include, but are not limited to, those provided in the original text. feed, The server 812 can also include one or more applications to display data feeds and / or real-time events via one or more display devices of the client computing devices 802, 804, 806, and 808. Real-time updates or feeds received from one or more third party information sources and continuous data streams can include real-time events related to sensor data applications, financial ticker applications, network performance measuring tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like. The server 812 can also include one or more applications to display

[0184] The distributed system 800 can also include one or more data repositories 814, 816. In certain aspects, these data repositories can be used to store data and other information. For example, one or more of the data repositories 814, 816 can store information for techniques to handle long text for pre-trained language models (e.g., intent scores, overall scores). The data repositories 814, 816 can reside in various locations. For example, data repositories used by the server 812 can be local to the server 812 or can be remote from the server 812 and in communication with the server 812 via a network-based or dedicated connection. The data repositories 814, 816 can be of different types. In certain aspects, data repositories used by the server 812 can be databases, such as relational databases, like those provided by Oracle® and other vendors. One or more of these databases can be adapted to implement data storage, updating, and retrieval to and from the databases in response to commands in a structured query language (SQL) format.

[0185] In certain aspects, one or more of the data repositories 814, 816 can also be used by applications to store application data. Data repositories used by applications can be of different types, such as, for example, key-value store repositories, object store repositories, or general purpose storage repositories supported by a file system.

[0186] In certain aspects, the techniques described in this disclosure to handle long text for pre-trained language model functionality can be provided as a service via a cloud environment. Figure 6 is a simplified block diagram of a cloud-based system environment in accordance with certain aspects in which various text handling related services can be provided as cloud services. In Figure 6 ​In the depicted embodiment, cloud infrastructure system 602 can provide one or more cloud services that can be requested by users using one or more client computing devices 604, 606, and 608. Cloud infrastructure system 602 can comprise one or more computers and / or servers that can include those described above for server 812. The computers in cloud infrastructure system 602 can be organized into general-purpose computers, specialized server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.

[0187] Network(s) 610 can facilitate communications and exchange of data between client 604, 606, and 608 and cloud infrastructure system 602. Network(s) 610 can comprise one or more networks. Networks can be the same or different types. Network(s) 610 can support one or more communication protocols (including wired and / or wireless protocols) to facilitate communications.

[0188] Figure 9 The depicted embodiment is merely an example of a cloud infrastructure system and is not intended to be limiting. It will be appreciated that in some other aspects, cloud infrastructure system 902 can have more or fewer components than those depicted, can combine two or more components, or can have a different configuration or arrangement of components than that depicted. Figure 9 The depicted components are merely an example and that in other aspects two or more components can be combined or have a different configuration or arrangement of components. For example, although three client computing devices are depicted, in other aspects any number of client computing devices can be supported. Figure 9 Three client computing devices are depicted, but in other aspects any number of client computing devices can be supported.

[0189] The term cloud service is generally used to refer to a service that is made available to users by the system of a service provider (e.g., cloud infrastructure system 902) over anetwork, such as the Internet, and on-demand. Typically, in a public cloud environment, servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is typically managed by the cloud service provider. The customer can access the cloud service provider's system via any networked device, including their own computers, laptops, tablets, and mobile devices. Customers can access the cloud service provider's system via a thin client or a thick client. Customers can utilize applications that are provided by the cloud service provider on-demand. A cloud service is designed to provide easy, scalable access to applications, resources and services. Several providers offer cloud services. For example, Oracle Several cloud services are offered, such as middleware services, database services, Java cloud services, and other services.

[0190] In certain aspects, cloud infrastructure system 902 can provide one or more cloud services using different models, such as under a software as a service (SaaS) model, a platform as a service (PaaS) model, an infrastructure as a service (IaaS) model, and other models (including hybrid models). Cloud infrastructure system 902 can include a suite of applications, middleware, databases, and other resources that implement the provisioning of the various cloud services.

[0191] A SaaS model enables applications or software to be delivered to customers as a service over a communication network, such as the Internet, without the customer having to purchase the underlying application's hardware or software. For example, customers can be provided with access to on-demand applications hosted by cloud infrastructure system 902 using a SaaS model. Oracle Examples of SaaS services provided include, without limitation, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0192] An IaaS model is typically used to provide infrastructure resources (e.g., servers, storage, hardware, and networking resources) as a cloud service to customers to provide elastic computing and storage capacity. Oracle Various IaaS services are provided by Oracle

[0193] A PaaS model is typically used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without the customer having to purchase, construct, and / or maintain the underlying hardware and software resources. Oracle Examples of PaaS services provided include, without limitation, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, various application development solution services, and others.

[0194] Cloud services are typically provided in a manner that is on-demand, self-service, elastically scalable, reliable, highly available, and secure. For example, a customer can order one or more services provided by cloud infrastructure system 902 via a subscription order. Cloud infrastructure system 902 then performs processing to provide the services requested in the customer's subscription order. Cloud infrastructure system 902 can be configured to provide one or even multiple cloud services.

[0195] The cloud infrastructure system 902 can provide cloud services via different deployment models. In a public cloud model, the cloud infrastructure system 902 can be owned by a third party cloud service provider and the cloud services are provided to any general public customer, where the customer can be an individual or an enterprise. In some other aspects, under a private cloud model, the cloud infrastructure system 902 can operate within an organization (e.g., within an enterprise organization) and the services are provided to customers within the organization. For example, the customers can be various departments of the enterprise or even individuals within the enterprise, such as a human resources department, a payroll department, etc. In some other aspects, under a community cloud model, the cloud infrastructure system 902 and the services provided can be shared by several organizations within a related community. Various other models, such as a hybrid of the models mentioned above, can also be used.

[0196] The client computing devices 904, 906, and 908 can be different types of (e.g., desktop computers, laptop computers, tablet computers, etc.) and can be capable of operating one or more client applications. Users can use the client devices to interact with the cloud infrastructure system 902, such as to request services provided by the cloud infrastructure system 902. For example, a user can use a client device to request the chatbot services described in this disclosure. Figure 8

[0197] In some aspects, the processing performed by the cloud infrastructure system 902 for providing the Chabot services can involve big data analytics. This analytics can involve using, analyzing, and manipulating large datasets to detect and visualize various trends, behaviors, relationships, etc. within the data. This analytics can be performed by one or more processors, which can process the data in parallel, perform simulations using the data, etc. For example, big data analytics can be performed by the cloud infrastructure system 902 for determining the intent of an utterance. The data used for this analytics can include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0198] As Figure 9 As depicted in the embodiments in

[0199] ​In some respects, to facilitate the efficient provisioning of these resources to support the various cloud services offered by the cloud infrastructure system 902 to different customers, resources can be bound to resource groups or resource modules (also known as "pods"). Each resource module or pod can include a pre-integrated and optimized combination of one or more types of resources. In some respects, different pods can be pre-provisioned for different types of cloud services. For example, a first pod can be provisioned for database services, and a second pod can be provisioned for Java services (the second pod may include a different combination of resources than the pods in the first pod), and so on. For some services, resources allocated for provisioning services can be shared between services.

[0200] The cloud infrastructure system 902 itself can internally use services 932 shared by different components of the cloud infrastructure system 902 and facilitating the services provided by the cloud infrastructure system 902. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0201] Cloud infrastructure system 902 may include multiple subsystems. These subsystems may be implemented in software or hardware, or a combination thereof. Figure 9 The depicted subsystem may include a user interface subsystem 912 that enables users or customers of the cloud infrastructure system 902 to interact with the cloud infrastructure system 902. The user interface subsystem 912 may include various interfaces, such as a network interface 914, an online store interface 916 (where cloud services provided by the cloud infrastructure system 902 are advertised and can be purchased by consumers), and other interfaces 918. For example, a customer may use a client device to request (service request 934) one or more services provided by the cloud infrastructure system 902 using one or more of the interfaces 914, 916, and 918. For example, a customer may access an online store, browse cloud services provided by the cloud infrastructure system 902, and place a subscription order for one or more services provided by the cloud infrastructure system 902 that the customer wishes to subscribe to. The service request may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for Chabot-related services provided by the cloud infrastructure system 902. As part of the order, the customer may provide information for identifying the input (e.g., words).

[0202] In some aspects (such as) Figure 9As part of this processing, OMS 920 can be configured to: create an account for the customer (if not already created); receive billing and / or accounting information from the customer to be used to bill the customer for providing the requested service to the customer; verify the customer information; after verification, reserve the order for the customer; and orchestrate various workflows to prepare the order for provisioning.

[0203] Once properly verified, OMS 920 can then invoke an order provisioning subsystem (OPS) 924 that is configured to provision resources (including processing resources, memory resources, and networking resources) for the order. Provisioning can include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provisioned and the types of resources provisioned can depend on the type of cloud service that the customer has ordered. For example, according to one workflow, OPS 924 can be configured to determine the particular cloud service that is being requested and identify the number of pods that can have been pre-configured for that particular cloud service. The number of pods allocated for the order can depend on the size / amount / level / scope of the requested service. For example, the number of pods to allocate can be determined based on the number of users to be supported by the service, the duration of the service being requested, and the like. The pods allocated can then be customized for the particular requesting customer for providing the requested service.

[0204] Cloud infrastructure system 902 can send a response or notification 944 to the requesting customer to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) can be sent to the customer that enables the customer to begin using and taking advantage of the benefits of the requested service.

[0205] Cloud infrastructure system 902 can provide services to a plurality of customers. For each customer, cloud infrastructure system 902 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 902 can also collect usage statistics regarding the customer’s use of the subscribed services. For example, statistics such as the amount of storage used, the amount of data transferred, the number of users, and the amount of system up-time and system down-time, among others, can be collected. This usage information can be used to bill the customer. For example, billing can be done on a monthly cycle.

[0206] Cloud infrastructure system 902 can provide services to multiple customers in parallel. Cloud infrastructure system 902 can store information about these customers (potentially including proprietary information). In some respects, cloud infrastructure system 902 includes an Identity Management Subsystem (IMS) 928, which is configured to manage customer information and provide separation of the managed information, ensuring that information related to one customer cannot be accessed by another customer. IMS 928 can be configured to provide various security-related services, such as information access management, authentication and authorization services, and services for managing customer identities and roles and related functions.

[0207] Figure 10 An exemplary computer system 1000 is illustrated that can be used to implement certain aspects. For example, in some aspects, the computer system 1000 can be used to implement, such as Figure 4 The system 400 shown is for processing long texts for a pre-trained language model, and any of the various servers and computer systems described above. For example... Figure 10 As shown, the computer system 1000 includes various subsystems, including a processing subsystem 1004 that communicates with multiple other subsystems via a bus subsystem 1002. These other subsystems may include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. The storage subsystem 1018 may include non-transitory computer-readable storage media, including storage medium 1022 and system memory 1010.

[0208] Bus subsystem 1002 provides mechanisms for allowing various components and subsystems of computer system 1000 to communicate with each other as intended. While bus subsystem 1002 is schematically shown as a single bus, alternative aspects of the bus subsystem may utilize multiple buses. Bus subsystem 1002 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of various bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (which may be implemented as a Mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0209] The processing subsystem 1004 controls the operation of the computer system 1000 and can comprise one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). A processor can include a single-core processor or multiple-core processor. Processing resources of the computer system 1000 can be organized into one or more processing units 1032, 1034, etc. A processing unit can comprise one or more processors, one or more cores from the same or different processors, combinations of cores and processors, or other combinations of cores and processors. In some aspects, the processing subsystem 1004 can include one or more special-purpose co-processors, such as a graphics processor, a digital signal processor (DSP), or the like. In some aspects, some or all of the processing units of the processing subsystem 1004 can be implemented using custom circuitry, such as application-specific integrated circuitry (ASIC) or field-programmable gate array (FPGA) circuitry.

[0210] In some aspects, the processing units in the processing subsystem 1004 can execute instructions stored in the system memory 1010 or on the computer-readable storage media 1022. In various aspects, the processing units can execute a variety of programs or code instructions and can maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code to be executed can be resident in the system memory 1010 and / or on the computer-readable storage media 1022 (potentially including being resident on one or more storage devices). Through suitable programming, the processing subsystem 1004 can provide various functionalities described above. In aspects in which the computer system 1000 executes instances of one or more virtual machines, one or more of the processing units can be allocated to each virtual machine.

[0211] In certain aspects, a processing acceleration unit 1006 can optionally be provided to perform customized processing or to off-load processing of some of the instructions from the processing subsystem 1004, thereby speeding up the overall processing of the computer system 1000.

[0212] The I / O subsystem 1008 can include devices and mechanisms for inputting information to and / or for outputting information from the computer system 1000. In general, use of the term input device is intended to encompass all possible types of devices and mechanisms for inputting information to the computer system 1000. User interface input devices can include, for example, a keyboard, pointing devices such as a mouse or trackball, touchpads or touch screens incorporated into the display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, audio input devices with voice command motion sensor, Microsoft 360 game controller, providing an interface device for receiving input using gestures and spoken commands. The user interface input device can also include an eye gesture recognition device, such as detecting eye activity from a user (e.g., "blinking" when taking a picture and / or making a menu selection) and translating the eye gestures into inputs into an input device (such as a Google ) Google blink detector. Additionally, the user interface input device can include a voice recognition sensing device that enables a user to interact with a voice recognition system (e.g., navigator) through voice commands.

[0213] Other examples of user interface input devices include, without limitation, three dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphics tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser rangefinders, eye gaze tracking devices. Additionally, the user interface input devices can include, for example, medical imaging input devices including computed tomography, magnetic resonance imaging, position emission tomography and medical ultrasonography devices. The user interface input devices can also include, for example, audio input devices including MIDI keyboards, digital musical instruments and the like.

[0214] In general, use of the term output device is intended to include all possible types of devices and mechanisms for outputting information from computer system 1000 to a user or other computer. User interface output devices can include a display subsystem, indicator lights, or non-visual displays such as audio output devices. Display subsystems can include cathode ray tubes (CRT), flat-panel devices such as liquid crystal displays (LCDs) or plasma display panels, projection devices, touch-screen displays, and the like. For example, user interface output devices can include, but are not limited to, various display devices that convey text, graphics, and audio / video information in visual, auditory or tactile form such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, voice output devices, and modems.

[0215] Storage subsystem 1018 provides a repository or data store for storing information and data used by computer system 1000. Storage subsystem 1018 provides a tangible, non-transitory computer-readable storage medium for storing basic programming and data constructs that provide the functionality of some aspects. Software (e.g., programs, code modules, instructions) that when executed by processing subsystem 1004 provide the functionality described above can be stored in storage subsystem 1018. These software modules or instructions can be executed by one or multiple processing units of processing subsystem 1004. Storage subsystem 1018 can also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0216] Storage subsystem 1018 can include one or more non-transitory memory devices, including volatile memory devices and non-volatile memory devices. As shown, storage subsystem 1018 includes a system memory 1010 and a computer-readable storage medium 1022. System memory 1010 can include a number of memories including a volatile main random access memory (RAM) for storage of instructions and data during program execution and a non-volatile read only memory (ROM) or flash memory, in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within computer system 1000, such as during start-up, can typically be stored in the ROM. The RAM typically contains data and / or program modules that are presently being operated and executed by processing subsystem 1004. In some implementations, system memory 1010 can include a number of different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or the like. Figure 10

[0217] By way of example, and not limitation, as depicted, system memory 1010 can include an operating system 1016, program data 1014, and program modules 1012, which Figure 10 include, for example, various applications 1012, such as a web browser, a middle tier application, a relational database management system (RDBMS), etc., program data 1014, and an operating system 1016. By way of example, operating system 1016 can include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, a variety of commercially-available or UNIX-like operating systems (including without limitation the various GNU / Linux operating systems, the Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Android®, Windows® Phone, BlackBerry® 10 operating systems, etc., and / or other mobile operating systems. ​​​Mobile operating systems, such as the OS operating system, and other operating systems.

[0218] Computer-readable storage media 1022 can store programming and data constructs that provide the functionality of some aspects. Computer-readable media 1022 can also be used to provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1000. Software (programs, code modules, instructions) that, when executed by processing subsystem 1004, provide the functionality described above can be stored in storage subsystem 1018. By way of example, computer-readable storage media 1022 can include non-volatile memory such as a hard disk drive, a magnetic disk drive, an optical disk drive (e.g., a CD ROM, a digital video disk (DVD), a Blu-ray disk, or other optical media), or a solid-state memory drive (SSD), as well as flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, or other nonvolatile storage media. Computer-readable storage media 1022 can also include volatile storage media such as a random access memory (RAM), a dynamic RAM (DRAM), a static RAM (SRAM), a fast page mode DRAM (FPM DRAM), a double data rate (DDR) DRAM, a synchronous DRAM (SDRAM), a double data rate type two SDRAM (DDR2 SDRAM), a double data rate type three SDRAM (DDR3 SDRAM), a double data rate type four SDRAM (DDR4 SDRAM), a double data rate type five SDRAM (DDR5 SDRAM), or other volatile storage media.

[0219] In certain aspects, storage subsystem 1018 can also include a computer- readable storage media reader 1020 that can further connect to computer-readable storage media 1022. Reader 1020 can receive data from and / or write data to the computer-readable storage media. In certain aspects, the computer-readable storage media 1022 can include volatile memory, non-volatile memory, removable storage, and / or non-removable storage.

[0220] In certain aspects, computer system 1000 can support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, computer system 1000 can provide support for executing one or more virtual machines. In certain aspects, computer system 1000 can execute a program, such as a hypervisor, that facilitates the virtualization of a computing resource. Each virtual machine can be allocated storage, computation (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs an operating system independently of the operating systems executed by other virtual machines executed by computer system 1000. Thus, multiple operating systems can potentially be run simultaneously by computer system 1000.

[0221] ​The communications subsystem 1024 provides an interface to other computer systems and networks. The communications subsystem 1024 serves as an interface for receiving data from and transmitting data to other systems from the computer system 1000. For example, the communications subsystem 1024 can enable the computer system 1000 to establish a communication channel to one or more client devices via the Internet for receiving and sending information to and from the client devices. For example, the communications subsystem can be used to transmit responses to the user regarding inquiries to the Chabot.

[0222] The communications subsystem 1024 can support both wired and / or wireless communications protocols. For example, in certain aspects, the communications subsystem 1024 can include radio frequency (RF) transceiver components (e.g., for use in a wireless telecommunications network), global positioning system (GPS) receiver components, and / or other components. In some aspects, the communications subsystem 1024 can provide cellular network connectivity (e.g., 3G, 4G, or EDGE network technology, advanced data network technology such as 3G, 4G, or EDGE (enhanced data rates for global evolution), Wi-Fi (IEEE 802.XX family standards, or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components, in addition to or instead of a wireless interface. In some aspects, the communications subsystem 1024 can provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

[0223] The communications subsystem 1024 can receive and transmit various forms of data. For example, in some aspects, the communications subsystem 1024 can receive input communications in the form of structured and / or unstructured data feeds 1026, event updates 1030, and the like, in addition to other forms. For example, the communications subsystem 1024 can be configured to receive (or send) data feeds 1026, update streams 1028, event updates 1030, and the like, from social media networks and / or other communication services in real-time or near real-time. For example, the communications subsystem 1024 can be configured to receive data feeds 1026 related to current events, trending topics, and / or the like. feeds, updates, network feeds (such as a rich site summary (RSS) feed), and / or real-time updates from one or more third party information sources.

[0224] In certain aspects, the communications subsystem 1024 can be configured to receive data in the form of continuous data streams, which can include event streams 1028 and / or event updates 1030 of real-time events that can be continuous in nature and / or unbounded in quantity and have no explicit end. Examples of applications generating continuous data can include, for example, sensor data applications, financial ticker applications, network performance monitoring applications (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0225] The communication subsystem 1024 also can be configured to communicate data from the computer system 1000 to other computer systems or networks. The data can be communicated in various forms including as structured and / or unstructured data feeds 1026, event updates 1030, and the like. The data can be communicated to one or more databases that can be in communication with one or more stream data source computers coupled to the computer system 1000.

[0226] The computer system 1000 can be one of various types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a handheld computer, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, the Figure 10 description of the computer system 1000 depicted is intended only as a specific example. For example, hardware and / or software Figure 10 depicted can be used in other systems. Those of ordinary skill in the art will appreciate that many other configurations are possible.

[0227] While specific aspects have been described above, various modifications, changes, substitutions, and equivalents choosing to implement various aspects. The embodiments are not limited to operation within certain specific data processing environments, but are free to operate within a plurality of data processing environments. Additionally, although a specific series of acts has been described, variations of the method are possible, as will be appreciated by those skilled in the art. For example, the order of the acts can be rearranged. Additional acts can be added or other acts can be removed. Further, some acts can be performed concurrently or with partial concurrence, particularly with regard to

[0228] Further, while certain aspects have been described as comprising hardware and software components, it should be appreciated that other alternatives can be used. For example, the mechanisms of a particular aspect can be implemented in hardware, software, or a combination of hardware and software. Various features and aspects of the above-described

[0229] Where a device, system, component or module is described as being configured to perform certain operations or implement certain functionality, such configuration can be accomplished, for example, by designing electronic circuitry to perform the operation, by programming programmable electronic circuitry (such as a microprocessor) to perform the operation, or by programming a processor or core or any combination of processors or cores to perform the operations (e.g., by executing computer instructions or code), or any combination thereof. Processes can communicate using a variety of techniques including, but not limited to, conventional techniques for interprocess communication, and different pairs of processes can use different techniques, or the same pair of processes can use different techniques at different times.

[0230] Specific details are given in this disclosure to provide a thorough understanding of the aspects. The aspects can, however, be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have not been shown in detail or have been omitted so as not to obscure the aspects. This description provides example aspects only and is not intended to limit or

[0231] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent that changes and modifications can be made by persons skilled in the art without departing from the scope of the claims. While specific aspects have been described above, one of ordinary skill in the art will appreciate that various modifications and changes can be made thereto without departing from the spirit and scope of the claims. Accordingly, it is intended that the claims be construed to cover all such modifications and changes.

Claims

1. A method for natural language processing, comprising: The trained machine learning model is generated through the following operations: Access a set of natural language discourses; Determine the length of the set of natural language utterances; When the length of the set of natural language utterances exceeds a predetermined threshold for word segments: The set of natural language discourses is divided into multiple overlapping lexical fragments and chunks; Assign labels and confidence scores to each lexical segment in the word block; The final label and associated confidence score are determined for each of the plurality of overlapping lexical fragment chunks by merging two confidence scores, one of which comes from the first chunk and the other from the second chunk. The final annotated labels of the set of natural language discourses are determined based at least on the final labels and associated confidence scores of each of the multiple overlapping lexical fragment chunks. as well as Store the final annotated labels in memory; as well as The machine learning model is trained using the set of natural language utterances and the final annotated labels, wherein the trained machine learning model is obtained. Access natural language discourse that includes multiple lexical units; Using the trained machine learning model to generate labeled natural language discourse, wherein using the trained machine learning model to generate labeled natural language discourse includes using the trained machine learning model to label the plurality of lexical units with a plurality of labels, wherein at least one of the plurality of labels corresponds to one of a plurality of named entity categories. Store the labeled natural language utterances.

2. The method as described in claim 1, wherein, Each word block and its corresponding label sequence are treated as a separate example during training.

3. The method as described in claim 1, wherein, The set of natural language discourse with a partition size of N is decomposed into (NL) / (KL) overlapping word blocks, where K is the word block size and L is the overlap size.

4. The method of claim 1, wherein, The final annotated label and overall score of the determined lexical segment are based at least in part on the confidence score of the first selected lexical segment from the plurality of overlapping lexical segments.

5. The method of claim 1, wherein, The overall score for determining a word segment is based on the position of the word segment within a plurality of overlapping word segment blocks. If the word segment is in the first half of the plurality of overlapping word segment blocks, the overall score is a first confidence score from the first block, and if the word segment is in the second half of the plurality of overlapping word segment blocks, the overall score is a second confidence score from the second block.

6. The method of claim 1, wherein, The overall score is determined based on the maximum confidence score of the multiple overlapping word blocks.

7. The method of claim 1, wherein, The predetermined threshold for the word segment is 512 words, and the predetermined number of overlapping word segments is 128 words.

8. A system for natural language processing, comprising: One or more processors; as well as A memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions, when executed by the one or more processors, causing the one or more processors to perform operations including: The trained machine learning model is generated through the following operations: Access a set of natural language discourses; Determine the length of the set of natural language utterances; When the length of the set of natural language utterances exceeds a predetermined threshold for word segments: The set of natural language discourses is divided into multiple overlapping lexical fragments and chunks; Assign labels and confidence scores to each lexical segment in the word block; The final label and associated confidence score for each word block in the overlapping lexical fragment word block are determined by merging two confidence scores, one of which comes from the first word block and the other from the second word block. The final annotated labels of the set of natural language discourses are determined based at least on the final labels and associated confidence scores of each of the multiple overlapping lexical fragment chunks. as well as Store the final annotated labels in memory; as well as The machine learning model is trained using the set of natural language utterances and the final annotated labels, wherein the trained machine learning model is obtained. Access natural language discourse that includes multiple lexical units; Using the trained machine learning model to generate labeled natural language discourse, wherein using the trained machine learning model to generate labeled natural language discourse includes using the trained machine learning model to label the plurality of lexical units with a plurality of labels, wherein at least one of the plurality of labels corresponds to one of a plurality of named entity categories. Store the labeled natural language utterances.

9. The system of claim 8, wherein, Each word block and its corresponding label sequence are treated as a separate example during training.

10. The system of claim 8, wherein, The set of natural language discourse with a partition size of N is decomposed into (NL) / (KL) overlapping word blocks, where K is the word block size and L is the overlap size.

11. The system of claim 8, wherein, The final annotated label and overall score of the determined lexical segment are based at least in part on the confidence score of the first selected lexical segment from the plurality of overlapping lexical segments.

12. The system of claim 8, wherein, The overall score for determining a word segment is based on the position of the word segment within the overlapping word segment block. If the word segment is in the first half of the overlapping word segment block, the overall score is a first confidence score from the first word segment. If the word segment is in the second half of the overlapping word segment block, the overall score is a second confidence score from the second word segment.

13. The system of claim 8, wherein, The overall score is determined based on the maximum confidence score of the multiple overlapping word blocks.

14. The system of claim 8, wherein, The predetermined threshold for the word segment is 512 words, and the predetermined number of overlapping word segments is 128 words.

15. A non-transitory computer-readable medium storing a plurality of instructions executable by one or more processors, the plurality of instructions causing the one or more processors to perform operations including: The trained machine learning model is generated through the following operations: Access a set of natural language discourses; Determine the length of the set of natural language utterances; When the length of the set of natural language utterances exceeds a predetermined threshold for word segments: The set of natural language discourses is divided into multiple overlapping lexical fragments and chunks; Assign labels and confidence scores to each lexical segment in the word block; The final label and associated confidence score for each word block in the overlapping lexical fragment word block are determined by merging two confidence scores, one of which comes from the first word block and the other from the second word block. The final annotated labels of the set of natural language discourses are determined based at least on the final labels and associated confidence scores of each of the multiple overlapping lexical fragment chunks. as well as Store the final annotated labels in memory; as well as The machine learning model is trained using the set of natural language utterances and the final annotated labels, wherein the trained machine learning model is obtained. Access natural language discourse that includes multiple lexical units; Using the trained machine learning model to generate labeled natural language discourse, wherein using the trained machine learning model to generate labeled natural language discourse includes using the trained machine learning model to label the plurality of lexical units with a plurality of labels, wherein at least one of the plurality of labels corresponds to one of a plurality of named entity categories. Store the labeled natural language utterances.

16. The non-transitory computer-readable medium of claim 15, wherein, Each word block and its corresponding label sequence are treated as a separate example during training.

17. The non-transitory computer-readable medium of claim 15, wherein, The set of natural language discourse with a partition size of N is decomposed into (NL) / (KL) overlapping word blocks, where K is the word block size and L is the overlap size.

18. The non-transitory computer-readable medium of claim 15, wherein, The overall score of the determined word segment and the final annotated label are based at least in part on the confidence score of the first selected word segment from the plurality of overlapping word segments.

19. The non-transitory computer-readable medium of claim 15, wherein, The overall score for determining a word segment is based on the position of the word segment within the overlapping word segment block. If the word segment is in the first half of the overlapping word segment block, the overall score is a first confidence score from the first word segment. If the word segment is in the second half of the overlapping word segment block, the overall score is a second confidence score from the second word segment.

20. The non-transitory computer-readable medium of claim 15, wherein, The overall score is determined based on the maximum confidence score of the multiple overlapping word blocks.

Images