Real-time natural language processing and fulfillment

Abstract

A real-time feedback confirmation system and method for requesting a virtual assistant response from the evolving semantic state of at least a portion of an utterance. A user accesses a virtual assistant on an electronic device having a system and / or method configured to capture commands, questions, and / or requests from audio, such as speech, uttered by a speaking user. The speech may be captured by a speech engine configured to transcribe the speech into text that is matched with regular expressions of fragment patterns to generate fragments, and / or the speech may be processed by a machine learning model to identify fragments. The fragments are identified by a domain handler configured to update a data structure of the current semantic state of the utterance in real time on the interface of the electronic device.

Description

[Technical Field]

[0001] Priority claims This application claims priority to U.S. Patent Application No. 18 / 055,821, entitled "REAL-TIME NATURAL LANGUAGE PROCESSING AND FULFILLMENT," filed November 15, 2022, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] background Traditional natural language understanding system techniques update at the end of each sentence after the entire sentence statement has been identified as a unit. For example, some natural language understanding techniques interpret the meaning of an entire sentence after its completion has been received by the system. Currently, techniques used to calculate changes to the semantic state of the system as a result of an utterance occur only upon detection of the end of the utterance. Summary of the Invention

[0003] overview Methods for processing fragments in a natural language understanding environment are described, both as computer-executed methods and / or as computer-readable media containing instructions that, when executed by a computer, cause the computer to perform the steps of the methods. One embodiment includes using a speech engine that performs acoustic speech recognition. The speech engine generates a continuous transcription of the speech. For example, a user accesses a virtual assistant using an electronic device having a visual display configured to capture commands, questions, and / or requests from audio, including but not limited to the user's speech. One embodiment includes continuous transcription, although in some embodiments, the transcription may be substantially continuous, intermittent, and / or have delays and / or pauses.

[0004] In one embodiment, the transcription of the audio is inferred by a speech engine. The speech engine is configured to receive input speech and generate a text transcription. Specifically, the speech engine is configured to transcribe at least portions of the speech, including, but not limited to, sentence fragments, words, at least portions of words, and / or at least portions of utterances. The system uses the speech engine to transcribe the audio into text that forms the transcription. The fragments are then sent to a fragment consolidator. The fragment consolidator's rules define the sequence of fragments that the fragment consolidator should search for, and the rules include what message and / or messages to output if the rule is matched.

[0005] The fragment synthesizer sends raw fragments to the domain handler, or the synthesizer matches rules with sequences of fragments and invokes the rules to send messages to the domain handler. The system's contextual relevance combined with fragment parsing causes the semantic state to be updated to incorporate commands that have been spoken so far. As a result, the ongoing updating of context allows suggestions to be made to the user on how to complete their thoughts and therefore how to complete sentences that they are in the middle of.

[0006] In another embodiment, a simple system without fragment consolidation may be used, as long as the application relies only on a raw stream of fragments. For example, some implementations may not have a fragment consolidator. As a result, fragments are sent directly to a domain handler. The domain handler outputs its continuously updated semantic state, which is the cumulative semantic understanding of the utterances processed so far.

[0007] A database is provided. The database contains fragment patterns. A fragment pattern is a short segment of text. The fragment pattern can be compared to at least one word of the transcript, and the resulting fragments are recognized and generated by a fragment classifier. It is within the scope of the present invention for the fragment classifier to include, but not be limited to, a sequence-to-sequence neural network, which is a general machine learning model. The fragment classifier infers the fragments directly from the audio.

[0008] A sequence of at least one word from a transcription of at least a portion of a user's utterance that matches a regular expression in a fragment pattern may form a fragment. Fragments in a transcription as short as part of a word are identified as "fragments" by the fragment identifier. It is within the scope of the present invention for a sequence of at least one word in a transcription to have at least one word identified by the fragment identifier as a "fragment" of text. For example, the fragment identifier recognizes text from the transcription and associates them with an intent to output fragments to be identified by the domain handler. For example, words and / or word combinations that are not recognized as "fragments" are ignored. Some examples of fragments are "classic cheeseburger" or "one dozen" or "chocolate brownie bar," which can be matched by the fragment identifier using fragment patterns such as:

[0009] a. ["classic"]("cheeseburger"|"hamburger") b. "twelve"|"dozen"|"one dozen" c. (["chocolate"].(("brownie". ["bar"|bars"])|"brownies"))(["chocolate"].(("brownie(singular)".["bar"(singular)|"bars"(plural)])|"brownies"(plural))) These fragments are processed immediately and / or simultaneously as they are identified by a "domain handler" that updates a data structure representing the current semantic state of the longer utterance, which may still be in progress. In this way, understanding of the utterance occurs incrementally based on the identified fragments, rather than all at once at the end of a short utterance. Some examples of variables and legal values ​​in the semantic state data structure are as follows:

[0010] a. INTENT=DELETE, FLAVOR=chocolate, ITEM=shake b. INTENT=CHANGE, TARGET=first_item, ITEM=cheeseburger, TOPPING=onions c. INTENT=ADD, SIZE=large, ITEM=fries, TOPPING=chili Aspects of the semantic state represented in the domain handler may be streamed to the user in real time. This can be used to indicate to the user that the system's understanding is correct during the utterance and / or when at least a portion of the utterance is detected by the system, and / or at any time before the utterance is completed. If the system's understanding is incorrect, the user is quickly alerted so that the user can make adjustments. It is within the scope of the present invention that during the utterance is not limited to a particular percentage of the utterance. The detection of at least a portion of the utterance may begin at the beginning of the user's utterance of at least a portion of the utterance and end any time after the beginning, allowing the system to detect the fragment.

[0011] The semantic state represented by the domain handler can be used to act on already understood intent or partial intent while the user is still speaking.

[0012] Alternatively, the domain handler can simply process all fragments immediately, thereby taking immediate action depending on the usage, with little or no semantic state being updated. An example would be a series of several commands in one utterance, where each subcommand is sorted from the fragment and processed immediately without keeping track of semantic state.

[0013] The fragments may be further processed by a fragment consolidator before they are processed by the domain handler. The fragment consolidator will wait and / or pause until the fragment consolidator has had an opportunity to identify subsequent fragment changes and / or disambiguate meaning or user intent before sending the fragments to the domain handler. When such sequences are detected, the fragments may be modified, removed, and / or processed in a different order. Furthermore, additional messages may be inserted to clarify the meaning of the collection of fragments before being sent to the domain handler.

[0014] The semantic states, together with the results of the domain handler, can then influence or bias subsequent transcriptions by acoustic speech recognition in initial and further steps. Such context-dependent influences based on the evolutionary semantic states in the domain handler can be used to improve the accuracy of the evolutionary transcription by biasing the acoustic or linguistic scores used by the speech engine and / or any other related means. This is an improvement over prior art techniques, which would only be updated at the end of each utterance after the utterance has been understood as a unit.

[0015] Additional explanation It is within the scope of the present invention for at least one word to consist of at least one character, including but not limited to letters, numbers, alphanumeric characters, common punctuation marks such as "." and / or "-", and spaces.

[0016] It is within the scope of this invention that alphanumeric characters consist of both letters and numbers and / or other symbols, including but not limited to punctuation marks and / or mathematical symbols.

[0017] It is within the scope of the present invention that audio includes, but is not limited to, speech, spoken words, statements and / or voiced sounds.

[0018] In some embodiments, the domain handler maintains a conversation state data structure. While a semantic state data structure may contain many items of the same or different types, the conversation state remembers the most recently mentioned entity of any type. This is useful for functions such as pronoun disambiguation. The conversation state is also volatile, in that values ​​age over time as the conversation progresses. Some embodiments remove values ​​from the conversation state after a certain period of time.

[0019] The domain handler, conversational state changes, and / or semantic state changes may cause real-time display updates and / or changes as a result of matching snippets in the transcription against the snippet patterns. It is within the scope of the present invention for displays, including but not limited to visual displays, vibrations, light units, mechanical activation, and / or auditory feedback, to inform the user of the status of the domain handler's output. In one example, the auditory feedback may be a phrase such as "yup" and / or any suitable non-verbal cue. For example, the visual display may be part of an electronic device having a user interface, e.g., with a text message visible to the user on the screen of the user interface. In another example, the display may be a plurality of light units configured to light up in a predetermined manner according to the status update. In yet another example, the display may be a series of vibrations corresponding to the status update. In another example, the update may be displayed as an audio output, such as using text-to-speech (TTS). In one example, the displayed update may move a motor, such as adjusting the position of a robotic arm. In one embodiment, the displayed updates may be virtual motion in a simulation such as a video game and / or a virtual reality environment such as the Metaverse.

[0020] Some examples of electronic devices include mobile devices such as automobiles, handheld devices such as smartphones, tablets and notebook computers, stationary devices such as kiosks and vending machines, and home appliances such as refrigerators and water coolers.

[0021] It is within the scope of the present invention that speech includes, but is not limited to, clear and / or unclear sounds.

[0022] It is within the scope of the present invention that the automatic speech recognition engine includes, but is not limited to, a machine learning model and / or a neural network configured to uniquely map inputs including, but not limited to, words, at least portions of words, at least portions of utterances, sentences, fragments, text, audio and / or video to outputs including, but not limited to, transcriptions, identified fragments and / or interpretation data structures.

[0023] A fragment pattern can be an expression such as plain text and / or a regular expression, and a particular piece of text that matches a regular expression in a fragment pattern is a "fragment." Each regular expression fragment pattern can match one or more actual sequences of words. The matching actual sequence is a fragment. A fragment can be as small as at least part of a word. A fragment can have slots that can be filled with different values ​​for each instance of the same fragment.

[0024] The domain handler can update the semantic state. The domain handler can update the conversation state data structure with information from the snippet. In another embodiment, the domain handler can interact with a dialogue manager. The dialogue manager has both a conversation state that tracks things like entity values ​​needed for pronoun disambiguation, and a semantic state, such as the list of items in a shopping cart of elements shown in a visual display. In one example of pronoun disambiguation, if there are five items in the shopping cart and the user says "delete that," the scope of the deleted items will be limited to the single most recently mentioned item.

[0025] A domain handler takes in "fragments." In one example, the fragments "change it," "make it...," "replace it," and "instead of it" are all mapped to the fragment "INTENT.MODIFY." A domain handler can take in multiple types of input; in practice, a software engineer may choose to create an abstract object that covers all types of input, or the software engineer may provide an interface to the domain handler to accept different types of messages. For example, three types of messages a domain handler may receive include, but are not limited to, 1) "raw" fragments that the synthesizer simply passes through; 2) messages from the fragment synthesizer that are sent when the fragment synthesizer detects a linguistic pattern for which it has rules; and 3) natural language understanding data structures that represent the semantic information of an entire sentence.

[0026] Examples of three types of messages that a domain handler may receive are: 1) INTENT.MODIFY 2) MODIFY target=second_item option=mustard action=add 3) INTENT=info_question, TOPIC=dessert_menu The fragment identifier identifies these fragments and knows their associated fragments which are passed to the fragment aggregator, which then passes those raw fragments to the domain handler or creates other messages to pass to the domain handler.

[0027] The fragment integrator is configured to detect patterns such as "add options to an item" (where options are fragments such as "TOPPING.MUSTARD" which matches the fragment "mustard" and items are fragments such as "ITEM.CHEESEBURGER"), and the integrator matches its rules with the incoming fragment and outputs a message or series of messages to the domain handler, in this case to add a topping to a menu item.

[0028] Some embodiments include a further step of analyzing the transcription using sentence-level natural language understanding. This occurs in parallel with the text fragment identifier. Just before the domain handler is an arbitrator that selects between the fragment (or edited fragment stream) and / or an interpretation of the natural language understanding results. The arbitrator selects the natural language understanding results if the natural language understanding function indicates a successful interpretation of the transcription. The arbitrator takes three types of input: a natural language understanding data structure, a summary message from the integrator when the integrator matches the rules, or the raw fragments. Both the arbitrator and the domain handler must know how to process all three types of messages. If the arbitrator receives only one type of input without receiving another type within a certain period of time, it passes that input through; otherwise, if there are two or more inputs within the same period of time, the arbitrator selects which of the inputs to output to the domain handler. None of the inputs selected for output are transformed. For example, the arbitrator decides whether the full sentence natural language understanding results or the fragment consolidator natural language understanding results are processed by the domain handler.

[0029] In one embodiment, a separate algorithm may be used to identify entire utterances or sentences, for example, by looking for interrogative words as the start of a sentence and pauses as the end of a sentence. A full utterance-level natural language understanding engine is used to generate a natural language understanding interpretation. A decision process in the arbitrator may choose to use full utterance-level natural language understanding results or process fragment-level natural language understanding results. For example, a natural language understanding result that could not understand the utterance would be discarded, and a fragment-level natural language understanding response would be used instead.

[0030] In another embodiment, the transcription is analyzed to identify a sequence of tokens that is assumed to be a full-sentence utterance, and the sequence of tokens is then processed by full-utterance natural language understanding. More generally, any extended sequence of tokens, whether forming a single or multiple sentences, or even less than a sentence, such as an entire clause, can be identified. One example looks for an interrogative word, then a pause, by performing explicit token matching or capturing the interrogative word as a fragment, and sends the token sequence from the interrogative word to the pause to full-sentence natural language understanding. Then, after receiving the full-sentence natural language understanding response, the arbitrator can examine it to determine whether to use it or discard it. A response of "I didn't understand" would be an example of a result to discard from full-utterance natural language understanding. The arbitrator implements decision logic to determine whether to use full-utterance-level natural language understanding results or fragment-level results for the portion of the incoming token sequence where the transcribed word sequence is a token sequence.

[0031] To determine word sequences to apply natural language understanding to, some embodiments use heuristics such as identifying interrogative words such as "what," "when," "where," and / or "who" and / or pauses in the detected voice in the audio.

[0032] Some embodiments have an arbitrator that includes other inputs, such as text input, directly without using a speech engine and other input modalities.

[0033] An additional type of output in some embodiments is speech synthesized using text-to-speech (TTS), in which the speech output is generated from a message in an interpretation from natural language understanding.

[0034] Some embodiments, with or without parallel sentence-level natural language understanding, include a dialogue manager that can also control the conversation. For example, if the system needs to know additional information, such as the type of cheese for an item, the user is prompted for the additional information, including but not limited to the type of cheese and / or delivery address. Embodiments that have an arbitrator and / or dialogue manager can use the dialogue manager to select between different arbitration results that are sent to the domain handler.

[0035] The dialog manager can also perform other functions, such as formulating follow-up questions to request information from the user to complete changes to the semantic state.

[0036] Fragment Integration Although the parsing is eager, the intent or interpretation of the current fragment may change depending on future words.

[0037] For example: "delete that" (intended to remove the most recently mentioned item taken out of the conversation state) "delete that <item>" (intended to retrieve the semantic state of <item> and delete it) Pauses from the user can be used to disambiguate multiple possible interpretations of a fragment. Waiting for a pause or a future fragment that clarifies the meaning can avoid making decisions about fragments with meanings that may be disambiguated by future fragments. Those fragments will not be processed by the domain handler until enough information is available for meaning disambiguation. If a new word or fragment is added to the transcription before a pause that changes the meaning of the previous fragment has passed, the domain handler can instead make a decision according to the new context provided by that longer sequence of fragments.

[0038] For example, for the fragment "give me" followed by "a large," the fragment integrator needs to wait until the item is added to the order because there may be multiple items with large as an option, such as a Coke, coffee, or fries. In another example, following the fragments "give me," "pizza," and "mushrooms," the fragment integrator may wait for a period of time before calling the domain handler to add pizza to the semantic state, since pizza may have a list of toppings. The fragment integrator proceeds only after a period during which the user may have finished the sentence, perhaps without intending to add other items besides the mushrooms.

[0039] The look-ahead delay can be based on the user's speech rate. The look-ahead delay can be calculated by dividing the number of words by the duration of the speech, analyzing inter-word delays, and / or analyzing the duration between the identified start and end of one or more words.

[0040] Slot Inference If the transcription matches the start of a fragment containing a slot, the domain handler can invoke a semantic completion handler that displays a list of known possible slot values. The list is removed after the fragment is matched and sent to the handler. For example, after the fragments "please" and "a set," the domain handler might display a list of items that can be part of the set, such as a sandwich, a side dish, and a drink.

[0041] semantic completion A fragment-level natural language understanding system processes the semantics of an utterance in real time, that is, while the utterance is being spoken, rather than after it is completed. The system maintains a continuously updated semantic state of the evolving partial utterance. This semantic state of the system can be used to provide semantically relevant suggestions for each word of the utterance. This differs from traditional systems that provide autocomplete suggestions based solely on word sequence and not semantics. For example, the partial utterance "Add to the first item" will generate different suggestions based on what "first item" actually refers to. Thus, there are different possible suggestions for the same sequence of words, depending on the context. As another example, in the phrase "Add a hamburger with ketchup and without mustard," there are different semantically relevant suggestions at different points in the utterance. After the first two words, "Add a," suggestions could be menu items that have not yet been added to the cart, while after the word "with," suggestions could be hamburger toppings that have not yet been selected. Similarly, after the word "without," the suggestions will be limited to the toppings already selected.

[0042] For example, for "I'd like ...", semantic completion will point to popular items or items previously ordered by the current user. If there are chocolate donuts in the cart and the user says "give me a chocolate...", the user will be shown a shake because there is already a chocolate donut in the cart. However, if the user says "another chocolate...", the additional item will be contingent on the word "another", thereby showing the chocolate donut instead of the chocolate shake.

[0043] In one embodiment, the computer-implemented method is performed by simply matching the transcribed text with a list of trigger phrases, and the associated functionality includes, but is not limited to, displaying a list of menu items and / or displaying a list of options for the most recent menu item in the semantic state.

[0044] It is within the scope of this invention that another embodiment has any individual, including but not limited to a system developer, defining functions based on variable values ​​stored in the semantic state data structure, and the system invokes the functions at run time and / or as precompiled executables and performs semantic completion according to the system developer's definitions.

[0045] End-to-end speech recognition A method for processing fragments in a sequence-to-sequence neural network is described both as a computer-executed method and / or as a computer-readable medium containing instructions that, when executed by a computer, cause the computer to perform the steps of the method. While it is within the scope of the present invention that the method be configured for use with a full vocabulary speech engine, alternative embodiments of the method are contemplated for use with a partial vocabulary speech engine and / or at least a portion of a vocabulary speech engine. For example, the method need not use a full vocabulary speech engine. Instead, the method uses one or more key phrase spotters. These can be implemented as statistical models, such as neural network equivalents, machine learning models, and / or other signal processing designs capable of semantic text comparison. The key phrase spotter takes in audio data and outputs a probability for each of the key phrases that will cause the fragment parser to invoke a domain handler. If the probability exceeds a key phrase threshold, the system invokes the domain handler.

[0046] It is within the scope of the present invention for a recognizer that can be trained or specifically designed for a given set of possible commands to handle commands such as "Make it American cheese" to change the type of cheese on a hamburger. <x>of <y>It is within the scope of the present invention to implement a recognizer that recognizes phrases with slot values ​​such as "change to." A separate recognizer or large vocabulary speech engine can recognize the slot values ​​(of X and Y).

[0047] The neural network directly outputs fragment messages. Fragments are actual words that are matched against "fragment patterns," which in turn are mapped to messages. The messages are processed by domain handlers.

[0048] In another embodiment, the method includes receiving ongoing speech, e.g., a user speaking during an ordering process. The semantic state can be interactively updated according to the ongoing speech. For example, the system will interact with the utterance even before the end of the utterance, such as in the middle of a sentence. Interactively reflecting the semantic state in a user-viewable interface is an embodiment that signals the user's intent on the user's electronic device, including, but not limited to, a tablet and / or a smartphone. For example, the system interacts with the ongoing speech not only at the end of a sentence, but throughout the entire sentence, at intervals, and / or during intermittent and / or continuous monitoring.

[0049] In another embodiment, the semantic state represents an order that includes one item and / or multiple items from a menu or list of items. For example, a consumer may be ordering goods and / or services.

[0050] In another embodiment, one or more items on a menu or list of items may have modifiers, including but not limited to flavors, toppings, sizes, and / or side items, which may themselves recursively contain modifiers.

[0051] In another embodiment, a method includes receiving a portion of a transcription. A fragment and / or fragments within the transcript are identified. The fragments are understood by matching the fragments with a fragment pattern. The fragments have semantic meaning. The semantic state of the system is updated in real time as the sentence is processed fragment by fragment. A display, such as a visual display, is updated to indicate to the user the semantic state of the user's speech. For example, if a consumer orders three items, each of the items will appear in a shopping cart on the consumer's display in real time as each item is spoken and detected by the system.

[0052] These and other important objects, advantages and features of the present invention will become apparent as the description proceeds. [Brief explanation of the drawings]

[0053] [Figure 1] 1 is a schematic block diagram illustrating a system for a natural language understanding environment that updates a display in real time based on detecting the presence of a text fragment in a speech transcript. [Figure 1A] FIG. 2 is a schematic block diagram illustrating one embodiment of the system of FIG. 1 having a fragment consolidator. [Figure 1B] FIG. 1B illustrates an embodiment of the system of FIG. 1A in which a domain handler causes a dialogue manager to update semantic and conversational states. [Figure 1C] FIG. 1C illustrates an embodiment of the system of FIG. 1B in which an arbitrator selects between a natural language understanding output or a fragment consolidator output. [Figure 1D] FIG. 1C illustrates an embodiment of the system of FIG. 1B in which an arbitrator selects between a natural language understanding output or a fragment output from a fragment aggregator. [Figure 2] FIG. 1 illustrates an embodiment of end-to-end speech recognition performed using a machine-learned fragment classifier model. [Figure 3] 1 is a flowchart illustrating a method for processing fragments in a natural language understanding environment. [Figure 3A] 10 is a flowchart illustrating another method for processing fragments in a natural language understanding environment. [Figure 3B] 10 is a flowchart illustrating another method for processing fragments in a natural language understanding environment. [Figure 3C] 10 is a flowchart illustrating another method for processing fragments in a natural language understanding environment. [Figure 3D] 10 is a flowchart illustrating another method for processing fragments in a natural language understanding environment. [Figure 3E] 10 is a flowchart illustrating another method for processing fragments in a natural language understanding environment. [Figure 3F] 10 is a flowchart illustrating another method for processing fragments in a natural language understanding environment. [Figure 3G] 10 is a flowchart illustrating another method for processing fragments in a natural language understanding environment. [Figure 4] 10 is a flowchart illustrating a fragment processing method using a fragment classifier implemented using a sequence-to-sequence neural network. [Figure 4A] 10 is a flowchart illustrating another method for processing fragments using a fragment classifier implemented using a sequence-to-sequence neural network. [Figure 4B] 10 is a flowchart illustrating another method for processing fragments using a fragment classifier implemented using a sequence-to-sequence neural network. [Figure 4C] 10 is a flowchart illustrating another method for processing fragments using a fragment classifier implemented using a sequence-to-sequence neural network. [Figure 4D] 10 is a flowchart illustrating another method for processing fragments using a fragment classifier implemented using a sequence-to-sequence neural network. [Figure 4E] 10 is a flowchart illustrating another method for processing fragments using a fragment classifier implemented using a sequence-to-sequence neural network. [Figure 4F] 10 is a flowchart illustrating another method for processing fragments using a fragment classifier implemented using a sequence-to-sequence neural network. [Figure 5] 1 is a flowchart illustrating a method for processing fragments in a natural language understanding environment. [Figure 6] FIG. 1 is a device diagram illustrating a sequence of operations that can be performed when executing semantic completion software. [Figure 6A] FIG. 1 is a device diagram illustrating a sequence of operations that can be performed when executing semantic completion software. [Figure 6B] FIG. 1 is a device diagram illustrating a sequence of operations that can be performed when executing semantic completion software. [Figure 6C] FIG. 1 is a device diagram illustrating a sequence of operations that can be performed when executing semantic completion software. [Figure 6D] FIG. 1 is a device diagram illustrating a sequence of operations that can be performed when executing semantic completion software. [Figure 6E] FIG. 1 is a device diagram illustrating a sequence of operations that can be performed when executing semantic completion software. [Figure 7] 1 is a flowchart illustrating a method for processing fragments in a natural language understanding environment. DETAILED DESCRIPTION OF THE INVENTION

[0054] The drawings depict various embodiments of the present invention for purposes of illustration only. Those skilled in the art will readily appreciate from the following description that other alternative embodiments of the structures and methods described herein may be employed without departing from the principles of the present invention as described herein.

[0055] Detailed Description FIG. 1 is a block diagram illustrating a system 100 for a natural language understanding environment that updates a display in real time based on detecting the presence of fragments in a real-time speech transcript. The system 100 can include a voice 101, a speech engine 102, a text fragment identifier 103, and a domain handler 107. The speech engine 102 generates a continuous transcription 105 of speech. Sequences of words in the transcription 105, as short as one word, are identified as one or more fragments 106 by the text fragment identifier 103. The fragment pattern 104 is a database of regular expressions. Text in the transcription 105 that matches a regular expression in the fragment pattern 104 is a fragment 106. Once identified, the fragment 106 is processed immediately by the domain handler 107. The domain handler 107 continuously updates 108 a semantic state 109. A cumulative state 111 of understanding the utterance is the output of the updated semantic state 109. The domain handler 107 causes the visual display 110 to be updated 122 in real time as the domain handler 107 immediately processes the fragments 106. Changes in the semantic state 109 can cause display updates, such as an update of the visual display 110.

[0056] 1A is a block diagram illustrating one embodiment of FIG. 1, with a system 112 having a fragment consolidator 113. The fragment consolidator 113 stores fragments 106 for a delay period before an edited fragment stream 114 is processed by a domain handler 107.

[0057] 1B is a block diagram illustrating an embodiment of FIG. 1A with a system 115. A domain handler 107 causes a dialogue manager 117 to update the semantic state 109 and the conversation state 116.

[0058] 1C is a block diagram illustrating an embodiment of FIG. 1B with system 118. Speech engine 102 generates continuous transcription 105a, which is input to natural language understanding 119. Speech engine 102 generates continuous transcription 105b, which is input to text fragment identifier 103. Natural language understanding 119 outputs interpretation data structure 120a. Fragment consolidator 113 outputs edited fragment stream 114a. Arbitrator 121 selects between interpretation data structure 120a from natural language understanding 119 or edited fragment stream 114a from fragment consolidator 113. Arbitrator 121 selects or filters either interpretation data structure 120b or edited fragment stream 114b for output through domain handler 107.

[0059] Figure 1D is a block diagram illustrating an embodiment of Figure 1B with a system 123. An arbitrator 121 selects between an interpretation data structure 120a from natural language understanding 119 or a fragment 106a output from text fragment identifier 103. Heuristics 124 can be used to determine the word sequence to which natural language understanding 119 is applied. Arbitrator 121 outputs interpretation data structure 120a and fragment 106b.

[0060] FIG. 2 shows a diagram of an embodiment of an end-to-end speech recognition system 200 implemented as a machine learning model. Speech audio 201 is input to a fragment identifier 202. The fragment identifier 202 runs a machine learning model on the speech audio 201. The fragment identifier 202 uses the machine learning model to infer the presence of fragments 204 in the speech audio 201. The machine learning model is trained with speech audio samples labeled as corresponding to specific fragments. Supervised learning with requested, curated, or manually labeled audio data is one possible training approach. Unsupervised, semi-supervised, or reinforcement learning model training can also be effective in creating a fragment identification model that infers the presence of key fragments directly from the speech audio.

[0061] The fragment identifier 202 outputs fragments 204 derived from its inferences by a machine learning model 203. The system 200 uses a machine learning model 203, such as a neural network, to convert the speech 201 into fragments without the intermediate step of computing a transcription. The fragment consolidator 205 stores the fragments 204 for a delay period before the edited fragment stream 206 is processed by the domain handler 207. The fragments 204 are processed immediately upon identification by the domain handler 207. The domain handler 207 continuously updates 209 the semantic state 210. A cumulative state 211 of understanding the utterance is the output of the updated semantic state 210. In one embodiment, the domain handler 207 causes updates 212 to a visual display 208 as the domain handler 207 processes the fragment stream 206 in real time. Changes in the semantic state 210 cause display updates, such as to the visual display 208.

[0062] 3 is a flow chart illustrating a method 300 for processing snippets in a natural language understanding environment. The method 300 may include transcribing 301 audio using a speech engine to produce a transcript.

[0063] The method 300 may also include running 302 a fragment identifier on the portion of the transcription, which infers the presence of a fragment in that portion of the speech audio and outputs a fragment resulting from the mapping of the fragment to the fragment.

[0064] The method 300 may further include the step 303 of invoking a domain handler with the fragment in response to detecting the presence.

[0065] 3A is a flow chart illustrating another method 304 of processing fragments in a natural language understanding environment. Method 304 can include the steps of FIG. 3 and a step 305 of updating a conversation state data structure with entity values ​​from the fragment.

[0066] 3B is a flow chart illustrating another method 306 of processing fragments in a natural language understanding environment. The method 306 can include the steps of FIG. 3 and a step 307 in which the domain handler outputs the semantic state, the method further including displaying context-related information suggesting at least one word to speak, the suggestion depending on the semantic state.

[0067] 3C is a flow chart illustrating another method 308 for processing fragments in a natural language understanding environment. The method 308 may include the steps of FIG. 3 and, after detecting the presence, a step 309 of storing the fragment using a fragment aggregator for a delay period after which a call to a domain handler is made.

[0068] 3D is a flow chart illustrating another method 310 for processing fragments in a natural language understanding environment. Method 310 can include the steps of FIG. 3 and step 311 in which the domain handler causes a user interface to make visual changes in real time.

[0069] 3E is a flow chart illustrating another method 312 for processing fragments in a natural language understanding environment. Method 312 can include the steps of FIG. 3 and a step 313 for analyzing the transcription using whole-sentence natural language understanding to generate an interpretation data structure from a larger portion of the transcription, and selecting the fragment instead of the interpretation data structure as the fragment is received by the domain handler at a specified time.

[0070] 3F is a flow chart illustrating another method 314 for processing fragments in a natural language understanding environment. The method 314 can include the steps of FIG. 3 and a step 315 of updating a portion of the semantic state using a domain handler.

[0071] 3G is a flow chart illustrating another method 316 for processing fragments in a natural language understanding environment. The method 316 can include the steps of FIG. 3 and a step 315 of updating a portion of the semantic state using a domain handler.

[0072] The method 316 may also include a step 317 in which the semantic states cause a context-dependent bias in subsequent transcription by the speech engine.

[0073] 4 is a flow chart illustrating a method 400 for processing fragments in a sequence-to-sequence neural network. The method 400 may include a step 401 of running a fragment identifier on a portion of speech, the fragment identifier inferring the presence of a fragment in the portion of speech and outputting a fragment resulting from a mapping of fragments to the fragment.

[0074] The method 400 may also include invoking 402 a domain handler with the snippet in response to the inference.

[0075] 4A is a flow chart illustrating another method 403 for processing fragments in a sequence-to-sequence neural network. Method 403 may include the steps of FIG. 4 and a step 404 of updating a conversation state data structure with entity values ​​from the fragment.

[0076] 4B is a flow chart illustrating another method 405 for processing snippets in a sequence-to-sequence neural network. The method 403 may include the steps of FIG. 4 and a step 406 in which the domain handler outputs the semantic state, the method further including displaying context-related information suggesting at least one word to speak, the suggestion depending on the semantic state.

[0077] 4C is a flow chart illustrating another method 407 for processing fragments in a sequence-to-sequence neural network. The method 407 may include the steps of FIG. 4 and, after detecting presence, a step 408 of storing the fragment using a fragment aggregator for a delay period after which a call to a domain handler is made.

[0078] 4D is a flow chart illustrating another method 409 for processing fragments in a sequence-to-sequence neural network. Method 409 can include the steps of FIG. 4 and step 410 in which the domain handler causes the user interface to make visual changes in real time.

[0079] 4E is a flow chart illustrating another method 414 for processing fragments in a sequence-to-sequence neural network. The method 414 can include the steps of FIG. 4 and a step 415 of updating a portion of the semantic state using a domain handler.

[0080] 4F is a flow chart illustrating another method 416 for processing fragments in a sequence-to-sequence neural network. The method 416 can include the steps of FIG. 4 and step 417 in which semantic states create a context-dependent bias for the fragment classifier.

[0081] 5 is a flow chart illustrating a method 500 for processing snippets in a sequence-to-sequence neural network. The method 500 may include transcribing 501 audio using a speech engine to produce a transcript.

[0082] The method 500 may also include running 502 a fragment identifier on the portion of the transcription, the fragment identifier detecting the presence of a first fragment and outputting the first fragment.

[0083] The method 500 may also include invoking 503 the domain handler with a fragment selected from the group consisting of the first fragment and the second fragment in response to detecting the presence.

[0084] Method 500 may also include step 504 of invoking a domain handler with the second fragment in response to detecting the presence of the second fragment during the delay period, which emphasizes the concept of waiting and discarding the first fragment if a better second fragment appears.

[0085] 6 is a diagram of a display of a device 606 illustrating a sequence of actions 600 that may occur during execution of the fragment processing software. Semantic completions are indicated, thereby updating the semantic state 604 and displaying it to the user in real time. The device 606 has a user interface 601. A transcript 602 is a transcription of a portion of the user's speech in real time. In one example, the order item 605 is kids' mac and cheese. Based on the transcript 602 requesting kids' mac and cheese without the expected drink and side choice options yet being specified, a list of suggested drink choices 603a and side choices 603b is displayed to indicate that the possible drink choices 603a are apple juice or milk, and the possible side choices 603b are French fries or yogurt.

[0086] 6A is a diagram illustrating a display of a device 606 showing a sequence of actions 607 that may occur during execution of the fragment parsing software. Semantic completions are indicated, thereby updating the semantic state 604 and displaying it to the user in real time. The device 606 has a user interface 601. A transcript 602 is a transcription of a portion of the user's speech in real time. In one example, the order item 605 is kids' mac and cheese with a side choice of yogurt 608a. Based on the transcript 602, a list of suggestions is displayed to indicate possible drink choices 603a are apple juice and / or milk.

[0087] 6B is a diagram of a display of device 606 illustrating a sequence of actions 609 that may be performed during execution of the fragment parsing software. Semantic completions are indicated, thereby updating the semantic state 604 and displaying it to the user in real time. Device 606 has a user interface 601. Transcription 602 is a transcription of a portion of the user's speech in real time. In one example, order items 605 are kids' mac and cheese with a side choice 608a that is yogurt and a drink choice 608b that is apple juice.

[0088] FIG. 6C is a diagram of a display of device 606 illustrating a sequence of actions 610 that may occur during execution of the fragment parsing software. Semantic completions are indicated, thereby updating the semantic state 604 and displaying the semantic completion to the user in real time. Device 606 has a user interface 601. Transcription 602 is a transcription of a portion of the user's speech in real time. In one example, if transcript 602 includes fragment 611, "dessert," a unique dessert menu 612 appears. In one example, order item 605a is kids' mac and cheese with a side choice 608a that is yogurt and a drink choice 608b that is apple juice. Order item 605b is Sprite with a size selection 613 that is large. Order item 605c is sweet potato fries. The dessert menu is presented by the domain handler in response to the transcription matching the fragment pattern "dessert." In various embodiments, the dessert menu continues to be displayed until another fragment pattern is matched that will invoke a special pop-up box, or until a certain period of time has elapsed, or until the transcription matches a fragment that is a specific dessert item, or until the beginning or end of the combination of criteria for removing the dessert menu pop-up box.

[0089] FIG. 6D is a diagram of a display of device 606 illustrating a sequence of actions 614 that may occur during execution of the fragment parsing software. Semantic completions are indicated, thereby updating the semantic state 604 and displaying it to the user in real time. Device 606 has a user interface 601. Transcription 602 is a transcription of a portion of the user's speech in real time. In one example of an edit, if transcript 602 contains the fragment "change it" followed by a clear reference to an item in the semantic state (order list) of "Kids Mac & Cheese," the system highlights that item. If transcript 602 also contains the word "change it" 615, the system adds an edit. If the edit mentions a specific choice of "milk" available for drink choice 608a "Drink," the system replaces the existing selection of the "apple juice" option with the specified choice of the "milk" option. Order item 605d is a brownie. Order item 605e is a chocolate chip cookie.

[0090] FIG. 6E is a diagram of a display of device 606 illustrating a sequence of actions 616 that may occur during execution of the fragment parsing software. A semantic completion is indicated, which updates the semantic state 604 and displays ordered items 605a-605e to the user in real time. Device 606 has a user interface 601. Transcription 602 is a transcription of a portion of the user's speech in real time. In one example of completion, when the user utters the command "That's it" 617, the system displays a message 618 listing the items in the order and also outputs text 619 as text-to-speech for the user to hear. During text-to-speech, device 606 reads the words on the screen to the user. Selected option 608a is yogurt, and selected option 608b is milk.

[0091] At the end of action sequence 616, the user has completed their meal order at the restaurant. The system then charges the user's credit card for all items in the user's virtual shopping cart. The user may then pay for the ordered items. Action sequence 616 shows a total of $24 USD. In some embodiments, payment may be made in other currencies, including virtual token currencies stored on a blockchain such as Bitcoin or other distributed ledger.

[0092] 7 is a flow chart illustrating a method 700 for processing snippets in a natural language understanding environment. The method 700 may include transcribing 701 audio using a speech engine to produce a transcript.

[0093] The method 700 may also include a step 702 of matching snippets in the transcript with snippet patterns.

[0094] The method 700 may also include extracting 703 sentences from the transcript.

[0095] The method 700 may also include performing 704 full-sentence natural language understanding on the sentence to identify the sentence as understandable and generate an interpretation data structure.

[0096] The method 700 may also include a step 705 of selecting an interpretation data structure and discarding the fragment in response to identifying the sentence as parsable.

[0097] The method 700 may also include a step 706 of invoking a domain handler with the interpretation data structure.

[0098] In some embodiments, a platform is provided to different product suppliers, such as restaurants, to implement a particular menu for the restaurant. Such a platform may include a text or graphical configuration interface that allows the supplier to input or upload a list of the supplier's menu items. For each type of item, the configuration tool may accept a set of configurable options for that type and a price for each option.

[0099] Finally, it should be noted that the language used herein has been chosen primarily for readability and instructional purposes, and not to limit or restrict the subject matter of the present invention. Accordingly, the described embodiments are illustrative of the present invention and are not exhaustive. Appendix 1 Pseudocode for the Dialogue Manager This appendix presents pseudocode for a dialogue manager that can implement functions including, but not limited to, formulating follow-up questions to request information from the user to complete changes to the semantic state. / / Parameters barge_in_delay = 3 seconds / / Time for TTS to play before being allowed to barge in threshold_A = ? / / Pause from the last token until full domain natural language understanding is invoked threshold_B = ? / / Time until the last update before TTS output is sent / / Initialization state = {} / / empty state head = "" / / empty partial_query = "" / / empty last_match_time = current_time last_time_time = -1; change=false; no_update_time = 0; / / loop while (streaming) { if (text input via touch interface) { change = false (new_result, new_state) = full_match ( (text_query) , state) partial_query = ""; head = "" / / Reset to empty state = new_state / / New state from match last_match_time = -1 / / Important so TTS is not triggered by threshold B if the user is tapping (may need threshold C to eventually trigger if the user does not take any action) Continue while looping } pause_time = current_time - last_token_time; if (new_token) { partial_query . = new_token last_token_time = new_token_time; change=true } if change == true { if (fragmented_parsing has good results) { / / Proceed to update cart change = false (new_result, new_state) = full_match ( "what is in my cart") , state) / / Note that this synchronizes the state and the fragment parse result with the full match partial_query = ""; head = "" / / Reset to empty state = new_state / / New state from match last_match_time = current_time } else if pause_time > threshold_A { change = false (new_result, new_state) = full_match ( (head . partial_query) , state) if (new_result is good) <--Note that a match alone is not enough, you need to detect if it is a good match (not "Sorry, I don't understand"). partial_query = ""; head = "" / / Reset to empty state = new_state / / New state from match last_match_time = current_time else head = [ 1000 head . partial_query ] state = state / / No change } } if last_match_time > 0 { no_update_time = current_time - last_match_time; / / Note this does not update until the first match } if no_update_time > threshold_B { speak ( new_result.spoken_response, barge-in_delay) / / Note that this is the new result, good or bad, so if the user says something wrong and waits for a response, they will get a response to their bad input. partial_query = ""; head = "" / / Reset to empty state = new_state / / New state from match last_match_time = current_time } }< / y> < / x>

Claims

1. 1. A computer-implemented method comprising: transcribing the audio using a speech engine to produce a transcription; running a fragment classifier on the portion of the transcription, the fragment classifier detecting fragments and outputting the fragments; The method comprises: The method further comprising invoking a domain handler with the fragment in response to detecting the fragment.

2. The method of claim 1 , further comprising updating a conversation state data structure with entity values ​​from the fragment.

3. 10. The method of claim 1, wherein the domain handler outputs a semantic state, the method further comprising displaying context-related information that suggests at least one word to speak, the suggestion depending on the semantic state.

4. The method of claim 1 , further comprising, after detection, storing the fragments using a fragment consolidator for a delay period, wherein the invocation of the domain handler occurs after the delay period.

5. The method of claim 1 , wherein the domain handler causes a user interface to make visual changes in real time.

6. The method of claim 1 , further comprising using the domain handler to update a portion of a semantic state.

7. The method of claim 6 , wherein the semantic states result in context-dependent biasing of subsequent transcription by the speech engine.

8. 1. A computer-implemented method comprising: transcribing the audio using a speech engine to produce a transcription; matching fragments in the transcript with fragment patterns; extracting sentences from the transcription; performing full-sentence natural language understanding on the sentence to identify that the sentence is understandable and generating an interpretation data structure; in response to identifying the sentence as parsable, selecting the interpretation data structure and discarding the fragment; and invoking a domain handler with the interpreted data structure.

9. 1. A computer-implemented method comprising: running a fragment identifier on a portion of speech, the fragment identifier inferring the presence of a fragment within the portion of speech and outputting the fragment resulting from the mapping of fragments to the fragment; The method comprises: The method further comprising invoking a domain handler with the fragment in response to the inference.

10. 10. The method of claim 9, further comprising updating a conversation state data structure with entity values ​​from the fragment.

11. 10. The method of claim 9, wherein the domain handler outputs a semantic state, the method further comprising displaying context-related information that suggests at least one word to speak, the suggestion depending on the semantic state.

12. 10. The method of claim 9, further comprising, after detecting the presence, storing the fragments using a fragment consolidator for a delay period, wherein the invocation of the domain handler occurs after the delay period.

13. The method of claim 9 , wherein the domain handler causes a user interface to make visual changes in real time.

14. The method of claim 9 , further comprising updating a portion of a semantic state using the domain handler.

15. The method of claim 14 , wherein the semantic state induces a context-dependent bias of the fragment classifier.

16. 1. A computer-implemented method comprising: transcribing the audio using a speech engine to produce a transcription; and running a fragment identifier on the portion of the transcription, the fragment identifier detecting the presence of a first fragment and outputting the first fragment; The method comprises: The method further includes invoking a domain handler with a fragment selected from the group consisting of the first fragment and the second fragment in response to detecting the presence.

17. 17. The method of claim 16, further comprising updating a conversation state data structure with entity values ​​from the fragment.

18. 17. The method of claim 16, wherein the domain handler outputs a semantic state, the method further comprising displaying context-related information that suggests at least one word to speak, the suggestion depending on the semantic state.

19. 17. The method of claim 16, further comprising, after detecting the presence, storing the fragments using a fragment consolidator for a delay period, wherein the invocation of the domain handler occurs after the delay period.

20. The method of claim 16, wherein the domain handler causes a user interface to make visual changes in real time.

21. analyzing the transcription using whole-sentence natural language understanding to generate an interpretation data structure from a larger portion of the transcription; selecting the fragment in place of the interpretation data structure as the fragment is received by the domain handler at a specified time; 17. The method of claim 16, further comprising:

22. The method of claim 16 , further comprising updating a portion of a semantic state using the domain handler.

23. 23. The method of claim 22, wherein the semantic states result in context-dependent biasing of subsequent transcription by the speech engine.

24. A computer-readable medium containing instructions that, when executed by a computer, cause the computer to perform steps, the steps including: transcribing the audio using a speech engine to produce a transcription; running a fragment classifier on the portion of the transcription, the fragment classifier detecting fragments and outputting the fragments; The steps include: The computer-readable medium further comprising invoking a domain handler with the fragment in response to detecting the fragment.

25. 25. The computer-readable medium of claim 24, wherein the steps further include updating a conversation state data structure with entity values ​​from the fragment.

26. 25. The computer-readable medium of claim 24, wherein the domain handler outputs a semantic state, and the step further comprises displaying context-related information suggesting at least one word to speak, the suggestion depending on the semantic state.

27. 25. The computer-readable medium of claim 24, wherein the steps further include, after detection, storing the fragments using a fragment consolidator for a delay period, the invocation of the domain handler occurring after the delay period.

28. 25. The computer-readable medium of claim 24, wherein the domain handler causes a user interface to make visual changes in real time.

29. analyzing the transcription using whole-sentence natural language understanding to generate an interpretation data structure from larger portions of the transcription; selecting the fragment in place of the interpretation data structure as the fragment is received by the domain handler at a specified time; 25. The computer-readable medium of claim 24, further comprising:

30. 26. The computer-readable medium of claim 25, further comprising updating a portion of a semantic state using the domain handler.

31. 31. The computer-readable medium of claim 30, wherein the semantic states result in context-dependent biasing of subsequent transcription by the speech engine.

32. A computer-readable medium containing instructions that, when executed by a computer, cause the computer to perform steps, the steps including: running a fragment identifier on a portion of speech, the fragment identifier inferring the presence of a fragment within the portion of speech and outputting the fragment resulting from the mapping of fragments to the fragment; The steps include: In response to the inference, the steps further include invoking a domain handler with the fragment.

33. 33. The computer-readable medium of claim 32, wherein the steps further include updating a conversation state data structure with entity values ​​from the fragment.

34. 33. The computer-readable medium of claim 32, wherein the domain handler outputs a semantic state, and wherein the step further includes displaying context-related information that suggests at least one word to speak, the suggestion depending on the semantic state.

35. 33. The computer-readable medium of claim 32, wherein the steps further include storing the fragments using a fragment consolidator for a delay period after detecting the presence, the invocation of the domain handler occurring after the delay period.

36. 33. The computer-readable medium of claim 32, wherein the domain handler causes a user interface to make visual changes in real time.

37. 34. The computer-readable medium of claim 33, wherein the steps further comprise updating a portion of a semantic state using the domain handler.

38. 38. The computer-readable medium of claim 37, wherein the semantic state induces a context-dependent bias of the fragment identifier.

Images