Real-time natural language processing and implementation

Abstract

A system and method for real-time feedback acknowledgement for requesting a virtual assistant response from an evolving semantic state of at least a portion of an utterance. A user accesses a virtual assistant on an electronic device having systems and / or methods configured to capture commands, questions, and / or fulfillment requests from audio (e.g., speech made by a speaking user). The speech may be intercepted by a speech engine configured to transcribe the speech into text that matches a regular expression of a schema of segments to generate segments, and / or the speech may be processed with a machine learning model to identify segments. The segment is identified by a domain handler configured to update a data structure of a current semantic state of the utterance in real-time on an interface of the electronic device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application Ser. No. 18 / 055,821, filed on November 15, 2022, entitled “REAL-TIME NATURAL LANGUAGE PROCESSING AND FULFILLMENT,” the entire contents of which are incorporated herein by reference. Background Art

[0003] Conventional natural language understanding system technologies perform updates at the end of each sentence, after the entire sentence is recognized as a unit. For example, some natural language understanding technologies interpret the meaning of the entire sentence length after the entire sentence is received by the system. Currently, technologies for calculating the change in the system's semantic state as a result of an utterance only occur when the end of the utterance is detected. Summary of the Invention

[0004] Methods for processing segments in a natural language understanding environment are described, for use in both a computer-implemented method and / or a computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method. One embodiment includes using a speech engine to perform acoustic speech recognition. The speech engine generates a continuous transcript of the speech. For example, a user accesses a virtual assistant using an electronic device having a visual display that is configured to capture commands, questions, and / or fulfillment requests from audio, including but not limited to the user's speech. Although one embodiment includes a continuous transcript, in some embodiments, the transcript can be substantially continuous, intermittent, and / or have delays and / or pauses.

[0005] In one embodiment, the transcript of the audio is inferred by a speech engine. The speech engine is configured to receive input speech and generate a text transcript. Specifically, the speech engine is configured to transcribe at least a portion of the speech, including but not limited to a fragment of a sentence, a word, at least a portion of a word, and / or at least a portion of an utterance. The system uses the speech engine to transcribe the audio into text to form a transcript. The fragment is then sent to a fragment integrator. The rules of the fragment integrator define the sequence of fragments that the fragment integrator should look for, and the rules include what (one or more) messages to output if the rule matches.

[0006] The fragment integrator sends the original fragment to the domain handler, or the integrator matches a rule to the fragment sequence, triggering the rule to send a message to the domain handler. The system's contextual relevance, combined with fragment parsing, updates the semantic state to include the command spoken so far. Thus, mid-utterance updates to the context enable suggestions to the user on how to complete their thought, thereby completing the user's mid-sentence.

[0007] In another embodiment, a simpler system without segment integration can be used, as long as the application relies only on the raw segment stream. For example, in some implementations, there may be no segment integrator. Therefore, segments are sent directly to the domain handler. The domain handler outputs its continuously updated semantic state, which is the cumulative semantic understanding of the utterance processed so far.

[0008] A database is provided. The database contains fragment patterns. Fragment patterns are short segments of text. The fragment patterns can be compared to at least one word of the transcript, and the resulting fragments are identified and generated by a fragment identifier. It is within the scope of the present disclosure that the fragment identifier includes, but is not limited to, a sequence-to-sequence neural network, a common machine learning model. The fragment identifier infers the fragments directly from the audio.

[0009] A sequence of at least one word in a transcript from at least a portion of a user utterance that matches a regular expression of a segment pattern may form a segment. A segment in the transcript (as short as a portion of a word) is identified by a segment identifier as a "segment". It is within the scope of the present disclosure that a sequence of at least one word in the transcript has at least one word that is identified by the segment identifier as a "segment" of text. For example, the segment identifier identifies text from the transcript and associates it with an intent to output a segment for recognition by a domain handler. For example, words and / or word combinations that are not identified as "segments" are ignored. Some examples of segments are "classic cheeseburger" or "one dozen" or "chocolate brownie bar", which can be matched by the segment identifier using a segment pattern, for example:

[0010] a.["classic"]("cheeseburger"|"hamburger")

[0011] b."twelve"|"dozen"|"one dozen"

[0012] c.(["chocolate"].(("brownie".["bar"|"bars"])|"brownies"))

[0013] As segments are recognized by the "domain handler," they are processed immediately and / or concurrently, with the domain handler updating a data structure representing the current semantic state of a potentially ongoing long-form utterance. In this way, utterance understanding progresses incrementally based on recognized segments, rather than all at once at the end of a short utterance. Some examples of variables and legal values ​​in the semantic state data structure are:

[0014] a.INTENT=DELETE,FLAVOR=chocolate,ITEM=shake

[0015] b.INTENT=CHANGE,TARGET=first_item,ITEM=cheeseburger,TOPPING=onions

[0016] c.INTENT=ADD,SIZE=large,ITEM=fries,TOPPING=chili

[0017] Various aspects of the semantic state represented in the domain handler can be streamed to the user in real time. This can be used to demonstrate to the user that the system's understanding is correct at any time during the utterance and / or when the system detects at least a portion of the utterance and / or before the utterance is complete. If the system's understanding is incorrect, the user is quickly alerted so they can make adjustments. It is within the scope of the present disclosure to not be limited to a specific percentage of the utterance. The detection of at least a portion of the utterance can begin when the user utters the sound of at least a portion of the utterance and end at any time after the start, so that the system can detect the segment.

[0018] While the user is still in the utterance, the semantic state represented by the domain handler can be used to act on the intent or part of the intent that has been understood.

[0019] Alternatively, the domain handler can simply process all fragments, taking action immediately, depending on the application, with little or no updated semantic state. An example is a series of commands in a statement, where each subcommand is sequenced by fragment and processed immediately without tracking semantic state.

[0020] The fragment integrator can perform additional processing on the fragments before they are processed by the domain handler. The fragment integrator will wait and / or pause before sending the fragments to the domain handler until the fragment integrator has an opportunity to identify subsequent fragment changes and / or disambiguate the user's meaning or intent. When such a sequence is detected, the fragments can be modified, deleted, and / or processed in a different order. In addition, additional messages can be inserted before being sent to the domain handler to clarify the meaning of the fragment collection.

[0021] The semantic state and the results of the domain processing can influence or bias subsequent transcripts of acoustic speech recognition, both in the initial and additional steps. This context-dependent influence can be used to improve the accuracy of the evolving transcript by biasing the acoustic or linguistic scores used by the speech engine and / or some other related means, based on the evolving semantic state in the domain processing. This is an improvement over conventional techniques, which only update each utterance at the end of the utterance after it has been understood as a unit.

[0022] Additional Notes

[0023] It is within the scope of the present disclosure that at least one word is formed by at least one character, including but not limited to letters, numbers, alphanumeric characters, common punctuation marks (such as "." and / or "-"), and spaces.

[0024] It is within the scope of the present disclosure that alphanumeric characters consist of both letters and numbers and / or other symbols, including but not limited to punctuation marks and / or mathematical symbols.

[0025] It is within the scope of this disclosure that audio includes, but is not limited to, speech, voice, spoken words, statements, and / or human voice.

[0026] In some embodiments, the domain handler maintains a conversation state data structure. While the semantic state data structure can contain many items of the same or different types, the conversation state stores the most recent referenced entity for any type. This is useful for functions such as pronoun disambiguation. Conversation state is also more volatile because values ​​become outdated over time as the conversation progresses. Some embodiments remove values ​​from the conversation state after a specific period of time.

[0027] As a result of matching segments in the transcript to segment patterns, domain handlers, dialog state changes, and / or semantic state changes can cause real-time display updates and / or changes. It is within the scope of the present disclosure for a display to signal the state of a domain handler output to a user, including but not limited to visual displays, vibrations, light-emitting elements, mechanical activation, and / or auditory feedback. In one example, auditory feedback can be a phrase such as "hmm" and / or any suitable non-verbal signal. For example, the visual display can be part of an electronic device having a user interface, such as a text message visible to the user on a screen of the user interface. In another example, the display can be a plurality of light-emitting elements configured to illuminate in a predetermined pattern based on the status update. In another example, the display can be a series of vibrations corresponding to the status update. In another example, the update can be displayed as an audio output, such as using text-to-speech (TTS). In one example, the displayed update can also cause a motor (e.g., a motor that adjusts the position of a robotic arm) to move. In one example, the displayed update can be a virtual action in a simulation (e.g., a video game and / or a virtual reality environment such as the Metaverse).

[0028] Some examples of electronic devices include mobile devices such as automobiles; portable devices such as smartphones, tablets, and laptops; stationary devices such as kiosks and vending machines; and appliances such as refrigerators and water coolers.

[0029] It is within the scope of this disclosure that speech includes, but is not limited to, articulated and / or inarticulate sounds.

[0030] It is within the scope of the present disclosure that an automatic speech recognition engine includes, but is not limited to, a machine learning model and / or a neural network configured to uniquely map input from the following items, wherein the items include, but are not limited to, a word, at least a portion of a word, at least a portion of an utterance, a sentence, a segment, text, audio and / or video to an output, wherein the output includes, but is not limited to, a transcript, a recognized segment and / or an interpreted data structure.

[0031] A fragment pattern can be a representation such as plain text and / or a regular expression, and a specific text that matches a regular expression of a fragment pattern is a "fragment." Each regular expression fragment pattern can match one or more actual word sequences. The actual sequence that matches is a fragment. A fragment can be as small as at least a portion of a word. A fragment can have slots, which can be archived by calling different instances of the same fragment.

[0032] The domain handler can update the semantic state. The domain handler can update the dialog state data structure using information from the snippet. In another embodiment, the domain handler can interact with a session manager. The session manager maintains both dialog state and semantic state. The dialog state tracks things like entity values ​​needed to disambiguate pronouns, and the semantic state is, for example, a list of items in a shopping cart displayed as an element in a visual display. In the pronoun disambiguation example, if there are five items in the shopping cart and the user says "delete it," the scope of the deleted items will be limited to the single most recently mentioned item.

[0033] Domain handlers receive "snippets". In one example, the snippets "change...", "make...", "replace...", "instead of..." all map to the snippet "INTENT.MODIFY". Domain handlers are capable of accepting multiple types of input, although in practice software engineers may decide to create abstract objects that cover all types of input, or they give domain handlers an interface that accepts different types of messages. For example, three types of messages that a domain handler may receive include, but are not limited to: 1) "raw" snippets that the integrator simply passes through, 2) messages sent from the snippet integrator when it detects a language pattern (it has rules for this), and 3) natural language understanding data structures that represent semantic information about the entire sentence.

[0034] Examples of three types of messages that a domain handler might receive:

[0035] 1)INTENT.MODIFY

[0036] 2) MODIFY target=second_item option=mustard action=add

[0037] 3)INTENT=info_question,TOPIC=dessert_menu

[0038] The fragment recognizer identifies the fragments and learns about their related fragments, which are passed to the fragment integrator, which then passes the original fragment to the domain handler, or the fragment integrator composes other messages to pass to the domain handler.

[0039] The fragment integrator is configured to detect a pattern such as "add OPTION to ITEM" (where OPTION is a fragment that matches the fragment "mustard", such as "TOPPING.MUSTARD", and ITEM is a fragment such as "ITEM.CHEESEBURGER"). The integrator matches this rule with the incoming fragment and outputs a message or series of messages to the domain handler to implement the addition of a topping to the menu item in this case.

[0040] Some embodiments include an additional step of analyzing the transcript using sentence-level natural language understanding. This occurs in parallel with the text segment identifier. The domain handler is preceded by an arbitrator, which selects between the segment (or edited segment stream) and / or the interpretation produced by the natural language understanding. If the natural language understanding function indicates a successful interpretation of the transcript, the arbitrator selects the natural language understanding result. The arbitrator receives three types of input: a natural language understanding data structure, a summary message from the integrator when the integrator matches a rule, or the original segment. Both the arbitrator and the domain handler need to know how to handle all three types of messages. If the arbitrator receives only one type of input and no other type within a specific time period, the arbitrator passes its input on. Otherwise, if there are two or more inputs within the same time period, the arbitrator selects which one to output to the domain handler. Whatever is selected for output is not converted. For example, the arbitrator determines whether the domain handler processes the full sentence natural language understanding result or the segment integrator natural language understanding result.

[0041] In one embodiment, a separate algorithm can be used to identify entire utterances or sentences, for example, by looking for question words at the beginning of sentences and pauses at the end of sentences. A full utterance-level natural language understanding engine is used to generate natural language understanding interpretations. A decision-making process in the arbitrator can choose whether to utilize full utterance-level natural language understanding results or process segment-level natural language understanding results. For example, natural language understanding results for an utterance that cannot be understood will be discarded and segment-level natural language understanding responses will be used instead.

[0042] In another embodiment, the transcript is analyzed to identify word-gram sequences, which are assumed to be full sentences and are then processed by full-utterance natural language understanding. More generally, any extended word-gram sequence can be identified, whether forming a single sentence or multiple sentences, or even less than a sentence, such as a clause of an entire sentence. An example is to look for question words (by performing explicit word-gram matching, or by capturing question words as fragments), and then look for pauses, and send the word-gram sequence from the question words to the pause to the full-sentence natural language understanding. Then, after receiving the full-sentence natural language understanding response, the arbitrator can view it to decide whether it should be used or discarded. A response of "didn't get..." would be an example of discarding a result from the full-utterance natural language understanding. The arbitrator implements decision logic to decide whether to use the full-utterance level natural language understanding result or to use the fragment-level result for that portion of the incoming word-gram sequence, where the sequence of transcribed words is the word-gram sequence.

[0043] To determine the word sequence to which to apply natural language understanding, some embodiments use heuristics, such as identifying question words, such as "what," "when," "where," and / or "who," and / or pauses in speech detected in the audio.

[0044] Some embodiments have an arbitrator that includes other inputs, such as directly entered text, without using a speech engine and other input methods.

[0045] In some embodiments, the additional type of output is voice audio synthesized using text-to-speech (TTS).In some such embodiments, the voice output is generated by a message from a natural language understanding interpretation.

[0046] Some embodiments include a conversation manager that can also control the conversation, with or without concurrent sentence-level natural language understanding. For example, if the system needs to know additional information, such as the type of cheese an item is, the user can be prompted for additional information, including but not limited to the type of cheese and / or a shipping address. Embodiments with an arbitrator and / or conversation manager can use the conversation manager to select between different arbitration results to be sent to the domain handler.

[0047] The session manager can perform other functions, such as composing follow-up questions to request information from the user, thereby completing the change to the semantic state.

[0048] Fragment Integration

[0049] Parsing is urgent, but the intent or interpretation of the current segment may change with future words.

[0050] For example:

[0051] "Delete..." (meaning to delete the previously mentioned item retrieved from the conversation state)

[0052] "Delete <item>" (meaning to search for the semantic state of <item> and delete it)

[0053] A pause from the user can be used to disambiguate multiple possible interpretations of a segment. By waiting for a pause or a future segment to clarify its meaning, it is possible to avoid taking action on a segment whose meaning might be disambiguated by a future segment. The domain handler will not process the segment until it has enough information to disambiguate its meaning. If a new word or segment is added to the transcript before the pause has passed that changes the meaning of the previous segment, the domain handler can decide to take action instead based on the new context provided by the longer segment sequence.

[0054] For example, for a snippet like "give me" followed by "large," the snippet integrator must wait to add the item to the order because multiple items might have large options, such as cola, coffee, or fries. In another example, after the snippets "give me," "pizza," and "mushrooms," the snippet integrator might wait for a while before calling the domain handler to add the pizza to the semantic state. This is because a pizza might have a list of toppings. The snippet integrator only proceeds after a period of time, when the user might have already completed their sentence and not intended to add items other than mushrooms.

[0055] The lookahead delay can be based on the user's speech speed. The lookahead delay can be calculated by dividing the number of words by the speech time period, analyzing the inter-word delay, and / or analyzing the time period between the recognized start and end of one or more words.

[0056] Time Slot Inference

[0057] When a transcript matches the beginning of a segment containing a slot, the domain handler can call a semantic completion handler, which displays a list of known possible slot values. Once a segment is matched and sent to the handler, the list is removed. For example, after the segments "give me" and "meal," the domain handler can display a list of items that are part of the meal, such as a sandwich, a side dish, and a drink.

[0058] Semantic Completion

[0059] Segment-level natural language understanding systems process the semantics of an utterance in real time, that is, during the utterance itself, rather than after it is completed. The system maintains a continuously updated semantic state of the evolving partial utterance. This semantic state of the system can be used to provide semantically relevant suggestions for each word of the utterance. This contrasts with conventional systems that provide auto-complete suggestion features based solely on word sequence, rather than semantics. For example, the partial utterance "for the first item add" will generate different suggestions based on the actual meaning of "first item." Thus, different possible suggestions are available for the same sequence of words, depending on the context. Another example is the phrase "add a burger with ketchup but no mustard," where different semantically relevant suggestions are available at different points in the utterance. After the first two words, "add one," the suggestions might be menu items not yet in the cart, while after the word "with" the suggestions might be burger toppings that have not yet been selected. Similarly, after the word "without," the suggestions would be limited to toppings that have already been selected.

[0060] For example, for "I want...", semantic completion displays popular items or items previously ordered by the current user. If there's a chocolate donut in the cart and the user says "Give me a chocolate...", the milkshake will be displayed to the user because there's already a chocolate donut in the cart. However, if the user says "Another chocolate...", the added item is conditional on the word "another," which will result in the chocolate donut being displayed instead of the chocolate milkshake.

[0061] In one embodiment, the computer-implemented method is performed by simply matching the transcribed text to a list of trigger phrases and related functions, including but not limited to displaying a list of menu items and / or displaying a list of options for the latest menu item in a semantic state.

[0062] It is within the scope of the present disclosure that alternative embodiments allow any individual (including but not limited to system developers) to define functions based on variable values ​​stored in the semantic state data structure. The system calls the functions at runtime and / or as a precompiled executable file and performs semantic completion according to the system developer's definition.

[0063] End-to-end audio recognition

[0064] A method for processing segments in a sequence-to-sequence neural network is described, the method being for both a computer-implemented method and / or a computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method. While it is within the scope of the present disclosure to configure the method for use with a full-vocabulary speech engine, in alternative embodiments of the method, it is contemplated that it may be configured for use with portions and / or at least a portion of a vocabulary speech engine. For example, the method may not use a full-vocabulary speech engine. Instead, the method uses one or more key phrase spotters. These may be implemented as, but are not limited to, statistical models, such as neural network equivalents, machine learning models, and / or other signal processing designs capable of performing semantic text comparison. The key phrase spotter receives audio data and outputs a probability for each key phrase that will cause the segment parser to call a domain handler. When the probability exceeds a threshold for a key phrase, the system calls the domain handler.

[0065] It is within the scope of the present disclosure that a recognizer can be trained or designed specifically for a given set of possible commands to process, such as "make American cheese" to change the type of cheese on a burger. <x>Change to <y>A recognizer for phrases such as "") is also within the scope of the present disclosure. A separate recognizer or a large vocabulary speech engine can recognize the time slot values ​​(for X and Y).

[0066] The neural network directly outputs a fragment message. While the fragment is the actual word that matches the "fragment pattern," the fragment pattern is mapped to a message. This message is processed by the domain handler.

[0067] In another embodiment, the method includes receiving ongoing speech. For example, a user speaks during an ordering process. The semantic state can be interactively updated based on the ongoing speech. For example, the system interacts with the speech even before the end of the speech (e.g., mid-sentence). Interactively reflecting the semantic state in the user-visible interface is an embodiment in which the user's intent is signaled on the user's electronic device (e.g., including but not limited to a tablet and / or smartphone). For example, the system interacts with the ongoing speech at intervals throughout the sentence and / or during intermittent and / or continuous monitoring, rather than only at the end of the sentence.

[0068] In another embodiment, the semantic state represents an order for an item and / or multiple items in a menu or a list of items. For example, a consumer may have ordered goods and / or services.

[0069] In another embodiment, an item and / or multiple items from a menu or item list may have a qualifier. Such qualifiers may include, but are not limited to, flavor, ingredients, size, and / or side dish items, which may themselves recursively include qualifiers.

[0070] In another embodiment, the method includes receiving a portion of a transcript. Identifying a segment and / or multiple segments in the transcript. Understanding the segments by matching the segments to segment patterns. The segments have semantic meaning. As the sentence is processed segment by segment, the semantic state of the system is updated in real time. Updating a display (e.g., a visual display) to show the user the semantic state of their speech. For example, if a consumer orders three items, each of the three items will appear in a shopping cart on the consumer's display in real time as each item is spoken and detected by the system.

[0071] These and other important objects, advantages and features of the present disclosure will become clear as the specification proceeds. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a high-level block diagram illustrating a system for updating a displayed natural language understanding environment in real time based on detecting the presence of a text segment in a speech transcript;

[0073] Figure 1A is shown with a fragment integrator Figure 1 A high-level block diagram of an embodiment of the system in FIG.

[0074] Figure 1B yes Figure 1A A schematic diagram of an embodiment of a system in which a domain handler causes a conversation manager to update a semantic state and a dialog state;

[0075] Figure 1C yes Figure 1B A schematic diagram of an embodiment of a system in which an arbitrator selects between a natural language understanding output or a fragment integrator output;

[0076] Figure 1D yes Figure 1B A schematic diagram of an embodiment of a system in which an arbitrator selects between a natural language understanding output or a fragment output from a fragment integrator;

[0077] Figure 2 is a schematic diagram of an embodiment of end-to-end audio recognition implemented using a machine learning segment identifier model;

[0078] Figure 3 is a flow chart illustrating a method for processing segments in a natural language understanding environment;

[0079] Figure 3A is a flow chart illustrating an alternative method for processing snippets in a natural language understanding environment;

[0080] Figure 3B is a flow chart illustrating an alternative method for processing snippets in a natural language understanding environment;

[0081] Figure 3C is a flow chart illustrating an alternative method for processing snippets in a natural language understanding environment;

[0082] Figure 3D is a flow chart illustrating an alternative method for processing snippets in a natural language understanding environment;

[0083] Figure 3E is a flow chart illustrating an alternative method for processing snippets in a natural language understanding environment;

[0084] Figure 3F is a flow chart illustrating an alternative method for processing snippets in a natural language understanding environment;

[0085] Figure 3G is a flow chart illustrating an alternative method for processing snippets in a natural language understanding environment;

[0086] Figure 4 is a flow chart illustrating a method for processing segments using a segment identifier implemented using a sequence-to-sequence neural network;

[0087] Figure 4A is a flowchart illustrating an alternative method for processing segments using a segment identifier implemented using a sequence-to-sequence neural network;

[0088] Figure 4B is a flowchart illustrating an alternative method for processing segments using a segment identifier implemented using a sequence-to-sequence neural network;

[0089] Figure 4C is a flowchart illustrating an alternative method for processing segments using a segment identifier implemented using a sequence-to-sequence neural network;

[0090] Figure 4D is a flowchart illustrating an alternative method for processing segments using a segment identifier implemented using a sequence-to-sequence neural network;

[0091] Figure 4E is a flowchart illustrating an alternative method for processing segments using a segment identifier implemented using a sequence-to-sequence neural network;

[0092] Figure 4F is a flowchart illustrating an alternative method for processing segments using a segment identifier implemented using a sequence-to-sequence neural network;

[0093] Figure 5 is a flow chart illustrating a method for processing segments in a natural language understanding environment;

[0094] Figure 6 is a schematic diagram of a device illustrating a sequence of operations that may be performed when running semantic completion software;

[0095] Figure 6A is a schematic diagram of a device illustrating a sequence of operations that may be performed when running semantic completion software;

[0096] Figure 6B is a schematic diagram of a device illustrating a sequence of operations that may be performed when running semantic completion software;

[0097] Figure 6C is a schematic diagram of a device illustrating a sequence of operations that may be performed when running semantic completion software;

[0098] Figure 6D is a schematic diagram of a device illustrating a sequence of operations that may be performed when running semantic completion software;

[0099] Figure 6E is a schematic diagram of a device illustrating a sequence of operations that may be performed when running semantic completion software; and

[0100] Figure 7 is a flow chart illustrating a method for processing segments in a natural language understanding environment.

[0101] The accompanying drawings depict various embodiments of the present disclosure for illustrative purposes only. Those skilled in the art will readily recognize from the following detailed description that other alternative embodiments of the structures and methods shown herein may be employed without departing from the principles of the present disclosure as described herein. DETAILED DESCRIPTION

[0102] Figure 1 The present invention is a block diagram of a system 100 for a natural language understanding environment that updates a display in real time based on real-time detection of segments within a speech transcript. System 100 may include audio 101, a speech engine 102, a text segment identifier 103, and a domain processing program 107. Speech engine 102 produces a continuous transcript 105 of speech. Sequences of words as short as one word in transcript 105 are identified as one or more segments 106 by text segment identifier 103. Segment patterns 104 are a database of regular expressions. Text from transcript 105 that matches a regular expression in segment pattern 104 is a segment 106. Segments 106 are processed immediately upon recognition by domain processing program 107. Domain processing program 107 continuously updates 108 a semantic state 109. The cumulative understanding state 111 of the utterance is the output of the updated semantic state 109. As domain processing program 107 processes segments 106, it can cause visual display 110 to update 122 in real time. Changes in semantic state 109 may result in a display update, such as an update to visual display 110 .

[0103] Figure 1A is a diagram showing a system 112 with a fragment integrator 113 Figure 1 The segment integrator 113 stores the segments 106 for a delay period before the domain handler 107 processes the edited segment stream 114 .

[0104] Figure 1B The system 115 is shown Figure 1A The domain handler 107 causes the session manager 117 to update the semantic state 109 and the dialog state 116 .

[0105] Figure 1C The system 118 is shown Figure 1B 1. Block diagram of an embodiment of the invention. The speech engine 102 generates a continuous transcript 105a, which is input to the natural language understanding 119. The speech engine 102 generates a continuous transcript 105b, which is input to the text segment identifier 103. The natural language understanding 119 outputs a deciphered data structure 120a. The segment integrator 113 outputs an edited segment stream 114a. The arbitrator 121 selects between the deciphered data structure 120a from the natural language understanding 119 or the edited segment stream 114a from the segment integrator 113. The arbitrator 121 selects or filters the deciphered data structure 120b or the edited segment stream 114b for output through the domain handler 107.

[0106] Figure 1D It is shown that the system 123 Figure 1B 1. An arbitrator 121 selects between a deciphered data structure 120a from natural language understanding 119 or a segment 106a output from text segment identifier 103. Heuristics 124 may be used to determine the word sequence to which natural language understanding 119 should be applied. Arbitrator 121 outputs deciphered data structure 120a and segment 106b.

[0107] Figure 2 2 is a schematic diagram of an embodiment of an end-to-end audio recognition system 200 implemented as a machine learning model. Speech audio 201 is input to a segment identifier 202. Segment identifier 202 executes the machine learning model on speech audio 201. Segment identifier 202 uses the machine learning model to infer the presence of segments 204 in speech audio 201. The machine learning model is trained on speech audio samples that are labeled as corresponding to specific segments. Supervised learning using requested, curated, or manually labeled audio data is one possible training approach. Unsupervised, semi-supervised, or reinforcement learning model training can also be used to create a segment identifier model that infers the presence of key segments directly from the speech audio.

[0108] The segment identifier 202 outputs segments 204 derived from its inference by the machine learning model 203. The system 200 uses the machine learning model 203, such as a neural network, to convert the audio 201 into segments without the intermediate step of computing a transcript. The segment integrator 205 stores the segments 204 for a delay period before the domain processor 207 processes the edited segment stream 206. The segments 204 are processed immediately upon being identified by the domain processor 207. The domain processor 207 continuously updates 209 the semantic state 210. The cumulative understanding state 211 of the utterance is the output of the updated semantic state 210. In one embodiment, as the domain processor 207 processes the segment stream 206 in real time, the domain processor 207 can cause 212 the visual display 208 to update. Changes in the semantic state 210 can cause a display, such as the visual display 208, to update.

[0109] Figure 3 is a flow chart illustrating a method 300 for processing segments in a natural language understanding environment. The method 300 may include a step 301 of transcribing audio using a speech engine to create a transcript.

[0110] The method 300 may further include executing a segment identifier on a portion of the transcript 302. The segment identifier infers the presence of a segment in the portion of the speech audio and outputs the segment according to the mapping of the plurality of segments to the segment.

[0111] The method 300 may further include step 303 of invoking a domain handler with the segment in response to detecting the presence of the segment in the portion of the speech audio.

[0112] Figure 3A is a flow chart illustrating an alternative method 304 for processing segments in a natural language understanding environment. The method 304 may include Figure 3 and step 305 of updating the session state data structure with entity values ​​from the fragment.

[0113] Figure 3B is a flow chart illustrating an alternative method 306 for processing segments in a natural language understanding environment. The method 306 may include Figure 3 The method 306 further comprises the step of displaying context-related information suggesting at least one word to be spoken, the suggestion being dependent on the semantic state.

[0114] Figure 3C is a flow chart illustrating an alternative method 308 for processing segments in a natural language understanding environment. The method 308 may include Figure 3 and step 309 of storing the fragment using the fragment integrator for a delay period after detecting the presence of the fragment, wherein calling the domain handler occurs after the delay period.

[0115] Figure 3D is a flow chart illustrating an alternative method 310 for processing segments in a natural language understanding environment. The method 310 may include Figure 3 The step and the domain processing program cause the user interface to change in real time on the display in step 311.

[0116] Figure 3E is a flow chart illustrating an alternative method 312 for processing segments in a natural language understanding environment. The method 312 may include Figure 3 and step 313, where step 313 is analyzing the transcript using full-sentence natural language understanding to generate a parsed data structure from a larger portion of the transcript, and since the segment is received by the domain processing program at a specified time, the segment is selected instead of the parsed data structure.

[0117] Figure 3F is a flow chart illustrating an alternative method 314 for processing segments in a natural language understanding environment. The method 314 may include Figure 3 and step 315 of updating a portion of the semantic state using the domain handler.

[0118] Figure 3G is a flow chart illustrating an alternative method 316 for processing segments in a natural language understanding environment. The method 316 may include Figure 3 and step 315 of updating a portion of the semantic state using the domain handler.

[0119] The method 316 may also include a step 317 where the semantic state causes a context-dependent bias in the speech engine for subsequent transcriptions.

[0120] Figure 4 4 is a flow chart illustrating a method 400 for processing segments in a sequence-to-sequence neural network. The method 400 may include a step 401 of executing a segment identifier on a portion of speech audio, the segment identifier inferring the presence of a segment in the portion of speech audio and outputting the segment, the segment being obtained based on a mapping of multiple segments to the segment.

[0121] The method 400 may also include a step 402 of invoking a domain handler using the snippet in response to the inference.

[0122] Figure 4A is a flow chart illustrating an alternative method 403 for processing segments in a sequence-to-sequence neural network. The method 403 may include Figure 4 and step 404 of updating the dialog state data structure using entity values ​​from the fragment.

[0123] Figure 4B is a flow chart illustrating an alternative method 405 for processing segments in a sequence-to-sequence neural network. The method 403 may include Figure 4 The method 403 further comprises the step of displaying context-related information suggesting at least one word to be spoken, the suggestion being dependent on the semantic state.

[0124] Figure 4C is a flow chart illustrating an alternative method 407 for processing segments in a sequence-to-sequence neural network. The method 407 may include Figure 4 and step 408 of storing the fragment using the fragment integrator for a delay period after detecting the presence of the fragment, wherein calling the domain handler occurs after the delay period.

[0125] Figure 4D is a flow chart illustrating an alternative method 409 for processing segments in a sequence-to-sequence neural network. The method 409 may include Figure 4 The step and the domain processing program cause the user interface to change in real time on the display at step 410.

[0126] Figure 4E is a flow chart illustrating an alternative method 414 for processing segments in a sequence-to-sequence neural network. The method 414 may include Figure 4 and step 415 of updating a portion of the semantic state using the domain handler.

[0127] Figure 4F is a flow chart illustrating an alternative method 416 for processing segments in a sequence-to-sequence neural network. The method 416 may include Figure 4 The steps of and semantic state cause a context-dependent bias of the segment identifier 417 .

[0128] Figure 5 is a flow chart illustrating a method 500 for processing segments in a sequence-to-sequence neural network. The method 500 may include a step 501 of transcribing audio using a speech engine to create a transcript.

[0129] The method 500 may also include a step 502 of executing a segment identifier on a portion of the transcript, the segment identifier detecting the presence of a first segment and outputting the first segment.

[0130] The method 500 may further include step 503 : in response to detecting the presence of the first fragment, invoking a domain handler using a fragment selected from the group consisting of the first fragment and the second fragment.

[0131] Method 500 may further include step 504: in response to detecting the presence of the second fragment during the delay period, invoking the domain handler with the second fragment. This step 504 emphasizes the concept of waiting and discarding the first fragment if a better second fragment appears.

[0132] Figure 6 6 is a schematic diagram of a display of a device 606 illustrating a sequence of operations 600 that can be performed when running segment processing software. Semantic completion is shown, whereby the semantic state is updated 604 and displayed to the user in real time. The device 606 has a user interface 601. The transcript 602 is part of the real-time user speech. In one example, the order item 605 is Kids Mac & Cheese. Based on the request for Kids Mac & Cheese in the transcript 602, but the options for the expected beverage selection and side dish selection have not yet been specified, a list of suggestions for beverage selection 603a and side dish selection 603b is displayed to show that the possible beverage selection 603a is apple juice or milk, and the possible side dish 603b is French fries or yogurt.

[0133] Figure 6A 6 is a schematic diagram of a display of a device 606 illustrating a sequence of operations 607 that may be performed when running segment parsing software. Semantic completion is illustrated, whereby semantic state is updated 604 and displayed to the user in real time. Device 606 has a user interface 601. Transcript 602 is a portion of real-time user speech. In one example, order item 605 is a kids' macaroni and cheese dish with a yogurt option 608a for a side dish. Based on transcript 602, a suggestion list is displayed showing possible beverage options 603a as apple juice and / or milk.

[0134] Figure 6B 6 is a schematic diagram of a display of a device 606 illustrating a sequence of operations 609 that can be performed when running segment parsing software. Semantic completion is illustrated, whereby the semantic state is updated 604 and displayed to the user in real time. Device 606 has a user interface 601. Transcript 602 is a portion of the real-time user speech. In one example, order item 605 is a kids' macaroni and cheese, with side selection 608a being yogurt and beverage selection 608b being apple juice.

[0135] Figure 6C 6 is a schematic diagram of a display of a device 606 illustrating a sequence of operations 610 that can be performed when running snippet parsing software. Semantic completion is shown, whereby the semantic state is updated 604 and displayed to the user in real time. Device 606 has a user interface 601. Transcript 602 is a portion of real-time user speech. In one example, when transcript 602 includes snippet 611 "dessert," a unique dessert menu 612 appears. In one example, order item 605a is kids' macaroni and cheese, side option 608a is yogurt, and beverage option 608b is apple juice. Order item 605b is Sprite with a large size option 613. Order item 605c is sweet potato fries. In response to the transcript matching the snippet pattern "dessert," the dessert menu is presented by the domain handler. In various embodiments, the dessert menu remains visible until another snippet pattern is matched that invokes a special popup, until a certain period of time has passed, until the transcript matches a snippet that is a specific dessert item, or until the first or last of these combinations of criteria are met to remove the dessert menu popup.

[0136] Figure 6D 6 is a schematic diagram of a display of a device 606 showing a sequence of operations 614 that can be performed when running segment parsing software. Semantic completion is shown, whereby the semantic state is updated 604 and displayed to the user in real time. The device 606 has a user interface 601. The transcript 602 is part of the real-time user speech. In the example of editing, when the transcript 602 includes the segment "change" and is followed by an explicit reference to an item in the semantic state (order list) "kids macaroni and cheese", the system highlights the item. When the transcript 602 also includes the word "change" 615, the system performs an edit. When the edit references the specific choice "milk" available for the beverage selection 608a "beverage", the system replaces the existing option "apple juice" with the specified option "milk". Order item 605d is a brownie. Order item 605e is a chocolate chip cookie.

[0137] Figure 6E 6 is a schematic diagram of a display of device 606 illustrating a sequence of operations 616 that can be performed when running segment parsing software. Semantic completion is shown, whereby the semantic state is updated 604 and order items 605a-605e are displayed to the user in real time. Device 606 has a user interface 601. Transcript 602 is a portion of the real-time user speech. In the completed example, when the user utters the command 617 "That's it," the system displays a message 618 listing the items in the order and outputs this text 619 as text-to-speech for the user to hear. During the text-to-speech process, device 606 reads the words on the screen aloud to the user. The selected option 608a is yogurt, and the selected option 608b is milk.

[0138] At the end of operation sequence 616, the user has completed their order at the restaurant. The system proceeds to charge their credit card for all items in their virtual shopping cart. The user can then pay for the ordered items. Operation sequence 616 displays a total of $24. In some embodiments, payment can be made in other currencies.

[0139] Figure 7 is a flow chart illustrating a method 700 for processing segments in a natural language understanding environment. The method 700 may include a step 701 of transcribing audio using a speech engine to create a transcript.

[0140] The method 700 may also include a step 702 of matching segments in the transcript to segment patterns.

[0141] The method 700 may further include a step 703 of extracting sentences from the transcript.

[0142] The method 700 may further include step 704 of performing full-sentence natural language understanding on the sentences to identify sentences that can be understood and generate an interpretation data structure.

[0143] The method 700 may also include a step 705 of selecting an interpretation data structure and discarding the segment in response to identifying that the sentence can be parsed.

[0144] The method 700 may further include the step 706 of invoking a domain handler using the interpreted data structure.

[0145] Some embodiments provide a platform for different product suppliers (e.g., restaurants) to implement their specific menus. Such a platform may include a text or graphical configuration interface that allows suppliers to enter or upload their menu item lists. The configuration tool may accept a set of configurable options for each type of item and the value of each option.

[0146] Finally, it should be noted that the language used in the specification is primarily selected for readability and instructional purposes, and may not be selected to describe or limit the subject matter of the present disclosure. Accordingly, the described embodiments are illustrative of the present disclosure, not exhaustive.

[0147] Appendix 1: Pseudocode for the Session Manager

[0148] This appendix illustrates pseudocode for a conversation manager that can perform functions including, but not limited to, composing follow-up questions to request information from the user to complete a semantic state change.

[0149]

[0150]

[0151]

[0152] < / y> < / x>

Claims

1. A computer-implemented method comprising: transcribe the audio using a speech engine to create a transcript; executing a segment identifier on a portion of the transcript, the segment identifier detecting segments and outputting the segments; as well as In response to the detection, a domain handler is called with the fragment.

2. The method of claim 1 further comprising updating a dialog state data structure using entity values ​​from the fragment.

3. The method according to claim 1, wherein The domain processing program outputs a semantic state, the method further comprising displaying context-related information that suggests at least one word to be spoken, the suggestion being dependent on the semantic state.

4. The method according to claim 1, further comprising: After detection, the fragment is stored using a fragment integrator for a delay period, wherein invoking the domain handler occurs after the delay period.

5. The method according to claim 1, wherein The domain handler causes the user interface to change on the display in real time. The method of claim 1 , further comprising updating a portion of a semantic state using the domain handler.

7. The method according to claim 6, wherein: The semantic state causes a context-dependent bias in the speech engine's response to subsequent transcripts.

8. A computer-implemented method comprising: transcribe the audio using a speech engine to create a transcript; matching segments in the transcript to segment patterns; extracting sentences from the transcript; performing full-sentence natural language understanding on the sentence to identify that the sentence can be understood and generate an interpretation data structure; In response to identifying that the sentence can be parsed, selecting the interpretation data structure and discarding the segment; as well as A domain handler is called using the interpreted data structure.

9. A computer-implemented method comprising: executing a segment identifier on a portion of the speech audio, the segment identifier inferring the presence of a segment in the portion of the speech audio and outputting the segment, the segment being derived from the mapping of the plurality of segments to the segment; and In response to the inference, a domain handler is invoked using the fragment.

10. The method of claim 9, further comprising updating a dialog state data structure with entity values ​​from the fragment.

11. The method according to claim 9, wherein The domain processing program outputs a semantic state, the method further comprising displaying context-related information that suggests at least one word to be spoken, the suggestion being dependent on the semantic state.

12. The method according to claim 9, further comprising: After detecting the presence of the fragment, the fragment is stored using a fragment integrator for a delay period, wherein invoking the domain handler occurs after the delay period.

13. The method according to claim 9, wherein: The domain handler causes the user interface to change on the display in real time.

14. The method of claim 9, further comprising updating a portion of a semantic state using the domain handler.

15. The method according to claim 14, wherein The semantic state induces a context-dependent bias in the segment identifier.

16. A computer-implemented method comprising: transcribe the audio using a speech engine to create a transcript; executing a segment identifier on a portion of the transcript, the segment identifier detecting the presence of a first segment and outputting the first segment; as well as In response to detecting the presence of the first fragment, a domain handler is invoked with a fragment selected from the group consisting of the first fragment and a second fragment.

17. The method of claim 16, further comprising updating a dialog state data structure using entity values ​​from the fragment.

18. The method according to claim 16, wherein The domain processing program outputs a semantic state, the method further comprising displaying context-related information that suggests at least one word to be spoken, the suggestion being dependent on the semantic state.

19. The method according to claim 16, further comprising: After detecting the presence of the first fragment, the fragment is stored using a fragment integrator for a delay period, wherein invoking the domain handler occurs after the delay period.

20. The method according to claim 16, wherein The domain handler causes the user interface to change on the display in real time.

21. The method of claim 16, further comprising analyzing the transcript using whole-sentence natural language understanding to generate a parsed data structure from a larger portion of the transcript; and Since the fragment is received by the domain handler at the specified time, the fragment is selected instead of the interpretation data structure.

22. The method of claim 16, further comprising updating a portion of a semantic state using the domain handler.

23. The method according to claim 22, wherein The semantic state causes a context-dependent bias in the speech engine's response to subsequent transcripts.

24. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the following steps: transcribe the audio using a speech engine to create a transcript; executing a segment identifier on a portion of the transcript, the segment identifier detecting segments and outputting the segments; and In response to the detection, a domain handler is called with the fragment.

25. The computer-readable medium of claim 24, further comprising updating a dialog state data structure using entity values ​​from the fragment.

26. The computer-readable medium of claim 24, wherein: The domain processing program outputs a semantic state, the steps further comprising displaying context-related information, the context-related information suggesting at least one word to be spoken, the suggestion being dependent on the semantic state.

27. The computer-readable medium of claim 24, further comprising: After detection, the fragment is stored using a fragment integrator for a delay period, wherein invoking the domain handler occurs after the delay period.

28. The computer-readable medium of claim 24, wherein: The domain handler causes the user interface to change on the display in real time.

29. The computer-readable medium of claim 24, further comprising analyzing the transcript using whole-sentence natural language understanding to generate a parsed data structure from a larger portion of the transcript; and Since the fragment is received by the domain handler at the specified time, the fragment is selected instead of the interpretation data structure.

30. The computer-readable medium of claim 25, further comprising updating a portion of a semantic state using the domain handler.

31. The computer-readable medium of claim 30, wherein: The semantic state causes a context-dependent bias in the speech engine's response to subsequent transcripts.

32. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the following steps: executing a segment identifier on a portion of the speech audio, the segment identifier inferring the presence of a segment in the portion of the speech audio and outputting the segment, the segment being derived from the mapping of the plurality of segments to the segment; and In response to the inference, a domain handler is invoked using the fragment.

33. The computer-readable medium of claim 32, the steps further comprising updating a dialog state data structure using entity values ​​from the fragment.

34. The computer-readable medium of claim 32, wherein: The domain processing program outputs a semantic state, the steps further comprising displaying context-related information, the context-related information suggesting at least one word to be spoken, the suggestion being dependent on the semantic state.

35. The computer-readable medium of claim 32, further comprising: After detecting the presence of the fragment, the fragment is stored using a fragment integrator for a delay period, wherein invoking the domain handler occurs after the delay period.

36. The computer-readable medium of claim 32, wherein: The domain handler causes the user interface to change on the display in real time.

37. The computer-readable medium of claim 33, the steps further comprising updating a portion of a semantic state using the domain handler.

38. The computer-readable medium of claim 37, wherein: The semantic state induces a context-dependent bias in the segment identifier.

Images