Real-time feedback for efficient conversation handling

Abstract

The present disclosure relates to real-time feedback for efficient dialogue processing. Technologies for improving the efficiency of dialogue processing by prompting and processing user feedback in substantially real time are described. A dialogue system receives speech input from a user and processes a first portion of the speech input to determine an initially discerned intent. The dialogue system causes a visual indication of the initially discerned intent to be displayed. The visual indication of the discerned intent is used to guide the dialogue so that the user can correct or confirm the initially discerned intent in a natural manner. If the initially discerned intent is inaccurate, the user can provide feedback to correct the dialogue system, which then processes a second portion of the speech input to determine a modified discerned intent. Thus, the dialogue system can use feedback to instantly correct misunderstandings.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority under 35 USC 119(e) to U.S. patent application No. 62 / 899,641, filed on September 12, 2019, entitled “IMPROVING COGNITIVE UNDERSTANDING IN A DIALOG SYSTEM USING USER FEEDBACK,” the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to a dialog system. More specifically, but not limitedly, the present disclosure describes a technology for improving the experience of dialog processing and increasing the efficiency of dialog processing by providing visual cues for guiding the dialog. Background Art

[0004] Today, more and more devices enable users to interact with the device directly using voice or spoken speech. For example, a user can speak to such a device in natural language, wherein the user can ask a question or make a statement requesting that an action be performed. In response, the device performs the requested action or responds to the user's question using voice output. Since interacting directly using voice is a more natural and intuitive way for humans to communicate with their surroundings, the popularity of such voice-based systems is growing at an astronomical rate.

[0005] In conventional systems, conversations between users and dialog systems can be long and frustrating when the dialog system misunderstands the meaning of received utterances. Conventional dialog systems operate as a series of transactions rather than a continuous stream. For example, a user holds down a button to speak a sentence, the dialog system determines and provides a response, and then the conversation ends. But if the dialog system misunderstands the meaning of the sentence, the dialog system may return an incorrect response, so the dialog system must start a new session and interact with the user repeatedly until the correct meaning is determined. Summary of the invention

[0006] The present disclosure generally relates to a dialog system. More particularly, techniques for improving the experience of dialog processing and increasing the efficiency of dialog processing by providing visual cues for guiding the dialog are described. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, codes, or instructions executable by one or more processors, and the like.

[0007] In some embodiments, a dialog system receives speech input from a user. The dialog system processes a first portion of the speech input to determine an initially discerned intent. The dialog system causes a visual indication of the initially discerned intent to be displayed. The dialog system processes a second portion of the speech input to determine a modified discerned intent, wherein the second portion of the speech input corrects the initially discerned intent in response to the visual indication of the initially discerned intent.

[0008] In some aspects, the dialog system processes the speech input substantially in real time. In some aspects, the visual indication includes one or more of: text specifying a domain associated with the discerned intent; text specifying the discerned intent; an image indicating a domain associated with the discerned intent; or an image indicating the discerned intent.

[0009] In some aspects, the method further includes performing, by the dialog system, a task corresponding to the modified identified intent. In some aspects, the method further includes determining, by the dialog system, a response based on the modified identified intent, and providing, by the dialog system, the response to the user.

[0010] In some aspects, determining the initially identified intent includes: calculating, by the dialogue system, multiple scores of the corresponding multiple potential intents and the first part of the speech input; and determining, by the dialogue system, that the score of a specific intent among the multiple potential intents exceeds a threshold, thereby setting the specific intent as the initially identified intent, wherein displaying a visual indication of the initially identified intent is triggered by determining that the score exceeds the threshold.

[0011] In some aspects, determining the initially discerned intent includes: determining, by an automatic speech recognition subsystem of a dialog system, a first text utterance corresponding to a first portion of a speech input; providing, by the automatic speech recognition subsystem, the first text utterance to a natural language understanding subsystem of the dialog system; and determining, by the natural language understanding subsystem, the initially discerned intent based on the first text utterance.

[0012] Embodiments also include systems and computer-readable media (eg, non-transitory computer-readable memory) configured to perform the methods described herein.

[0013] The foregoing together with other features and embodiments will become more apparent when referring to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a simplified block diagram depicting a dialog system according to some embodiments.

[0015] Figure 2 is a simplified flow chart describing a method of techniques for using feedback for efficient conversation handling in accordance with certain embodiments.

[0016] Figure 3 is a simplified flow chart describing a technique for determining and displaying initially discerned intent, according to certain embodiments.

[0017] Figure 4A-4D Interface views according to some embodiments are described.

[0018] Figure 5 A simplified diagram of a distributed system for implementing an embodiment is depicted.

[0019] Figure 6 is a simplified block diagram of a cloud-based system environment in which various storage-related services may be provided as cloud services, according to certain embodiments.

[0020] Figure 7 An exemplary computer system is shown that can be used to implement certain embodiments. DETAILED DESCRIPTION

[0021] In the following description, for the purpose of explanation, specific details are set forth to provide a comprehensive understanding of certain embodiments. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments or designs.

[0022] Voice-enabled systems that are capable of dialoguing with a user via voice input and voice output may come in a variety of forms. For example, such systems may be provided as standalone devices, digital or virtual assistants, voice-enabled services, and the like. In each of these forms, the system is capable of receiving voice or speech input, understanding the input, generating a response or taking an action in response to the input, and outputting the response using voice output. In some embodiments, the dialog functionality in such voice-enabled systems is provided by a dialog system or infrastructure ("dialog system").

[0023] As described above, in conventional systems, conversations between users and dialogue systems can be long and frustrating due to problems with the overall flow of cognitive understanding and dialogue processing. For example, typically, the dialogue system will process the entire question or sentence, including determining the user's intent, and return a response. If the dialogue system incorrectly identifies the user's intent, the dialogue system may return an incorrect response and then interact with the user repeatedly as the user tries to clarify the query. This often requires starting a new dialogue session—for example, the dialogue system will return an incorrect answer and terminate the dialogue session, and the user will have to initiate a new dialogue session to try to correct the misunderstanding. In addition to being frustrating to the user, the longer it takes to resolve a user's query, the greater the consumption of processing time and computing resources.

[0024] The present disclosure describes techniques for reducing the amount of user interaction, time, memory, and processing resources used by a dialog system by providing real-time feedback associated with user input in real time. For example, a dialog system can process partial utterances, sentences, or questions to determine the initially discerned intent while the user is still speaking. As the user continues to speak, the system returns a visual indication of the initially discerned intent. In this way, the user can provide clarification or change the route while continuing to speak. This eliminates the memory required to initiate a new conversation and reduces the processing power required to handle multiple inputs and outputs caused by providing incorrect responses. These effects also improve the user experience.

[0025] Dialog processing can involve detecting the domain of an utterance and / or classifying the intent of an utterance. A domain is a category of an utterance - for example, travel, weather, and music are domains. The intent of an utterance is equivalent to the purpose of the utterance (e.g., the action that the user is requesting to perform). Examples of intents include Order_Pizza and Find_Directions. A given domain can correspond to multiple intents - for example, in the "travel" domain, an utterance can be classified using intents selected from Book_Flight, Upgrade_Seat, Request_Car, etc.

[0026] Domain and intent classification are each operations performed relatively early in the dialog processing pipeline, and therefore, it is beneficial to display an indication of the initially discerned intent and / or domain to improve cognitive understanding in the dialog system. By providing an indication of the initially discerned intent and / or domain relatively early in the dialog, the dialog system can prompt and process user utterances in a guided manner to ensure that the dialog system is operating with the correct domain and / or intent.

[0027] In some embodiments, while the user speaks one or more sentences, the dialog system continuously presents its understanding of the context. User feedback can be generated through words, pictures, or through natural language. For example, the dialog system detects user input including the word "John Howie" and determines with a certain degree of confidence that John Howie is a contact (for example, the system determines that the recognized user intent is to retrieve contact information). The dialog system can display a list of all "John" in the user's contacts on the screen. Then, after receiving additional information later in the discourse or through the next sentence, the dialog system can re-evaluate the request. For example, the user can correct the system by continuing to speak and saying "No, I am talking about the steakhouse (No, I am talking about the steakhouse)". This may cause the dialog system to modify the discerned intent related to identifying information about the restaurant. The prompts provided by the dialog system allow the user to see that the system may misunderstand them. Then, the user can adjust their speech midway so that the dialog system can seamlessly provide more information. By using real-time processing to indicate the discerned intent midway, the dialog system prompts the user to provide clarification or correction when necessary without having to stop and start the conversation.

[0028] Figure 1 An example of a dialog system 100 according to some embodiments is shown. The dialog system 100 is a special-purpose computing system that can be used to process large amounts of data using large numbers of computer processing cycles. Figure 1 The number of devices described in is provided for illustrative purposes. A different number of devices may be used. For example, although Figure 1 Each device, server, and system in is shown as a single device, but multiple devices may be used instead.

[0029] The dialog system 100 is configured to receive speech or voice input 104 (also referred to as voice utterances) from a user 102. The dialog system 100 may then understand the speech input. The dialog system 100 may maintain a dialog with the user 102 and may perform one or more actions or cause one or more actions to be performed based on the understanding of the speech input. The dialog system 100 may prepare an appropriate response and output the response to the user using speech or voice output 122. The dialog system 100 may further prepare and output a visual output 128.

[0030] In some embodiments, the processing performed by the dialog system is implemented by a pipeline of components or subsystems, including a speech input component 105, a wake-up word detection (WD) subsystem 106, an automatic speech recognition (ASR) subsystem 108, a natural language understanding (NLU) subsystem 110, which includes a named entity recognizer (NER) subsystem 112 and a semantic parser subsystem 114, a dialog manager (DM) subsystem 116, a natural language generator (NLG) subsystem 118, a text-to-speech (TTS) subsystem 120, a speech output component 124, a visual feedback generator subsystem 126, and a display 130. The subsystems listed above can be implemented solely in software (e.g., using code, programs, or instructions that can be executed by one or more processors or cores), hardware, or a combination of hardware and software. In some embodiments, one or more of the subsystems can be combined into a single subsystem. Additionally or alternatively, in some embodiments, the functions performed by a particular subsystem described herein can be implemented by multiple subsystems.

[0031] The speech input component 105 includes hardware and software configured to receive the speech input 104. In some examples, the speech input component 105 can be part of the dialog system 100. In some other examples, the speech input component 105 can be separate from the dialog system 100 and communicatively coupled to the dialog system 100. The speech input component 105 can, for example, include a microphone coupled to software configured to digitize the speech input and transmit it to the wake-up word detection subsystem 106.

[0032] The wake-up word detection (WD) subsystem 106 is configured to listen and monitor the audio input stream for input corresponding to a special sound or word or set of words (referred to as a wake-up word). After detecting the wake-up word configured for the dialogue system 100, the WD subsystem 106 is configured to activate the ASR subsystem 108. In some embodiments, the user may be provided with the ability to activate or deactivate the WD subsystem 106 (e.g., by speaking the wake-up word and pressing a button). When activated (or when operating in an activation mode), the WD subsystem 106 is configured to continuously receive an audio input stream and process the audio input stream to identify audio or voice input corresponding to the wake-up word. When an audio input corresponding to a wake-up word is detected, the WD subsystem 106 activates the ASR subsystem 108.

[0033] As described above, the WD subsystem 106 activates the ASR subsystem 108. In some embodiments of the voice-enabled system, a mechanism other than a wake word may be used to trigger or activate the ASR subsystem 108. For example, in some embodiments, a button on the device may be used to trigger ASR subsystem 108 processing without a wake word. In such embodiments, the WD subsystem 106 may not be provided. When the button is pressed or activated, speech input received after the button activation is provided to the ASR subsystem 108 for processing. In some embodiments, the ASR subsystem 108 may be activated upon receiving input to be processed.

[0034] The ASR subsystem 108 is configured to receive and monitor spoken voice input after a trigger or wake-up signal (e.g., a wake-up signal may be sent by the WD subsystem 106 after a wake-up word is detected in the voice input, a wake-up signal may be received after a button is activated, etc.), and convert the voice input into text. As part of the processing of the ASR subsystem, the ASR subsystem 108 performs speech-to-text conversion. The spoken speech or voice input may be in a natural language form, and the ASR subsystem 108 is configured to generate corresponding natural language text in the language of the voice input. The text generated by the ASR subsystem is then fed to the NLU subsystem 110 for further processing. The voice input received by the ASR subsystem 108 may include one or more words, phrases, clauses, sentences, questions, etc. The ASR subsystem 108 is configured to generate a text utterance for each spoken clause, and feed the text utterance to the NLU subsystem 110 for further processing.

[0035] The NLU subsystem 110 receives text generated by the ASR subsystem 108. The text received by the NLU subsystem 110 from the ASR subsystem 108 may include text utterances corresponding to spoken words, phrases, clauses, etc. The NLU subsystem 110 translates each text utterance (or a series of text utterances) into its corresponding logical form. In some embodiments, the NLU subsystem 110 includes a named entity recognizer (NER) subsystem 112 and a semantic parser (SP) subsystem 114. The NER subsystem 112 receives the text utterance as input, identifies the named entities in the text utterance, and tags the text utterance with information related to the identified named entities. The tagged text utterances are then fed to the SP subsystem 114 configured to generate a logical form for each tagged text utterance (or a series of tagged text utterances). The logical form generated for the utterance may identify one or more intents corresponding to the text utterance. The intent of an utterance identifies the purpose of the utterance. Examples of intents include "order pizza" and "find directions". An intent may, for example, identify an action requested to be performed. In addition to intents, the logical form generated for the text utterance may also identify slots (also referred to as parameters or independent variables) for the identified intents. For example, for the speech input "I'd like to order a large pepperoni pizza with mushrooms and olives", the NLU subsystem 100 may identify the intent Order_Pizaa. The NLU subsystem may also identify and fill in slots, such as Pizza_Size (filled with "large") and Pizza_Toppings (filled with "mushrooms" and "olives"). The NLU subsystem may generate logical forms using machine learning-based techniques, rules (which may be domain-specific), or a combination of the two. The logical forms generated by the NLU subsystem 110 are then fed to the DM subsystem 116 for further processing.

[0036] The DM subsystem 116 is configured to manage a dialogue with a user based on a logical form received from the NLU subsystem 110. As part of dialogue management, the DM subsystem 116 is configured to track dialogue states, initiate the execution of one of a plurality of actions or tasks or perform one of a plurality of actions or tasks itself, and determine how to interact with the user. These actions may include, for example, querying one or more databases, generating execution results, and other actions. For example, the DM subsystem 116 is configured to interpret the intent identified in the logical form received from the NLU subsystem 110. Based on these interpretations, the DM subsystem 116 may initiate one or more actions that it interprets as requested by the voice input provided by the user. In some embodiments, the DM subsystem 116 performs dialogue state tracking based on current and past voice input and based on a set of rules (e.g., dialogue policies) configured for the DM subsystem 116. These rules may specify different dialogue states, conditions for transitioning between states, actions to be performed when in a particular state, etc. These rules may be domain specific. The DM subsystem 116 also generates responses to be transmitted back to the users involved in the conversation. These responses may be based on the actions initiated by the DM subsystem 116 and the results of these actions. The responses generated by the DM subsystem 116 are fed to the NLG subsystem 118 and / or the visual feedback generator subsystem 126 for further processing.

[0037] The NLG subsystem 118 is configured to generate natural language text corresponding to the response generated by the DM subsystem 116. The text can be generated in a form that enables them to be converted into speech by the TTS subsystem 120. The TTS subsystem 120 receives text from the NLG subsystem 118 and converts each text into speech or voice audio, which can then be output to the user via the audio or voice output component 124 of the dialogue system (e.g., a speaker or a communication channel coupled to an external speaker). Alternatively or in addition, the NLG subsystem 118 can transmit the text output to the visual feedback generation subsystem 126 for further processing. In some instances, the speech output component 124 can be part of the dialogue system 100. In some other instances, the speech output component 124 can be separate from the dialogue system 100 and communicatively coupled to the dialogue system 100.

[0038] The visual feedback generator subsystem 126 is configured to generate visual indications that can indicate cognitive understanding of the dialog system at a given time. The visual indication can include icons, images, and / or text. The visual indication can indicate the discerned intent (e.g., an icon of pizza when the discerned intent is to order pizza). Alternatively or in addition, the visual indication can indicate the discerned domain. The domain can represent an intent family or group. Therefore, the discerned domain can be linked to one or more intents. For example, the visual indication can be an icon representing travel, which can correspond to multiple intents, such as booking a flight, booking a hotel, canceling a reservation, and modifying a reservation. The visual feedback generator subsystem transmits information representing the visual output to the display 130 to provide the visual output 128 to the user 102. The display 130 can be, for example, a display screen on a mobile device, a monitor, a television, an information kiosk, etc. The display presents the visual output to the user 102.

[0039] As described above, the various subsystems of the dialog system 100 work in cooperation to provide the following functionality: enabling the dialog system 100 to receive speech input 104 and respond using speech output 122, and to maintain a dialog with a user using natural language speech. The various subsystems described above may be implemented using a single computer system or using multiple computer systems working in cooperation. For example, for a device implementing a voice-enabled system, the subsystems of the dialog system 100 described above may be implemented entirely on the device with which the user interacts. In some other implementations, some components or subsystems of the dialog system 100 may be implemented on the device with which the user interacts, while other components may be implemented away from the device, on some other computing device, platform, or server.

[0040] As described above, in some embodiments, a pipeline of subsystems can be used to implement the dialogue system 100. In some embodiments, one or more subsystems can be combined into a single subsystem. In some embodiments, the functions provided by a particular subsystem can be provided by multiple subsystems. Multiple subsystems can also be used to implement a particular subsystem.

[0041] In some embodiments, one or more functions of the dialog system 100 may be implemented using machine learning techniques. For example, supervised machine learning techniques, such as those implemented using neural networks (e.g., deep neural networks), may be used to implement one or more functions of the dialog system 100. As an example, a neural network trained to perform the ASR function performed may be provided, and such a trained model may be used by the ASR subsystem 108 for processing of the ASR subsystem. Such a neural network implementation may take speech input as input and output text utterances to the NLU subsystem. Machine learning-based models may also be used by other subsystems of the dialog system 100.

[0042] Figure 2 A simplified flow chart illustrating a method 200 for efficient conversation processing using feedback according to some embodiments is described. The method may be implemented in software (e.g., code, instructions, program), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. Figure 2 The software may be stored on a non-transitory storage medium (eg, on a memory device). Figure 2 The methods presented and described below are intended to be illustrative and not limiting. Figure 2 The various processing steps are described as occurring in a particular order or sequence, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel. In some embodiments, Figure 2 The processing described in can be accomplished by Figure 1 Describes the dialog system implementation.

[0043] At 202, the dialogue system receives speech input from the user. The speech input may include one or more sentences and may include pauses (usually of relatively short duration). The dialogue system may receive the speech input, for example, via a speech input component 105 (e.g., a microphone of a user device or kiosk). The speech input may be received as a waveform. The speech input may be in a specific language and in the form of a natural language query. The speech input may specify questions that the user wants the dialogue system to answer, specify one or more actions that the user wants the dialogue system to initiate (e.g., "I would like a medium pepperoni pizza"), etc. The ASR subsystem 108 may process the speech input to generate text utterances, which the ASR subsystem 108 passes to the NLU subsystem 110 for further processing.

[0044] In some embodiments, instead of receiving spoken voice input (as in 202), input may be received in text form, for example, when a user types text during a chat with the dialog system. The techniques described in this disclosure may be used with systems that receive input in the form of spoken voice, text input, or a combination thereof.

[0045] At 204, the dialog system processes the first portion of the first speech input to determine the initially discerned intent. Processing the speech input may include NLU processing, which may be performed by the natural language understanding subsystem 110. The dialog system may process the speech input using NLU operations such as semantic parsing, intent recognition, slot filing, and domain determination. Processing the speech input may also include ASR processing, which may be performed by the ASR subsystem 108. For example, the automatic speech recognition subsystem 108 determines a first text utterance corresponding to the first portion of the speech input, and provides the first text utterance to the natural language understanding subsystem 110. The natural language subsystem 110 then determines the initially discerned intent based on the first text utterance. Alternatively or in addition, the text input received from the user may be processed directly by the NLU subsystem 110. The processing may also include determining emotions, identifying named entities, slot filing, and the like.

[0046] As a result of the processing, the dialog system may determine an initially discerned intent. The initially discerned intent may correspond to a task that the dialog system has determined that the user may ask the dialog system to perform. For example, the initially discerned intent may be Find_Flight, Get_Weather, or Order_Delivery. In some examples, the initially discerned intent may correspond to an intent determined based on a partial input, such as based on a partially spoken clause. Alternatively or in addition, the dialog system may identify an initially discerned domain. For example, the initially discerned domain may be travel or weather.

[0047] In some embodiments, the dialog system uses a machine learning model to determine the initially discerned intent. The machine learning model can be, for example, a neural network trained to classify the intent of an utterance using sample intents and utterances. As a specific example, a classifier model can be applied to determine that the utterance "How do I get to Charleston?" is best classified with the intent "Directions_To, (navigation to ...")" based on sample utterances of intents related to navigation. Similarly, a machine learning model, such as a neural network trained to predict possible domains based on sample domain to utterance pairs, can be used to determine the initially discerned domain. Examples of suitable model types include hidden Markov models (HMMs) and gated recurrent units (GRUs).

[0048] In some embodiments, as each voice input (e.g., first voice input, second voice input, etc.) is received, the dialog system can classify and maintain a set of ranked intents or domains. For example, three domains can be identified, ranked from 10 (most likely to match the voice input) to 1 (least likely to match the voice input). If a particular domain reaches an application-configurable threshold, an event can be triggered at 206 to display a visual indication. Figure 3 Techniques for determining an initial discerned intent based on such a threshold are further described.

[0049] The dialog system may process a first portion of the speech input while receiving additional portions of the speech input. The first portion of the speech input, the second portion of the speech input, and so on may follow each other substantially in real time as a continuous stream of spoken input from the user. Processing the input substantially in real time may include processing a stream of input data as it is received, rather than waiting for an indication that the input has terminated. For example, the system may process a sentence or a group of words while the user continues to provide speech input and the system continues to receive additional speech input. The dialog system may process the continuous stream of speech input in chunks or portions. For example, the NLU subsystem 110 processes one word or phrase at a time, quickly moving to the next word or phrase in the stream. A chunk may correspond to a sentence or phrase.

[0050] Alternatively or in addition, an integrated speech recognition and natural language understanding pipeline can be used to process speech input. For example, a single neural network can be used to generate a logical form from speech input (e.g., without the intermediate step of outputting a text utterance). Techniques for integrating speech recognition and natural language understanding are described in, for example, Speech Model Pre-training for End-to-End Spoken Language Understanding by Lugosh et al. (https: / / arxiv.org / pdf / 1904.03670.pdf) and From Audio to Semantics: Approaches to end-to-end spoken language understanding by Haghani et al. (https: / / arxiv.org / abs / 1809.09190).

[0051] At 206, the dialog system causes a visual indication of the initially identified intent to be displayed. Causing the display may include presenting the visual indication on a display element of the dialog system. Alternatively or in addition, causing the display may include transmitting an instruction to a remote display device, thereby causing the remote display device to display the visual indication. The visual indication includes one or more of the following: text specifying a domain associated with the identified intent; text specifying the identified intent; an image indicating a domain associated with the identified intent; or an image indicating the identified intent. The domain or intent may be specified in text form (e.g., Travel or Book_Flight). Alternatively or in addition, the domain or intent may be specified by an image such as an application or skill icon, an emoticon, or the like. For example, the dialog system recognizes that the user request is related to expenses, so an icon for "Expense Application" is displayed. This visual cue can be very helpful in reassuring users that they are interacting with supported functionality, and can also allow them to change the process midway if things go off track.

[0052] The dialogue system can process user input substantially in real time, and once the dialogue system determines the discerned intent, the dialogue system can present a visual indication to the user. When the user speaks one or more sentences, the dialogue system can continuously present an understanding of the context. User feedback can be generated through words, pictures, or through natural language. For example, the dialogue system predicts that the voice input corresponds to the "travel" domain. The dialogue system displays the text "travel". As another example, the dialogue system predicts that the initial discerned intent is to find a contact, and to identify the named entity John. The dialogue system displays a list of all contacts of the user named John on the screen. In some examples, the dialogue system can display a visual indication as a complete question or sentence, such as "Do you want to change your flight? (Do you want to change your flight?)". Alternatively or in addition, the dialogue system can display images such as pictures of airplanes, travel-related emoticons, etc., to indicate the initial discerned intent and / or domain.

[0053] In some embodiments, the dialogue system displays a visual indication of the confidence level in the initially discerned intent. For example, the dialogue system has low confidence in the initially discerned intent and displays an indication of the low confidence level. As a specific example, the dialogue system may display an emoticon or question mark with a confused expression to indicate that clarification is needed. As another example, the dialogue system may include a robot or may be communicatively coupled to a robot, and the indication may be a head nod from the robot as an indication that it is understanding the received voice input. In some embodiments, the dialogue system may display an emoticon to indicate the current emotion determined at 204.

[0054] In some embodiments, the dialog system displays the visual indication substantially simultaneously with receiving the continuous stream second voice input from the user. As used herein, "substantially simultaneously" means displaying the indication during the dialog, for example, the voice input may be received within about 1 second before and / or after displaying the recognized intent and / or without requiring the dialog system to determine that the user has finished speaking.

[0055] At 208, the dialog system determines whether the feedback suggests a modification of the initially discerned intent. If the initially discerned intent or domain is inaccurate, the user can provide feedback to guide the dialog system toward the correct intent or domain. The feedback can be part of a continuous stream of received speech input (e.g., a second portion of speech input, a third portion of speech input, etc.).

[0056] Based on the visual indication displayed at 206, the user can determine whether the initially discerned intent is accurate. The user can determine whether the discerned intent indicated by the visual indication actually corresponds to or does not correspond to the intent that the user is trying to convey or the action that the user seeks to initiate. The user can instantly determine whether the initially discerned intent is accurate. By using the visual prompts provided at 206, the user can continue to provide voice input while evaluating the visual indications. The visual indications can allow the user to see that the system may have misinterpreted the input provided. Therefore, the user is able to seamlessly provide additional information. Alternatively, if the intent is correct, the visual prompts can also be very helpful in convincing the user that she is interacting with the functionality supported by the system.

[0057] If the second portion of the speech input explicitly indicates or implies that the initially discerned intent is incorrect, the feedback can suggest modifying the initially discerned intent (e.g., "yes" at 208). For example, in the first portion of the speech input stream, the user refers to a steak house named John Howie, and the indication of the initially discerned intent displayed is an icon representing a contact (e.g., the initially discerned intent is to identify a contact named John Howie). In response to viewing the icon, the user corrects the system by continuing to speak and saying "No, I am talking about the steak house."

[0058] If the feedback confirms or overrides the initially discerned intent, the feedback may not suggest modifying the initially discerned intent (e.g., "No" at 208). For example, the user may confirm that the displayed discerned intent is correct (e.g., by saying "Yes, I want to buy a ticket to New York."). As another example, the user may continue to speak without regard to the displayed instructions, and the dialog system may continue to process the received input based on the initially discerned intent.

[0059] At 210, if the feedback at 208 suggests modifying the initially identified intent, the dialog system processes the second portion of the speech input to determine the modified identified intent. The second portion of the speech input corrects the initially identified intent. For example, the user notices that an incorrect domain icon is displayed at 206, and the user provides adjusted input midway. As a specific example, a continuous stream of speech input is: "Add my dinner expense for $100 [first speech input] using the expenses skill [second speech input, adjusted to specify the intended intent]"

[0060] The dialog system modifies the identified intent in response to the second portion of the speech input. The dialog system can utilize the second portion of the speech input (e.g., real-time feedback) to improve the system's cognitive understanding of the user input and context. For example, based on the second portion of the speech input, which specifies that the user wants to make a reservation at a steakhouse instead of calling a contact named John Howie, the dialog system selects the make reservation intent instead of the find contact intent.

[0061] At 212, the dialog system provides a response or performs an action based on the modified discerned intent. In some embodiments, the dialog system determines a response based on the modified discerned intent. The response may be further based on other aspects of the speech input, such as identified named entities and emotions. The dialog system may provide a response to the user. Providing a response may be performed by preparing and outputting a speech output. The dialog system may prepare a text response via the dialog manager subsystem 116 and the NLG subsystem 118. This text response may be passed to the text-to-speech subsystem 120, which converts the text response into a spoken response. The spoken response is then output as a speech output 122 via the speech output component 124. Alternatively or in addition, the text response may be displayed to the user via the display 130.

[0062] Alternatively or additionally, the dialog system can perform a task corresponding to the modified recognized intent based on the modified recognized intent determined at 210. For example, the system can order a meal, book a flight, retrieve information from a database, etc. based on the intent that has been adjusted in response to user feedback.

[0063] At 214, if the feedback at 208 does not suggest a modification of the initially discerned intent, the dialog system processes the second portion of the speech input in conjunction with the initially discerned intent. If the initially discerned intent determined at 204 and displayed at 206 is accurate, the dialog system may process the supplemental information received in the second portion of the speech input stream. For example, the dialog system may use the second portion of the speech input to fill in a slot in conjunction with the initially discerned intent determined based on the first portion of the speech input. Alternatively or in addition, the dialog system may use the second portion of the speech input in conjunction with the initially discerned intent to perform sentiment analysis.

[0064] At 216, the dialog system provides a response or performs an action based on the initially discerned intent. In a similar manner as described above with respect to 212, the dialog system may determine and provide a response and / or perform one or more actions based on the initially discerned intent.

[0065] Alternatively or additionally, Figure 2 The process described in can be applied to a set of domains. The dialog system can determine the initially discerned domain, display an indication of the initially discerned domain, and adjust the initially discerned domain as needed in a similar manner as described above with respect to intent.

[0066] Figure 3 Described are some embodiments of the present invention. Figure 2 The method 300 of determining the initial discerned intent at block 204 of the present invention may be implemented in software (e.g., code, instructions, program), hardware, or a combination thereof executed by one or more processing units (e.g., processors, cores) of a corresponding system. Figure 2 The software may be stored on a non-transitory storage medium (eg, on a memory device). Figure 3 The methods presented and described below are intended to be illustrative and not limiting. Figure 3 The various processing steps are described as occurring in a particular order or sequence, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel. In some embodiments, Figure 3 The processing described in can be accomplished by Figure 1 Describes the dialog system implementation.

[0067] At 302, the dialog system calculates multiple scores for the corresponding multiple potential intents and the first part of the speech input. The dialog system can use a machine learning model such as a logistic regression model or a neural network to calculate the score. Specific examples of suitable models include long short-term memory (LSTM) recurrent neural networks (e.g., see K Sreelakshmi et al., Deep Bi-Directional LSTM Network for Query Intent Detection, 8th International Conference on Advances in Computing and Communications (ICACC-2018), Rajkumar Buyy and Sherly K.K (2018)), and bidirectional encoder representations from transformers (BERT) (e.g., see Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". arXiv:1810.04805v2 (2018)). Such a model may have been previously trained on a dataset that pairs sample text inputs with corresponding intents. In some embodiments, the output of the model is a score indicating the probability that an intent supported by the system corresponds to the intent of the speech input received at 202.

[0068] At 304, the dialog system determines a threshold. The dialog system may store one or more thresholds for determining whether a particular intent is considered sufficiently likely to correspond to the user's actual intent. The dialog system may retrieve the stored thresholds. In some embodiments, the thresholds may be application-configurable thresholds—the dialog system may accept and apply input (e.g., from an administrator) to configure one or more thresholds. For example, the functionality of the dialog system may be provided as a cloud service to different companies, and different companies may configure different thresholds appropriate for their use.

[0069] At 306, the dialog system determines that the score of the particular intent exceeds the threshold. The dialog system can compare the score of each intent calculated at 302 with the threshold determined at 304 until a score of a particular intent among the multiple potential intents is found to exceed the threshold.

[0070] At 308, the dialog system sets the specific intent at 306 as the initially identified intent. Based on the score exceeding the threshold, the dialog system can determine that the corresponding specific intent is likely to be the actual intent of the user. Therefore, the dialog system sets the specific intent as the initially identified intent.

[0071] After 308, the dialog system may cause a visual indication to be displayed, as described above at 206. Causing the display of a visual indication of the initially identified intent is triggered by determining that the score for the particular intent exceeds a threshold at 306. If the particular intent reaches the threshold, an event will be triggered that can update the graphical interface or be intercepted by the dialog system.

[0072] Alternatively or additionally, Figure 3 The processing described in can be applied to a set of domains. The dialog system can assign scores to multiple domains. For example, a neural network can be trained to classify the domains of a speech input. This can include calculating a score indicating how well each of a set of domains matches the speech input. Alternatively or in addition, the model can calculate scores for intents, each intent being mapped to a corresponding domain. The dialog system can compare each score to a predetermined threshold. Therefore, the dialog system determines enough domains to trigger the display of a visual indication at 306. The dialog system can then display a visual indication of the domain (e.g., a weather icon, which can indicate that the initially discerned intent is related to the weather).

[0073] Figure 4A-4D Interface views 400A-400D are described according to some embodiments. Figure 1 The display 130 of the dialog system 100 may display the following Figure 4A-4D The interface shown provides a visual cue to the user to discern intent.

[0074] Figure 4A A first interface view 400A is shown. The first interface view may be displayed to the user to show input provided by the user to the dialog system. The input may be received as text input, or as voice input, which is processed by the dialog system to determine the text to be displayed. Figure 4A In the example shown, the user has provided the input "Add four..." The dialog system displays the input "Add four" in text bubble 402. This may correspond to the first portion of the continuous stream of voice input. In some embodiments, the dialog system may generate text based on the voice input and display the determined text utterance. In the event that the text output does not match the voice input correctly, the user may make corrections or adjustments mid-stream.

[0075] Figure 4B A second interface view 400B is shown. The second interface view displays a visual indication of the initially discerned intent 404. Figure 4B In the example shown, the visual indication of the initially discerned intent is a calculator icon. The visual indication of the initially discerned intent 404 represents the initially discerned intent determined by the dialog system. In this case, based on Figure 4A , the dialog system determines that the intention is likely enough to add numbers to trigger displaying a visual indication of the recognized intent. Adding numbers corresponds to a calculator skill. Therefore, the dialog system displays a calculator icon.

[0076] Figure 4C A third interface view 400C is shown. The third interface view 400C may be displayed to the user to show the voice or text input provided by the user to the dialog system. Figure 4C In the example shown, the user has provided the input “..apples to shopping list.” The dialog system displays the partial input “apples to shopping list” in text bubble 406 . This may correspond to the second portion of the continuous stream of speech input. Figure 4C The voice input response shown in Figure 4B The calculator icon indicates that the dialog system is not following that the user is referring to the shopping list, so the user must make sure to clearly indicate that this is referring to Figure 4C The shopping list in the second part of the continuous speech input stream shown in FIG.

[0077] Figure 4D A fourth interface view 400D is shown. The fourth interface view 400D displays a result 408. The result 408 is in response to Figure 4A and Figure 4C The voice input indicated in the figure adds four apples to the shopping list. Figure 4B The initial discerned intent shown indicates that the dialog system is able to manipulate the dialog to quickly determine the correct intent and determine and provide Figure 4D The results shown in .

[0078] The techniques described herein have several advantages. By processing user input and providing preliminary understanding of it in real time, the system can more accurately replicate natural speech interactions. For example, when two people are speaking, visual cues can be very useful in letting one person know whether the other understands what they are saying. Similarly, by leveraging real-time natural language processing, the dialog system can display visual indications to the user to let the user know whether the dialog system is understanding the user's speech as expected.

[0079] In contrast, in a typical system that processes voice or typed user input, the system will wait for the spoken clause to complete, and then spend some time processing the spoken clause and returning the result. If the result is wrong, or additional information is required, this may lead to an unexpected start and stop pattern of interaction and / or an extended series of inputs and outputs to make the conversation go smoothly. Compared with the technology described in this article, this results in more time resources, more processing resources, and more computing resources being consumed. The technology described in this article avoids these problems and saves time and computing resources. Therefore, the technology described in this article can reduce the overall time and processing required to perform the function requested by the user, because the system can give and receive feedback in real time to improve the system's cognitive understanding of the received voice input, rather than stopping and restarting the conversation when the user is misunderstood.

[0080] The above infrastructure may be implemented in a variety of different environments including a cloud environment (which may be various types of clouds including private, public, and hybrid cloud environments), an on-premises environment, a hybrid environment, and the like.

[0081] Figure 5 A simplified diagram of a distributed system 500 for implementing an embodiment is depicted. In the illustrated embodiment, the distributed system 500 includes one or more client computing devices 502, 504, 506, and 508 coupled to a server 512 via one or more communication networks 510. The client computing devices 502, 504, 506, and 508 may be configured to execute one or more applications.

[0082] In various embodiments, server 512 may be adapted to run one or more services or software applications that enable use of user feedback to improve cognitive understanding in a dialog system.

[0083] In some embodiments, server 512 may also provide other services or software applications that may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services to users of client computing devices 502, 504, 506, and / or 508, such as under a software as a service (SaaS) model. Users operating client computing devices 502, 504, 506, and / or 508 may in turn utilize one or more client applications to interact with server 512 to utilize the services provided by these components.

[0084] exist Figure 5 In the configuration described in , the server 512 may include one or more components 518, 520, and 522 that implement the functions performed by the server 512. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. It should be understood that a variety of different system configurations are possible, which may be different from the distributed system 500. Figure 5 The embodiment shown in is therefore one example of a distributed system for implementing an embodiment system and is not intended to be limiting.

[0085] According to the teachings of the present disclosure, a user can use client computing devices 502, 504, 506, and / or 508 to use user feedback to improve cognitive understanding in a dialog system. The client device can provide an interface that enables a user of the client device to interact with the client device. The client device can also output information to the user via this interface. Although Figure 5 Only four client computing devices are described, but any number of client computing devices may be supported.

[0086] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, Linux or Linux-like operating systems such as Google Chrome TM OS), including various mobile operating systems (e.g., Microsoft Windows Windows Android TM , Palm Portable handheld devices may include cellular phones, smart phones (e.g., ), tablet computers (e.g. ), personal digital assistants (PDAs), etc. Wearable devices may include Google Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., with or without Microsoft Gesture Input Device Game consoles, Sony System, by Various game systems provided, and others), etc. The client device may be able to execute various different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications) and may use various communication protocols.

[0087] The network(s) 510 may be any type of network familiar to those skilled in the art, and may support data communications using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internetwork Packet Exchange), etc. By way of example only, the network(s) 510 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suites), and / or any other wireless protocol) and / or any combination of these and / or other networks.

[0088] The server 512 may be composed of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, Server 512 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices of the server. In various embodiments, server 512 may be suitable for running one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0089] The computing systems in server 512 may run one or more operating systems, including any of those discussed above as well as any commercially available server operating system. Server 512 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include but are not limited to (International Business Machines) and other commercially available database servers.

[0090] In some implementations, server 512 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client computing devices 502, 504, 506, and 508. As examples, data feeds and / or event updates may include, but are not limited to: feed, Updates or real-time updates, which may include real-time events related to sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 512 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client computing devices 502, 504, 506, and 508.

[0091] The distributed system 500 may also include one or more data repositories 514, 516. In some embodiments, these data repositories may be used to store data and other information. For example, one or more of the data repositories 514, 516 may be used to store information such as conversation history. The data repositories 514, 516 may reside in a variety of locations. For example, the data repository used by the server 512 may be local to the server 512, or may be remote from the server 512 and communicate with the server 512 via a network-based or dedicated connection. The data repositories 514, 516 may be of different types. In some embodiments, the data repository used by the server 512 may be a database, such as a relational database, such as Oracle Database, or a web application. And other databases provided by suppliers. One or more of these databases can be adapted to implement storage, updating and retrieval of data in the database in response to SQL format commands.

[0092] In some embodiments, the application may also use one or more of the data stores 514, 516 to store application data. The data stores used by the application may be of different types, such as, for example, a key-value store, an object store, or a general store backed by a file system.

[0093] In certain embodiments, the NLU-related functionality described in this disclosure may be provided as a service via a cloud environment. Figure 6 is a simplified block diagram of a cloud-based system environment according to certain embodiments, in which various NLU-related services can be provided as cloud services. Figure 6 In the depicted embodiment, cloud infrastructure system 602 may provide one or more cloud services that a user may request using one or more client computing devices 604, 606, and 608. Cloud infrastructure system 602 may include one or more computers and / or servers. The computers in cloud infrastructure system 602 may be organized as general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0094] Network(s) 610 can facilitate data communication and exchange between clients 604, 606, and 608 and cloud infrastructure system 602. Network(s) 610 can include one or more networks. The networks can be of the same or different types. Network(s) 610 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0095] Figure 6 The embodiment described in the foregoing is only one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some other embodiments, the cloud infrastructure system 602 may have more Figure 6 The components described in the drawings may include more or fewer components, may combine two or more components, or may have components in different configurations or arrangements. For example, although Figure 6 Three client computing devices are described, but any number of client computing devices may be supported in alternative embodiments.

[0096] The term cloud service is generally used to refer to services that are made available to users on demand by a service provider's system (e.g., cloud infrastructure system 602) via a communication network such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can make use of the cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system can host applications, and users can order and use the applications on demand over the Internet without having to purchase infrastructure resources for executing the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle of Redwood Shores, California, offers cloud services. It provides several cloud services such as middleware services, database services, Java cloud services and other services.

[0097] In some embodiments, the cloud infrastructure system 602 may provide one or more cloud services using different models, such as a software as a service (SaaS) model, a platform as a service (PaaS) model, an infrastructure as a service (IaaS) model, and other models (including hybrid service models). The cloud infrastructure system 602 may include a set of applications, middleware, databases, and other resources that enable provisioning of various cloud services.

[0098] The SaaS model enables applications or software to be delivered to customers as a service over a communications network such as the Internet, without the customer having to purchase the underlying application hardware or software. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system 602. Oracle Examples of SaaS services provided include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, and others.

[0099] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware, and networking resources) to customers as cloud services to provide elastic computing and storage capabilities. Provides various IaaS services.

[0100] The PaaS model is typically used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without the customer having to purchase, build, or maintain such resources. Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and other services.

[0101] Cloud services are typically provided on an on-demand self-service, subscription-based, elastically scalable, reliable, highly available, and secure basis. For example, a customer may order one or more services provided by cloud infrastructure system 602 via a subscription order. Cloud infrastructure system 602 then performs processing to provide the services requested in the customer's subscription order. For example, cloud infrastructure system 602 uses historical context to influence conversation tasks. Cloud infrastructure system 602 may be configured to provide one or even multiple cloud services.

[0102] The cloud infrastructure system 602 can provide cloud services through different deployment models. In a public cloud model, the cloud infrastructure system 602 can be owned by a third-party cloud service provider, and cloud services are provided to any general public customer, wherein the customer can be an individual or an enterprise. In certain other embodiments, under a private cloud model, the cloud infrastructure system 602 can operate within an organization (e.g., within an enterprise organization) and services are provided to customers within the organization. For example, customers can be various departments of an enterprise such as a human resources department, a payroll department, or even individuals within an enterprise. In certain other embodiments, under a community cloud model, the cloud infrastructure system 602 and the services provided can be shared by several organizations in a related community. Various other models such as a mix of the models mentioned above can also be used.

[0103] Client computing devices 604, 606, and 608 may be of different types (such as Figure 5 The client device 502, 504, 506, and 508 described in the disclosure may be capable of operating one or more client applications. A user may use a client device to interact with the cloud infrastructure system 602 to request services provided by the cloud infrastructure system 602. For example, a user may use a client device to request NLU-related services described in the disclosure.

[0104] In some embodiments, the processing performed by the cloud infrastructure system 602 for providing NLU-related services may involve big data analysis. This analysis may involve using, analyzing, and manipulating large data sets to detect and visualize various trends, behaviors, relationships, etc. in the data. This analysis may be performed by one or more processors, thereby possibly processing data in parallel, performing simulations using data, etc. For example, the cloud infrastructure system 602 may perform big data analysis to discern intent based on received voice input. The data used for this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0105] like Figure 6 As described in the embodiment of FIG. 6 , the cloud infrastructure system 602 may include infrastructure resources 630, which are used to facilitate the provisioning of various cloud services provided by the cloud infrastructure system 602. The infrastructure resources 630 may include, for example, processing resources, storage or memory resources, networking resources, and the like.

[0106] In some embodiments, to facilitate efficient provisioning of these resources for use in supporting the various cloud services provided by the cloud infrastructure system 602 for different customers, the resources may be bundled into resource groups or resource modules (also referred to as "pods"). Each resource module or cluster may include a pre-integrated and optimized combination of one or more types of resources. In some embodiments, different pods may be pre-provisioned for different types of cloud services, which may include those described above for the server 512. For example, a first set of pods may be provisioned for database services, a second set of pods may be provisioned for Java services, and the like, and the second set of pods may include a different combination of resources than the pods in the first set of pods. For some services, the resources allocated for provisioning the service may be shared between services.

[0107] The cloud infrastructure system 602 itself may internally use services 632 that are shared by different components of the cloud infrastructure system 602 and that facilitate provisioning services for the cloud infrastructure system 602. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for implementing cloud support, email services, notification services, file transfer services, and the like.

[0108] The cloud infrastructure system 602 may include multiple subsystems. These subsystems may be implemented in software or hardware or a combination thereof. Figure 6 As depicted, the subsystem may include a user interface subsystem 612 that enables users or customers of the cloud infrastructure system 602 to interact with the cloud infrastructure system 602. The user interface subsystem 612 may include a variety of different interfaces, such as a web interface 614, an online store interface 616 in which cloud services provided by the cloud infrastructure system 602 are advertised and customers may purchase, and other interfaces 618. For example, a customer may use a client device to request (service request 634) one or more services provided by the cloud infrastructure system 602 using one or more of the interfaces 614, 616, and 618. For example, a customer may access an online store, browse cloud services provided by the cloud infrastructure system 602, and place a subscription order for one or more services provided by the cloud infrastructure system 602 to which the customer wishes to subscribe. The service request may include information identifying the customer and the one or more services to which the customer wishes to subscribe. For example, a customer may place a subscription order for NLU-related services provided by the cloud infrastructure system 602. As part of the order, the customer may provide voice input identifying the request.

[0109] In some embodiments, such as Figure 6, cloud infrastructure system 602 may include an order management subsystem (OMS) 620 configured to process new orders. As part of this process, OMS 620 may be configured to: create an account for a customer (if one has not already been created); receive billing and / or billing information from the customer to be used to bill the customer for providing the requested service to the customer; verify the customer information; after verification, book the order for the customer; and orchestrate various workflows to prepare the order for provisioning.

[0110] Once the verification is passed, the OMS 620 can call the order placement subsystem (OPS) 624, which is configured to place resources for the order, including processing resources, memory resources, and networking resources. The placement can include allocating resources to the order and configuring the resources to facilitate the services requested by the customer order. The manner in which resources are placed for the order and the type of resources placed can depend on the type of cloud service that the customer has ordered. For example, according to a workflow, the OPS 624 can be configured to determine the specific cloud service being requested and identify the number of clusters that may have been pre-configured for the specific cloud service. The number of clusters allocated for the order can depend on the size / volume / tier / scope of the requested service. For example, the number of clusters to be allocated can be determined based on the number of users to be supported by the service, the duration of the service being requested, etc. The allocated clusters can then be customized for the specific requesting customer to provide the requested service.

[0111] The cloud infrastructure system 602 may send a response or notification 644 to the requesting customer to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) may be sent to the customer that enables the customer to begin using and utilizing the benefits of the requested service. In some embodiments, for customers requesting NLU-related services, the response may include a response generated based on the identified intent.

[0112] The cloud infrastructure system 602 can provide services to multiple customers. For each customer, the cloud infrastructure system 602 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the order, and providing the requested services to the customer. The cloud infrastructure system 602 can also collect usage statistics about the customer's use of the subscribed services. For example, statistics can be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system up time and system down time. This usage information can be used to bill the customer. Billing can be done, for example, on a monthly basis.

[0113] The cloud infrastructure system 602 may provide services to multiple customers in parallel. The cloud infrastructure system 602 may store information of these customers, including possibly proprietary information. In certain embodiments, the cloud infrastructure system 602 includes an identity management subsystem (IMS) 628 that is configured to manage customer information and provide separation of managed information so that information associated with one customer cannot be accessed by another customer. The IMS 628 may be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles and related capabilities, and the like.

[0114] Figure 7 7 shows an exemplary computer system 700 that can be used to implement certain embodiments. For example, in some embodiments, the computer system 700 can be used to implement any of the ASR subsystem, NLU subsystem, and various servers and computer systems described above. Figure 7 As shown, computer system 700 includes various subsystems, including a processing subsystem 704 that communicates with multiple other subsystems via a bus subsystem 702. These other subsystems may include a processing acceleration unit 706, an I / O subsystem 708, a storage subsystem 718, and a communication subsystem 724. Storage subsystem 718 may include non-transitory computer-readable storage media, including storage media 722 and system memory 710.

[0115] The bus subsystem 702 provides a mechanism for the various components and subsystems of the computer system 700 to communicate with each other as expected. Although the bus subsystem 702 is schematically shown as a single bus, alternative embodiments of the bus subsystem can utilize multiple buses. The bus subsystem 702 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, etc.

[0116] The processing subsystem 704 controls the operation of the computer system 700 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core or multi-core processor. The processing resources of the computer system 700 may be organized into one or more processing units 732, 734, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 704 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some embodiments, some or all of the processing units of the processing subsystem 704 may be implemented using custom circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0117] In some embodiments, the processing unit in the processing subsystem 704 can execute instructions stored in the system memory 710 or on the computer-readable storage medium 722. In various embodiments, the processing unit can execute various programs or code instructions, and can maintain multiple programs or processes that are executed simultaneously. At any given time, some or all of the program code to be executed may reside in the system memory 710 and / or on the computer-readable storage medium 722, including possibly on one or more storage devices. Through appropriate programming, the processing subsystem 704 can provide the various functions described above. In an instance where the computer system 700 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0118] In certain embodiments, a processing acceleration unit 706 may optionally be provided for performing customized processing or offloading some of the processing performed by the processing subsystem 704 in order to speed up the overall processing performed by the computer system 700 .

[0119] The I / O subsystem 708 may include devices and mechanisms for inputting information to the computer system 700 and / or for outputting information from or via the computer system 700. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information to the computer system 900. User interface input devices may include, for example, keyboards, pointing devices such as a mouse or trackball, touch pads or touch screens incorporated into a display, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Windows, which enable a user to control and interact with input devices. Motion Sensor, Microsoft 360 game controllers, devices that provide an interface for receiving input using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices, such as devices that detect eye activity from the user (e.g., "blinking" when taking a picture and / or making a menu selection) and translate eye gestures into input devices (such as Google ) Blink detector. Additionally, the user interface input device may include a voice recognition sensing device that enables a user to interact with the voice recognition system through voice commands.

[0120] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing rods, game controllers and graphics boards, and audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser rangefinders, eye gaze tracking devices. In addition, user interface input devices may include, for example, medical imaging input devices, such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices, such as MIDI keyboards, digital musical instruments, etc.

[0121] In general, the term output device is used to include all possible types of devices and mechanisms for outputting information from the computer system 700 to a user or other computers. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as one using a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0122] The storage subsystem 718 provides a repository or data storage for storing information and data used by the computer system 700. The storage subsystem 718 provides a non-transitory computer-readable storage medium for storing basic programming and data constructs that provide the functionality of some embodiments. The storage subsystem 718 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 704, provide the above-described functionality. The software can be executed by one or more processing units of the processing subsystem 704. The storage subsystem 718 can also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0123] The storage subsystem 718 may include one or more non-volatile storage devices, including volatile and non-volatile storage devices. Figure 7 As shown, the storage subsystem 718 includes a system memory 710 and a computer-readable storage medium 722. The system memory 710 may include multiple memories, including: a volatile main random access memory (RAM) for storing instructions and data during program execution and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) may generally be stored in ROM, and the basic input / output system contains basic routines that help to transfer information between elements within the computer system 700, such as during startup. RAM generally contains data and / or program modules currently operated and executed by the processing subsystem 704. In some embodiments, the system memory 710 may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0124] As an example and not limitation, Figure 7 As described in the , system memory 710 can load executing application programs 712, which can include various applications, such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc., program data 714, and an operating system 716. As an example, the operating system 716 can include various versions of Microsoft Apple and / or Linux operating system, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or mobile operating systems such as iOS, Phone, OS, OS, OS operating system, etc.

[0125] Computer-readable storage media 722 may store programs and data structures that provide the functionality of some embodiments. Computer-readable media 722 may provide storage of computer-readable instructions, data structures, program modules, and other data for computer system 700. Software (programs, code modules, instructions) that provide the functionality described above when executed by processing subsystem 704 may be stored in storage subsystem 718. By way of example, computer-readable storage media 722 may include non-volatile memory such as a hard drive, a disk drive, an optical drive such as a CD ROM, a DVD, a Blu-ray player, or a DVD player. Disks or other optical media. Computer readable storage media 722 may include, but are not limited to: The computer-readable storage medium 722 may include a solid-state drive (SSD) based on a nonvolatile memory, such as a flash memory-based SSD, an enterprise-class flash drive, a solid-state ROM, etc.; a volatile memory-based SSD, such as a solid-state RAM, a dynamic RAM, a static RAM; a DRAM-based SSD; a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD using a combination of a DRAM and a flash memory-based SSD.

[0126] In certain embodiments, storage subsystem 718 may also include a computer-readable storage medium reader 720, which may be further connected to computer-readable storage medium 722. Reader 720 may receive and be configured to read data from a storage device such as a disk, flash drive, or the like.

[0127] In some embodiments, computer system 700 can support virtualization technology, including but not limited to virtualization of processing and memory resources. For example, computer system 700 can provide support for executing one or more virtual machines. In some embodiments, computer system 700 can execute programs, such as management programs that promote the configuration and management of virtual machines. Memory, computing (e.g., processor, core), input / output and networking resources can be allocated to each virtual machine. Each virtual machine usually runs independently of other virtual machines. Virtual machines usually run their own operating systems, which can be the same or different from the operating systems executed by other virtual machines executed by computer system 700. Therefore, computer system 700 may potentially run multiple operating systems simultaneously.

[0128] The communication subsystem 724 provides an interface to other computer systems and networks. The communication subsystem 724 acts as an interface for receiving data from other systems of the computer system 700 and transmitting data to the other systems. For example, the communication subsystem 724 can enable the computer system 700 to establish a communication channel to one or more client devices via the Internet for receiving information from the client devices and sending information to the client devices. For example, the communication subsystem can be used to communicate with a database to perform a query to identify a requestable entity.

[0129] The communication subsystem 724 can support both wired communication protocols and / or wireless communication protocols. For example, in some embodiments, the communication subsystem 724 may include a radio frequency (RF) transceiver component for accessing a wireless voice and / or data network (e.g., using cellular phone technology such as 3G, 4G or EDGE (Enhanced Data Rates for Global Evolution) and other advanced data network technologies, WiFi (IEEE 802.XX family standards, or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, in addition to or in lieu of a wireless interface, the communication subsystem 724 may provide a wired network connection (e.g., Ethernet).

[0130] The communication subsystem 724 can receive and transmit data in various forms. For example, in some embodiments, the communication subsystem 724 can receive incoming communications in the form of structured and / or unstructured data feeds 726, event streams 728, event updates 730, etc., among other forms. For example, the communication subsystem 724 can be configured to receive incoming communications from social media networks and / or other communication services such as feed, The user receives (or sends) data feed 726 in real time, updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0131] In certain embodiments, the communication subsystem 724 may be configured to receive data in the form of a continuous data stream, which may include an event stream 728 of essentially continuous or unbounded real-time events that may not have an explicit end and / or event updates 730. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0132] The communication subsystem 724 may also be configured to communicate data from the computer system 700 to other computer systems or networks. The data may be communicated in various forms such as structured and / or unstructured data feeds 726, event streams 728, event updates 730, etc. to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 700.

[0133] Computer system 700 may be one of various types, including a handheld portable device (e.g., Cellular phone, computing tablets, PDAs), wearable devices (such as Google head mounted display), personal computer, workstation, mainframe, kiosk, server rack, or any other data processing system. Due to the changing nature of computers and networks, the Figure 7 The description of the described computer system 700 is intended only as a specific example. Figure 7 Many other configurations of more or fewer components of the system described in are possible. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will understand other ways and / or methods to implement the various embodiments.

[0134] Although specific embodiments have been described, various modifications, variations, alternative constructions, and equivalents are possible. Embodiments are not limited to operations within certain specific data processing environments, but can be freely operated within multiple data processing environments. In addition, although certain embodiments have been described using a specific series of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be limiting. Although some flow charts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. A process can have additional steps not included in the figure. The various features and aspects of the above-described embodiments can be used individually or in combination.

[0135] In addition, although certain embodiments have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are possible. Certain embodiments may be implemented in hardware only or software only or in a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors.

[0136] Where a device, system, component or module is described as being configured to perform certain operations or functions, such configuration may be accomplished, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations, such as by executing computer instructions or code, or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0137] Specific details are given in this disclosure to provide a thorough understanding of the embodiments. However, embodiments can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures and techniques have been shown without unnecessary details to avoid obscuring the embodiments. This description only provides example embodiments and is not intended to limit the scope, applicability or configuration of other embodiments. Instead, the previous description of the embodiments will provide an enabling description for implementing various embodiments for those skilled in the art. Various changes can be made to the functions and arrangements of the elements.

[0138] Therefore, the description and drawings should be viewed in an illustrative rather than a restrictive sense. However, it will be apparent that additions, subtractions, deletions and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, although specific embodiments have been described, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

Claims

1. A method for dialog processing, comprising: receiving, by the dialog system, speech input from a user; processing, by the dialog system, a first portion of the speech input using a trained neural network to determine an initially discerned domain, wherein the neural network has been trained to predict likely domains based on sample domain-to-utterance pairs; causing, by the dialog system, display of a visual indication of the initially discerned domain, in response to determining the initially discerned domain and upon receiving the speech input, wherein the visual indication of the initially discerned domain comprises an image indicative of the initially discerned domain, the initially discerned domain comprising categories corresponding to a plurality of intents; as well as A second portion of the speech input is processed by the dialog system to determine a modified recognized domain, wherein the second portion of the speech input completes the first portion of the speech input and clarifies the initial recognized domain in response to the visual indication of the initial recognized domain, wherein the modified recognized domain is different from the initial recognized domain, and wherein the first portion of the speech input and the second portion of the speech input are processed as a continuous input stream.

2. The method of claim 1, wherein: The dialog system processes the speech input substantially in real time.

3. The method of claim 1, further comprising performing, by the dialog system, a task corresponding to the modified identified domain.

4. The method of claim 1, further comprising: determining, by the dialog system, a response based on the modified discerned domain; as well as The response is provided to the user by the dialog system.

5. The method of claim 1, wherein: Determining the initially identified domain includes: calculating, by the dialog system, a plurality of scores for a corresponding plurality of potential domains and the first portion of the speech input; and determining, by the dialog system, that a score of a specific domain among the plurality of potential domains exceeds a threshold, thereby setting the specific domain as the initially identified domain, wherein causing display of a visual indication of the initially identified domain is triggered by said determining that the score exceeds the threshold.

6. The method of claim 1, wherein: Determining the initially identified domain includes: determining, by an automatic speech recognition subsystem of the dialog system, a first text utterance corresponding to the first portion of the speech input; providing, by the automatic speech recognition subsystem, the first text utterance to the natural language understanding subsystem of the dialog system; and The initially discerned domain is determined, by the natural language understanding subsystem, based on the first text utterance.

7. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform a process comprising: receiving voice input from a user; processing a first portion of the speech input using a trained neural network to determine an initially discerned domain, wherein the neural network has been trained to predict likely domains based on sample domain-to-utterance pairs; in response to determining the initially discerned domain and upon receiving the speech input, causing display of a visual indication of the initially discerned domain, wherein the visual indication of the initially discerned domain comprises an image indicative of the initially discerned domain, the initially discerned domain comprising categories corresponding to a plurality of intents; as well as processing a second portion of the speech input to determine a modified recognized domain, wherein the second portion of the speech input completes the first portion of the speech input and clarifies the initial recognized domain in response to a visual indication of the initial recognized domain, wherein the modified recognized domain is different from the initial recognized domain, and wherein the first portion of the speech input and the second portion of the speech input are processed as a continuous input stream.

8. The non-transitory computer readable memory of claim 7, wherein: The speech input is processed in substantially real time.

9. The non-transitory computer readable memory of claim 7, the processing further comprising performing a task corresponding to the modified identified domain.

10. The non-transitory computer readable memory of claim 7, the processing further comprising: determining a response based on the modified discriminated domain; as well as The response is provided to the user.

11. The non-transitory computer readable memory of claim 7, wherein determining the initially discerned domain comprises: calculating a plurality of scores for a corresponding plurality of potential domains and the first portion of the speech input; as well as determining that a score of a specific domain among the plurality of potential domains exceeds a threshold, thereby setting the specific domain as the initially identified domain, wherein causing display of a visual indication of the initially identified domain is triggered by said determining that the score exceeds the threshold.

12. The non-transitory computer readable memory of claim 7, wherein determining the initially discerned domain comprises: determining, by an automatic speech recognition subsystem, a first text utterance corresponding to the first portion of the speech input; providing, by the automatic speech recognition subsystem, the first text utterance to the natural language understanding subsystem; as well as The initially discerned domain is determined, by the natural language understanding subsystem, based on the first text utterance.

13. A system for dialogue processing, comprising: one or more processors; A memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to perform a process including: receiving voice input from a user; processing a first portion of the speech input using a trained neural network to determine an initially discerned domain, wherein the neural network has been trained to predict likely domains based on sample domain-to-utterance pairs; In response to determining the initially discerned domain and upon receiving the speech input, causing display of a visual indication of the initially discerned domain, wherein the visual indication of the initially discerned domain comprises an image indicative of the initially discerned domain, the initially discerned domain comprising categories corresponding to a plurality of intents; and processing a second portion of the speech input to determine a modified recognized domain, wherein the second portion of the speech input completes the first portion of the speech input and clarifies the initial recognized domain in response to a visual indication of the initial recognized domain, wherein the modified recognized domain is different from the initial recognized domain, and wherein the first portion of the speech input and the second portion of the speech input are processed as a continuous input stream.

14. The system of claim 13, wherein: The speech input is processed in substantially real time.

15. The system of claim 13, the processing further comprising performing a task corresponding to the modified identified domain.

16. The system of claim 13, wherein the processing further comprises: determining a response based on the modified discriminated domain; as well as The response is provided to the user.

17. The system of claim 13, wherein: Determining the initially identified domain includes: calculating a plurality of scores for a corresponding plurality of potential domains and the first portion of the speech input; and determining that the score of the initially discriminated domain exceeds a threshold, wherein display of a visual indication of the initially identified domain is triggered by the determining that the score of the initially identified domain exceeds the threshold.

18. A system for dialog handling, comprising means for performing the steps of the method according to any one of claims 1 to 6.

Images