Streaming action implementation based on some assumptions

By real-time voice recognition and interpretation of user voice data on mobile devices and performing sub-actions related to voice commands in advance, the problems of unreliability and delay in applications on mobile devices are solved, and the user experience and device response speed are improved.

CN116888663BActive Publication Date: 2024-11-08GOOGLE LLC
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202180092877.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-12-08
Filing Date
2021-11-16
Publication Date
2024-11-08
Estimated Expiration
2041-11-16

AI Technical Summary

Technical Problem

The existing digital assistant interfaces installed on mobile devices may be slow, unreliable, or require network access to the server, resulting in latency and instability of user experience, suppressing the processing power of voice recognition on the device.

Method used

By receiving user voice data on the data processing hardware, performing voice recognition and partial interpretation in real time, identifying the application types and sub-actions required for the action in advance, thereby performing related sub-actions before the user finishes his speech, reducing the delay of user perception.

Benefits of technology

It significantly reduces user-aware voice command execution delay, improves voice processing capabilities on the device, and enhances user experience and device response speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116888663B_ABST
    Figure CN116888663B_ABST
Patent Text Reader

Abstract

A method (300) for streaming action implementation receives audio data (14) corresponding to an utterance (20), wherein the utterance includes a query (22) to perform an action (24), the action requiring execution of a sequence of sub-actions (26) in order to complete the action. Upon receiving the audio data, but before receiving a speech end condition (214), the method processes the audio data to generate an intermediate automatic speech recognition (ASR) result (212), performs a partial query interpretation on the intermediate ASR result to determine whether the intermediate ASR result identifies an application type required to perform the action, and when the intermediate ASR result identifies a specific application type, executes a first sub-action in the sequence of sub-actions by launching a first application (118) to be executed on a user device, wherein the first application is associated with the specific application type. The method completes execution of the action.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to streaming action implementation based on partial assumptions. Background Art

[0002] Users frequently interact with voice-enabled devices such as smartphones, smartwatches, and smart speakers through digital assistant interfaces. These digital assistant interfaces enable users to complete tasks and get answers to all of their questions through natural, conversational interactions. Ideally, when conversing with a digital assistant interface, a user should be able to communicate as if the user were talking to another person through spoken requests directed toward their voice-enabled device running the digital assistant interface. The digital assistant interface will provide these spoken requests to an automatic speech recognizer to process and recognize the spoken requests so that actions can be performed.

[0003] Digital assistant interfaces are moving to mobile devices, where the majority of speech recognition processing occurs on the device without incurring the additional latency of connecting to a server over a network and sending audio data to the server to perform speech recognition using a cloud-based speech recognition server. In addition to improved latency, other advantages of on-device speech recognition include improved reliability and privacy. In this way, digital assistant interfaces are becoming deeply integrated with various applications and operating systems running on mobile devices, enabling users to control their mobile devices using only their voice. However, these applications installed on the user's device may themselves be slow, unreliable, or require network access to a server, thereby inhibiting the benefits of on-device processing power provided by the digital assistant interface and may result in a slow user experience. Summary of the invention

[0004] One aspect of the present disclosure provides a method for streaming action implementation. The method includes receiving audio data corresponding to an utterance spoken by a user of a user device at data processing hardware, wherein the utterance includes a query to perform an action, wherein the query requires execution of a sequence of sub-actions in order to complete the action. When the audio data is received before a speech end condition is received, the method also includes processing, by the data processing hardware, a first portion of the received audio data using a speech recognizer to generate a first sequence of intermediate automatic speech recognition (ASR) results. When the audio data is received before a speech end condition is received, the method also includes performing, by the data processing hardware, a partial query interpretation on the first sequence of intermediate ASR results to determine whether the first sequence of intermediate ASR results identifies an application type required to perform the action, and when the first sequence of intermediate ASR results identifies a specific application type, by the data processing hardware, executing a first sub-action in the sequence of sub-actions by launching a first application to be executed on the user device, wherein the first application is associated with the specific application type. The method additionally includes, in response to receiving a speech end condition, completing execution of the action by the data processing hardware. Determining an end-of-speech condition may include detecting at least a minimum duration of non-speech in the received audio using a speech recognizer.

[0005] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, when the audio data is received before the voice end condition is received and after the first application is launched, the method further includes, by the data processing hardware, processing a second portion of the received audio data using the speech recognizer to generate a second sequence of intermediate ASR results, performing the partial query interpretation on the second sequence of intermediate ASR results by the data processing hardware to identify a search query for content in the first application, and performing the second sub-action in the action sequence by the data processing hardware by instructing the first application to execute the search query for the content in the first application. In these implementations, in response to launching the first application to be executed on the user device, the method further includes, by the data processing hardware, displaying an initial screen in a graphical user interface of the user device, the initial screen having a search area and / or a plurality of graphical elements, each graphical element representing a different category of content in the first application. Here, instructing the first application to execute a search query for the content includes entering text representing a search query for the content in the first application into the search area of ​​the first screen, or selecting the graphical element representing a content category including the content specified by the search query from the plurality of graphical elements of the first screen. These embodiments may also include, after the first application performs the search query for the content, displaying, by the data processing hardware, a second screen in the graphical user interface, the second screen presenting one or more results including the content specified by the search query. In these embodiments, the method may further include receiving a user input indication at the data processing hardware, the user input indication indicating a selection of one of the results presented in the second screen, and detecting, by the data processing hardware, the speech end condition in response to receiving the user input indication. These embodiments may also include, after the first application performs the search query for the content, processing, by the data processing hardware, a third portion of the received audio data using the speech recognizer to generate a third sequence of intermediate ASR results, performing, by the data processing hardware, the partial query interpretation on the third sequence of intermediate ASR results to determine whether to update the search query for more specific content, and when the partial query interpretation performed on the third sequence of intermediate search results determines to update the search query for more specific content, performing, by the data processing hardware, a third sub-action in the sub-action sequence by instructing the first application to update the search query for more specific content.

[0006] In some configurations, performing the partial query interpretation on the first sequence of the intermediate ASR results can determine that the first sequence of the intermediate ASR results identifies the specific application type and fails to specify a slot value associated with naming a specific application for completing the action. Here, executing the first sub-action in the sub-action sequence includes launching the first application as a default application associated with the specific application type. In these configurations, when the audio data is received before receiving the speech end condition and after launching the first application, the method also includes processing, by the data processing hardware, a second portion of the received audio data using a speech recognizer to generate a second sequence of intermediate ASR results, and performing, by the data processing hardware, a partial query interpretation on the second sequence of the intermediate ASR results to determine whether the second sequence of the intermediate ASR results identifies a second application for completing the action. In addition, in these configurations, when the second sequence of the intermediate ASR results identifies a second application for completing the action, the method further includes, by the data processing hardware, rolling back the execution of the first sub-action by stopping execution of the first application on the user device, and re-executing, by the data processing hardware, the first sub-action in the action sequence by launching the second application for execution on the user device.

[0007] In some examples, when the audio data is received before receiving the voice end condition and after launching the first application, the method further includes processing, by the data processing hardware, a second portion of the received audio data using a speech recognizer to generate a second sequence of the intermediate ASR results, and performing, by the data processing hardware, the partial query interpretation on the second sequence of the intermediate ASR results to identify the second sub-action in the sequence of sub-actions. In these examples, when the audio data is received before receiving the voice end condition and after launching the first application, the method further includes determining, by the data processing hardware, a rollback feasibility score associated with the second sub-action, wherein the rollback feasibility indicates a likelihood that the user experience will be degraded if the execution of the second sub-action must be rolled back, and executing, by the data processing hardware, the second sub-action when the rollback feasibility score satisfies a rollback feasibility threshold. When the rollback feasibility score does not satisfy the rollback feasibility threshold, the method may delay, by the data processing hardware, the execution of the second sub-action until the voice end condition is received. Also in these examples, the method may further include determining, by the data processing hardware, a confidence score for an identified second sub-action by performing the partial query interpretation on a second sequence of the intermediate ASR results, and when the confidence score for the second sub-action does not meet a confidence threshold, prompting, by the data processing hardware, the user to confirm whether the second sub-action is correctly identified.

[0008] Another aspect of the present disclosure provides a system for streaming action implementation. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving audio data corresponding to an utterance spoken by a user of a user device, wherein the utterance includes a query for performing an action, wherein the query requires a series of sub-actions to be performed in order to complete the action. When the audio data is received before receiving a speech end condition, the operations also include processing a first portion of the received audio data using a speech recognizer to generate a first sequence of intermediate automatic speech recognition (ASR) results. When the audio data is received before receiving a speech end condition, the operations further include performing a partial query interpretation on the first sequence of the intermediate ASR results to determine whether the first sequence of the intermediate ASR results identifies an application type required to perform the action, and when the first sequence of the intermediate ASR results identifies a specific application type, executing a first sub-action in the sub-action sequence by launching a first application to execute on the user device, wherein the first application is associated with the specific application type. The operations additionally include, in response to receiving a speech end condition, completing the execution of the action. Determining an end-of-speech condition may include detecting, using a speech recognizer, at least a minimum duration of non-speech in the received audio.

[0009] Implementations of the present disclosure may include one or more of the following optional features. In some embodiments, when the audio data is received before the voice end condition is received and after the first application is launched, the operation further includes processing a second portion of the received audio data using the speech recognizer to generate a second sequence of intermediate ASR results, performing the partial query interpretation on the second sequence of intermediate ASR results to identify a search query for content in the first application, and performing a second sub-action in the sub-action sequence by instructing the first application to execute a search query for the content in the first application. In these embodiments, in response to launching the first application to be executed on the user device, the operation further includes displaying an initial screen in a graphical user interface of the user device, the initial screen having a search area and / or multiple graphical elements, each graphical element representing a different category of content in the first application. Here, instructing the first application to execute a search query for the content includes entering text representing the search query for the content in the first application into the search area of ​​the first screen, or selecting the graphical element representing a content category including the content specified by the search query from the multiple graphical elements of the first screen. These embodiments may also include, after the first application executes the search query for the content, displaying a second screen in the graphical user interface, the second screen presenting one or more results including the content specified by the search query. In these embodiments, the operation may further include receiving an indication indicating a selection of one of the results presented in the second screen, and detecting the speech end condition in response to receiving the user input indication. These embodiments may also include, after the first application executes the search query for the content, processing a third portion of the received audio data using the speech recognizer to generate a third sequence of intermediate ASR results, performing the partial query interpretation on the third sequence of intermediate ASR results to determine whether to update the search query for more specific content, and when the partial query interpretation performed on the third sequence of intermediate search results determines to update the search query for more specific content, performing a third sub-action in the sub-action sequence by instructing the first application to update the search query for the more specific content.

[0010] In some configurations, performing the partial query interpretation on the first sequence of the intermediate ASR results can determine that the first sequence of the intermediate ASR results identifies the specific application type and fails to specify a slot value associated with naming a specific application for completing the action. Here, executing the first sub-action in the sub-action sequence includes launching the first application as a default application associated with the specific application type. In these configurations, when the audio data is received before receiving the speech end condition and after launching the first application, the operation also includes processing a second portion of the received audio data using the speech recognizer to generate a second sequence of intermediate ASR results, and performing a partial query interpretation on the second sequence of the intermediate ASR results by the data processing hardware to determine whether the second sequence of the intermediate ASR results identifies a second application for completing the action. In addition, in these configurations, when the second sequence of the intermediate ASR results identifies the second application for completing the action, the operation further includes rolling back the execution of the first sub-action by stopping execution of the first application on the user device, and re-executing the first sub-action in the action sequence by launching the second application on the user device by the data processing hardware.

[0011] In some examples, when the audio data is received before receiving the speech end condition and after launching the first application, the operation further includes processing a second portion of the received audio data using the speech recognizer to generate a second sequence of intermediate ASR results, and querying the execution portion of the second sequence of intermediate ASR results by the data processing hardware to identify the second sub-action in the sub-action sequence. In these examples, when the audio data is received before receiving the speech end condition and after launching the first application, the operation also includes determining a rollback feasibility score associated with the second sub-action, wherein the rollback feasibility indicates the likelihood that the user experience will be degraded if the execution of the second sub-action must be rolled back, and executing the second sub-action by the data processing hardware when the rollback feasibility score meets a rollback feasibility threshold. When the rollback feasibility score does not meet the rollback feasibility threshold, the operation may delay execution of the second sub-action until the speech end condition is received. Also in these examples, the operation may further include determining, by the data processing hardware, a confidence score for a second sub-action identified by performing the partial query interpretation on a second sequence of the intermediate ASR results, and when the confidence score for the second sub-action does not meet a confidence threshold, prompting, by the data processing hardware, the user to confirm whether the second sub-action is correctly identified.

[0012] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram of an example speech environment for performing an implementation based on partial speech recognition results.

[0014] Figures 2A-2H is used for Figure 1 Schematic diagram of an example assistant interface for a voice environment.

[0015] Figure 3 is a flowchart of an example operation arrangement of a method implemented based on partial speech recognition results.

[0016] Figure 4 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0017] The same reference symbols in different drawings denote the same elements. DETAILED DESCRIPTION

[0018] Traditionally, digital assistant interfaces that enable users to complete tasks and get answers to questions through natural conversational interactions require server processing to support speech recognition and language understanding models. These server-based models are not suitable for execution on devices because their size may exceed the available storage or memory on the device. However, recent advances in recurrent neural networks have enabled the development of new speech recognition and language understanding models with significantly reduced sizes (e.g., less than 0.5 gigabytes) suitable for storage and processing on devices. Similarly, digital assistant interfaces are moving to mobile devices, where many speech recognition processes occur on the device without causing additional delays to connect to the server over the network and send audio data to the server to perform speech recognition using a cloud-based speech recognition server. Since speech transcription can be performed in real time without a network connection, the user experience is greatly enhanced. In addition to improving latency, other advantages of speech recognition on devices include improved reliability and privacy. In this way, digital assistant interfaces are becoming deeply integrated with various applications and operating systems running on mobile devices, enabling users to control their mobile devices using only their voice. For example, users can perform multiple tasks across multiple applications running on the device, such as creating calendar invitations, finding and sharing photos with friends, or dictating emails. However, these applications installed on the user's device may themselves be slow, unreliable, or require network access to a server, thereby inhibiting the benefits provided by the on-device digital assistant interface and potentially resulting in a slow user experience.

[0019] Typically, in speech recognition systems, speech endpointing is the process of determining which portion of the input audio contains speech by determining the beginning and end of an utterance. The portion of the input audio corresponding to the speech is provided to a speech recognizer to obtain a speech recognition result or transcript of the audio. The user-perceived speech recognition latency is the time from when the user stops speaking to when the speech recognition result or transcript is output, typically for display on a screen of the user's device. Recent advances in running speech recognition models on-device have allowed real-time speech recognition results (e.g., streaming transcription) to be displayed as the user speaks and before the endpoint determines the end of the utterance.

[0020] However, although the user-perceived latency of actual speech recognition has been improved by processing speech on the device, the user-perceived latency of implementing the user query / command still exists because the end of the recognized utterance must be made before the actual implementation occurs. For example, a "play rock music playlist on YouTube Music" command spoken by a user to a digital assistant interface executed on the user's phone requires the execution of multiple sub-actions in order to complete the full action of streaming the user's rock music playlist from the YouTube Music service for audible output through an acoustic speaker. These sub-actions include launching the YouTube Music app on the user's phone, accessing the search box in YouTube Music to enter a query for a rock music playlist, or switching to the Playlists tab in YouTube Music to select a rock music playlist, and finally performing audible playback of the rock music playlist from the YouTube Music app. In this example, before launching the first sub-action that needs to be completed (e.g., launching the YouTube Music app), the system must wait for the end pointer to end the utterance and obtain the final speech recognition result.

[0021] Embodiments herein are directed to generating intermediate speech recognition results from a user utterance commanding the execution of an action, and using the intermediate speech recognition results to execute multiple sub-actions associated with a final action while the user is still speaking the utterance. Using the available intermediate speech recognition results to recognize and execute sub-actions associated with a final action before the user finishes speaking greatly reduces user-perceived latency because there is no need to wait for the final end before starting to execute the first sub-action in a sequence of sub-actions that need to be executed to complete the final action.

[0022] However, the intermediate recognition results of "play", "rock music" and "playlist" appear before the final end and can be used to identify possible sub-actions related to the query. For example, the intermediate recognition result of "play" can indicate that the user wants to play media content, which can include video content such as TV shows, movies or video clips, or audio content from one of multiple music applications on the user's device. Next, the intermediate recognition result of "rock music" provides a context that narrows the type of media content that the user wants to play to audio content. Now, the possible available sub-actions are to open a streaming radio application, a Spotify application or a YouTube application on the user's mobile phone, all of which are likely to output rock music. For example, streaming radio, Spotify and YouTube applications all include their own rock music channels, and Spotify and YouTube applications each include corresponding rock music playlists for users. Once the intermediate recognition result of "playlist" becomes available, the available possible sub-actions are now more specific to indicate that a rock music playlist needs to be accessed on the Spotify application or the YouTube application.

[0023] refer to Figure 1 In some embodiments, the speech environment 100 includes a user 10 speaking an utterance 20 to a speech-enabled device 110 (also referred to as device 110 or user device 110) executing a digital assistant interface 200. Here, the utterance 20 spoken by the user 10 can be captured by the device 110 in streaming audio 15 and can correspond to a query to perform an action 24, more specifically, a query 22 corresponding to the digital assistant interface 200 performing the action 24. When a hot word detector running on the device 110 detects a hot word 21 in the streaming audio 15 in a sleep or hibernation state, the user 10 can use the hot word 21 as a prefix of the query 22 (e.g., a call phrase) to trigger the device 110 from the sleep or hibernation state. The action 24 can also be referred to as an operation or task. In this sense, the user 10 can interact with the digital assistant interface 200 executing on the speech-enabled device 110 in a conversation to perform computing activities or find answers to questions.

[0024] Device 110 may correspond to any computing device associated with user 10 and capable of capturing audio from environment 100. Some examples of user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smart watches), music players, projection devices, smart appliances (e.g., smart TVs) and Internet of Things (IoT) devices, remote controls, smart speakers, etc. Device 110 includes data processing hardware 112d and memory hardware 112m, which communicates with data processing hardware 112d and stores instructions that, when executed by data processing hardware 112d, cause data processing hardware 112d to perform one or more operations related to speech processing.

[0025] The device 110 further includes an audio subsystem having an audio capture device (e.g., an array of one or more microphones) 114 for capturing audio within the speech environment 100 and converting it into electrical signals (e.g., audio data 14 ( Figure 2A-2E )). Although in the illustrated example, the device 110 implements an audio capture device 114 (also often referred to as a microphone 114), the audio capture device 114 may not physically reside on the device 110, but rather communicate with an audio subsystem (e.g., a peripheral device of the device 110). For example, the device 110 may correspond to a vehicle infotainment system that utilizes an array of microphones throughout the vehicle.

[0026] The device 110 may also include a display 116 to display graphical user interface (GUI) elements (e.g., windows, screens, icons, menus, etc.) and / or graphical content. For example, the device 110 may load or launch an application 118, 118a-n that generates GUI elements or other graphical content for the display 116. In addition, the elements generated in the display 116 may be selected by the user 10 and also serve to provide some form of visual feedback to the processing activities and / or operations occurring on the device 110. In addition, because the device 110 is a voice-enabled device 110, the user 10 may interact with the elements generated on the display 116 using various voice commands. For example, the display 116 may depict a menu of options for a particular application 118, and the user 10 may select an option by voice using the interface 200.

[0027] In some embodiments, device 110 communicates with remote system 130 via network 120. Remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). Device 110 may utilize remote resources 132 to perform various functions related to speech processing. For example, certain portions of assistant interface 200 may reside on remote system 130. In one example, speech recognizer 210 is executed locally on device 110 to provide on-device automatic speech recognition (ASR). In another example, speech recognizer 210 resides on remote system 130 to provide server-side ASR for assistant interface 200. In another example, the functionality of speech recognizer 210 spans device 110 and server 130 (e.g., device 110 and server 130 perform parallel ASR processing).

[0028] The voice-enabled interface 200 may field a query / command 22 conveyed in a spoken utterance 20 captured by the device 110 in the streaming audio 15. The voice-enabled interface 200 (also referred to as the interface 200 or assistant interface 200) generally facilitates receiving audio data 14 corresponding to the utterance 20 captured by the audio capture device 114, coordinating speech processing of the audio data 14, performing semantic interpretation of the audio data 14 to identify a query 22 to perform an action 24, and performing a sequence of sub-actions 26, 26a-n to complete the action 24. The interface 200 may be executed on the data processing hardware 112d of the device 110. When the microphone 114 of the device 110 captures the utterance 20 in the streaming audio 15 and converts the audio into audio data 14, the audio data 14 corresponding to the utterance 20 is forwarded to the interface 200 so that the interface 200 can perform speech recognition and semantic interpretation to recognize and ultimately complete the query 22 conveyed in the spoken utterance 20. While conventionally, implementation systems must wait for an end pointer (e.g., a portion of a speech recognizer or a separate component) to complete the end of the utterance 20 to obtain a final speech recognition result before beginning to perform the action 24 specified by the query / command 22, here, the interface 200 implements the sub-action 26 associated with the final action 24 while a partial speech recognition result (e.g., a partial hypothesis) is generated, and before the user 10 finishes speaking the query 22 in the utterance 20. By utilizing the partial speech recognition result, the interface 200 can provide the user 10 with real-time or near real-time feedback of the sub-action 26 that the interface 200 is performing in order to complete the action 24 specified by the query 22 at the end of the utterance 20. In other words, while the user 10 is actively speaking the utterance 20, the interface 200 strives to perform the sub-action 26 interpreted from the intermediate speech recognition result of the audio data 14.

[0029] In some implementations, in order to perform the sequence of actions 26 to complete action 24, the interface 200 interacts with a plurality of applications 118, 118a-n on the device 110 or accessible to the device 110. An application 118 generally refers to any application 118 configured to run on the device 110. Some types of applications 118 include media applications (e.g., video streaming applications, audio streaming applications, media player applications, media gallery applications, etc.), word processing applications, navigation applications, social media applications, communication applications (e.g., messaging applications, email applications, etc.), financial applications, organizational applications (e.g., address book applications), retail applications, entertainment applications (e.g., news applications, weather applications, sports applications), projection applications, etc. The interface 200 can be integrated with these applications 118 to enable the user 10 to control the applications on the device 110 using his or her voice. For example, the interface 200 is an application programming interface (API) or any other type of program or application configured to perform the functions of the interface 200.

[0030] Continue to refer Figure 1 , user 10 utters utterance 20 stating "Hey, computer, show me pictures of my vacation trip to Greece with my parents." Here, utterance 20 is a voice query / command 22 requesting interface 200 to perform action 24 of displaying a particular set of user photos. When interface 200 receives audio data 14 corresponding to utterance 20, portions of utterance 20 are processed as they are received at interface 200 to generate intermediate speech result 212 ( Figure 2A-2E ). As the user 10 is speaking the utterance 20, the interface 200 interprets the intermediate speech results 212 to identify sub-actions 26 associated with the candidate action 24. For example, once the utterance 20 is completed, the complete action 24 specified by the query 22 includes a sequence of four sub-actions 26, 26a-d, which can be performed while the user 10 is speaking the utterance 20, rather than waiting for the user 10 to finish speaking the utterance 20.

[0031] refer to Figures 2A-2H, the assistant interface 200 generally includes a speech recognizer 210, an interpreter 220, and an executor 230. The speech recognizer 210 receives the audio data 14 as input and processes the audio data 14 to generate a speech recognition result 212. When the speech recognizer 210 performs speech recognition on the audio data 14, the speech recognizer 210 can generate an intermediate speech result 212. Here, the intermediate speech recognition result 212 refers to a result generated on a subset of the audio data 14 (e.g., a portion of the audio data) rather than the entire audio data 14. In general, speech recognition can be for an entire phrase (e.g., an entire utterance) or a subset of a phrase, such as characters, word fragments, and / or words. In other words, each recognition result in a sequence of speech recognition results (e.g., a sequence 212 of intermediate speech recognition results) can correspond to a character, word fragment, and / or word that is combined to form a final recognition result (e.g., transcription) of the utterance 20. For example, when speech recognition occurs on an audio stream, a speech recognition model that performs speech recognition can generate output (i.e., results or hypotheses) in a streaming manner at each time step. Here, the frame-by-frame output may correspond to intermediate speech recognition results, which may be combined to form one or more sequences 212 of speech recognition results representing corresponding portions of the audio data 14. In some configurations, the speech recognition results correspond to the top N hypotheses at any given time step, so that multiple hypotheses may be processed at once to generate all possible partial query interpretations. Since the audio data 14 may correspond to an utterance 20 spoken by the user 10, a portion of the audio data 14 may correspond to a portion of the utterance 20. For example, when the audio data 14 corresponds to the utterance 20 "showing pictures of my vacation trip to Greece with my parents," a portion of the audio data 14 may be a sequence of characters that forms one or more words. As Figure 2B As shown in FIG. 2D , the first intermediate speech recognition result sequence 212 a of the first portion 14 , 14 a of the audio data 14 includes the phrase “showing photos”. The second intermediate speech recognition result sequence 212 b of the second portion 14 , 14 b of the audio data 14 includes the phrase “from my holiday trip to Greece”. The third intermediate speech recognition result sequence 212 c of the third portion 14 , 14 c of the audio data 14 includes the phrase “with my parents”.

[0032] The interpreter 220 receives one or more sequences 212 of intermediate speech recognition results and performs partial query interpretation on the one or more intermediate speech recognition results 212. For example, the interpreter 220 performs semantic interpretation (e.g., grammatical interpretation) on the sequence of intermediate speech recognition results 212 to understand a portion of the utterance 20 and its context, thereby identifying any candidate sub-actions 26 that may be associated with the final action 24, which will be specified once the query 22 is revealed when the user 10 finishes speaking the utterance 20. Here, because the interpreter 220 is interpreting the sequence of intermediate speech recognition results 212 corresponding to only a portion of the query 22, the interpreter 220 is able to derive the context of the sub-action 26 from the sequence of intermediate speech recognition results 212 corresponding to a portion of the utterance 20. Therefore, the result of the partial query interpretation performed on the sequence of intermediate speech results 212 includes an interpretation 222 that can characterize the sub-actions 26 that the query / command 22 will involve in order to complete the final action 24, which is unknown until the user finishes speaking the utterance 20. When the interpreter 220 performs a partial query interpretation, the interpretation 222 may suffer from some lost information due to the inherent fact that the interpretation cannot place the entire utterance 20 in context. In other words, as the number of sequences of intermediate speech recognition results 212 generated from the audio data 14 increases, the sub-actions 26 represented by the interpretations 222 can become increasingly specific. To this end, the interpreter 220 can form an interpretation 222 that is as complete as possible by deriving the missing intent from the available information from the sequence of intermediate speech recognition results 212. For example, the interpreter 220 can perform a partial query interpretation on an initial sequence of intermediate speech recognition results 212 to identify a specific application type required to perform an action 24, but because the interpreter 220 cannot confidently identify the specific application 118 from the initial sequence of intermediate speech recognition results 212, it cannot specify a slot value associated with naming the specific application 118 used to complete the action 24. In this example, because the slot value associated with naming the specific application 118 is unspecified (eg, null), the executor 230 may be configured to perform the first sub-action 26 by launching a default application 118 associated with the specific application type.

[0033] refer to Figure 2BIn the illustrated example, when the interpreter 220 generates a first interpretation 222, 222a for the phrase "show me photos" indicated by the first sequence 212, 212a of intermediate speech recognition results, the interpreter 220 clearly lacks the context of the specific photos to be displayed when interpreting. Due to this limited context, the interpreter 220 generates a more general interpretation 222, 222a that indicates that the user 10 may have issued a query 22 that may require launching the photo application 118 on the device 110. This means that even if the name of the specific photo application cannot be derived from the partial query interpretation performed on the first sequence 212a of intermediate speech recognition results, the interpreter 220 can infer a specific application type (e.g., a photo application) from the first intermediate speech recognition result sequence 212a "show me photos". In addition, as the user 10 continues to speak the utterance 20, the speech recognizer 210 generates additional sequences 212, 212b-d of intermediate speech recognition results, and the interpreter 220 performs partial query interpretation on each sequence of intermediate speech recognition results 212 to generate corresponding interpretations 222, 222b-d that provide additional context to the query 22. In other words, as the user 10 speaks, the interpreter 220 performs partial query interpretation on each sequence of intermediate speech recognition results 212 to identify and construct a sequence of sub-actions 26 that need to be performed in order to complete the action 24. The executor 230 may perform some or all of these sub-actions 26 before the user finishes speaking, and then finally completes the implementation of the action 24 when the user 10 finishes speaking the utterance 20 in response to receiving the end-of-speech condition. At the time of receiving the end-of-speech condition, the executor 230 may have already performed each sub-action 26 in the sequence of sub-actions 26 required to complete the query 22, and therefore, the action 24 has been completed at the time the end-of-speech condition is received.

[0034] exist Figure 2C , the interpreter 220 performs partial query interpretation on the second sequence 212, 212b of intermediate speech recognition results of “from my Greece holiday trip” to generate a second interpretation 222, 222b that identifies a search filter (e.g., a search query) on a location variable of the user 10 within the photo application 118. Based on the second interpretation 222b, the executor 230 may perform a second sub-action 22, 26c by filtering photos in the local photo gallery application with respect to the location variable. For example, Figure 2C Filtering a user's photo albums with respect to a location variable is shown.

[0035] In some examples, a subsequent interpretation 222 may invalidate a previous interpretation 222. For example, if a first interpretation 222, 222a causes the executor 230 to launch two plausible photo applications 118 in parallel, one of which is a local photo gallery application and the other is a third-party photo gallery accessible via a browser application, then a second interpretation 222, 222b that identifies the time filter as being specifically related to the user 10 (e.g., based on the word "my" preceding the word "vacation") will invalidate / roll back the action of opening the third-party photo gallery because that gallery will not include any user-specific photos.

[0036] refer to Figure 2D , the interpreter 220 performs partial query interpretation on the third sequence 212, 212c of intermediate speech recognition results of "and my parents" to generate a third interpretation 222, 222c, which identifies another search filter (e.g., search query) for the presence of the user's parents in the photos. This means that the third interpretation 222c will cause the executor 230 to perform a third sub-action 26, 26c, namely, further filtering the photos in the local photo library application by subject (i.e., the subject of the user's parents).

[0037] In some configurations, the interpreter 220 uses an interpretation model that generates a confidence level for a given interpretation 222. In some embodiments, the interpreter 220 generates multiple possible interpretations 222 for the same sequence of intermediate speech recognition results 212, and each possible interpretation 222 can have a corresponding confidence level. In addition, the speech recognizer 210 can generate multiple different candidate sequences of intermediate speech recognition results 212 for the same portion of the audio data 14, and the interpreter 220 can generate one or more possible interpretations 222 for each candidate sequence of the intermediate speech recognition results 222. However, in some methods, the interface 200 may only want to pursue a limited number of interpretations 222 (e.g., one interpretation 222) or an interpretation 222 indicating a confidence level higher than a certain interpretation confidence threshold. Here, when the interpreter 220 generates multiple possible interpretations 222 for a given sequence of intermediate speech recognition results 212 and has a confidence level that satisfies the interpretation confidence threshold, the executor 230 can process the corresponding sub-actions 26 characterized by the possible interpretations 222 in parallel. With the sub-actions 26 processed in parallel, the interface 200 may graphically display each parallel trajectory on the display 116 and enable the user 10 to select a particular trajectory, or even modify his or her utterance 20 to change the behavior of the interpreter 220 and / or the actuator 230 .

[0038] continue Figure 2BIn the example of -2D, when the executor 230 receives the first interpretation 222a that identifies the photo application as a specific application type required to perform the action 24, the executor 230 performs the first sub-action 26a by launching the first photo application 118. Here, although the partial query interpretation performed on the first sequence of intermediate ASR results 212a identifies a specific application type (e.g., a photo application), the partial query interpretation cannot specify a slot value associated with naming a specific photo application for completing the action 24. Therefore, the first photo application 118 corresponds to a default photo application that the user 10 may have previously designated for use as a default (e.g., for photos), or a default photo application that the user 10 uses most frequently. Figure 2B As shown, the display 116 may depict the launching of the first photo application 118 by showing a graphical icon indicating that the first photo application 118 is selected and continuing to display a graphical user interface (GUI) 240 of the first photo application 118 .

[0039] like Figure 2C As shown, when the actuator 230 receives the second interpretation 222b identifying the search filter related to the photos of the user during the vacation in Greece, the actuator 230 performs the second sub-action 26b by instructing the first photo application 118 launched on the device 110 to perform a search query for the photos of the user during the vacation in Greece. Figure 2B and 2C Initially, the application 118 displays two photo albums categorized by location upon launch. One album depicts photos of London and the other depicts photos of Greece. Here, the executor 230 may instruct the application to include text representing the search query (e.g., saying "Greece" as a search term 242). ST ) is entered into the search area 242 of the GUI 240 of the first photo application 118 displayed on the display 116. By executing the search query, Figure 2C Nine photo thumbnails from the album "Greece" are depicted. After the second sub-action 26b, the executor 230 receives a third interpretation 222c that identifies a search filter related to photos of the user 10 in Greece that include the user's parents. The executor 230 performs the third sub-action 26c by sorting the photos in the "Greece" album by person. Here, the search / filter selects four of the nine photos from the results of the previous sub-step 26b and displays these four photos within the first photo application 118. Figure 2DAs shown, the GUI 240 of the first photo application 118 includes two different methods with different GUI elements for searching or filtering content within the first photo application 118. In this example, the user 10 can use the first GUI element corresponding to the search area 242 to search for content of the application 118 (e.g., search for photos within the first photo application 118). The application 118 also includes a GUI element corresponding to a classification / feature option menu. Specific to this example, the menu 244 depicts a list of photo tags that allows the user 10 to classify his or her photos by tags that have been associated with the photos of the first photo application 118. Tags generally refer to identifiers that can be shared between one or more content elements (e.g., photos). In this example, the actuator 230 can select the option "Parents" within the menu 244 to instruct the actuator 230 to filter the user's Greek photos by whether the user's parents are present.

[0040] In some configurations, when the utterance 20 is completed or ends, the actuator 230 performs the complete implementation of the action 24. Here, the utterance 20 can be terminated when the speech recognizer 210 detects a certain specified non-speech minimum duration in the audio data 14. Based on the complete speech recognition results of the entire audio data 14 (or utterance 20), the actuator 230 can execute the highest confidence action 24. In these configurations, the actuator 230 can roll back (undo or terminate) the previous sub-action 26 that occurred before the entire action 24 was completed. For example, the actuator 230 rolls back one or more sub-actions 26 that are inconsistent with the entire action 24. In general, the sub-action execution process is intended to be in a final state that matches the final state of the execution process of the complete action 24. However, depending on the query / command 22, this may not always be the case. Therefore, rollback allows the interface 200 to flexibly adapt to different scenarios.

[0041] In some embodiments, the executor 230 may roll back the previous sub-action 26 based not only on the entire action 24, but also on other sub-actions 26 before the utterance 20 was terminated. To illustrate, as described above, the first interpretation 222 of the interpreter may have resulted in two plausible photo-related types of applications 118, one of which is a local photo gallery application and the other is a third-party photo gallery in a browser application. However, the subsequent interpretation 222 of the sub-action 26 confirms that the application 118 cannot be a third-party photo gallery application because it will not contain photos of the user 10. In this regard, the executor 230 will roll back the first sub-action 26 for launching the third-party photo gallery in favor of the local photo gallery application. If the executor 230 actually launches the third-party photo gallery, but does not launch the local photo gallery application, then based on the subsequent interpretation 222, the executor 230 can roll back the first sub-action 26 for launching the third-party photo gallery by stopping the execution of the third-party photo gallery, and re-execute the first sub-action 26 by launching the local photo gallery application instead.

[0042] Additionally or alternatively, there may be certain sub-actions 26 that the interface 200 cannot roll back, or rolling back the sub-actions 26 will negatively affect the user's experience. For example, when the sub-action 26 is to purchase an item on a retail application 118, the interface 200 may not be able to roll back such a purchase or do so without user intervention. Here, the sub-actions 26 that the interface 200 cannot roll back can be referred to as irreversible sub-actions 26. In these irreversible sub-actions 26, the actuator 230 can prompt the user 10 to authorize or confirm the action when executing the irreversible sub-action 26. Another method of the irreversible sub-action 26 is to identify the irreversible sub-action 26, and when the sub-action 26 is identified as irreversible, the actuator 230 waits to perform the implementation of the action 24. That is, it may be safer to interpret the entire context of the complete speech recognition result than to interpret the partial speech recognition result 212. In another method, before the end of the speech 20, but after the actuator 230 starts the application 118, the actuator 230 can determine the rollback feasibility score of the sub-action 26. Here, the rollback feasibility score indicates the likelihood that the user experience will be degraded or adversely affected if the executor 230 rolls back the sub-action 26. When the rollback feasibility score satisfies the rollback feasibility threshold, the executor 230 may continue to execute the sub-action 26. On the other hand, when the rollback feasibility score fails to satisfy the rollback feasibility threshold, the executor 230 may roll back the sub-action 26 or delay the rollback of the sub-action 26 to determine whether the full implementation of the action 24 indicates that the sub-action 26 should be rolled back, and roll it back accordingly.

[0043] Because the user 10 can see the results of the sub-action 26 performed by the interface 200 while the user 10 is speaking, in some cases, the user 10 can manually end the utterance before the user has finished speaking the entire utterance. That is, when the user 10 is looking for a specific result from the device 110, if the specific result is displayed to the user 10 before the utterance 20 is completed, the user 10 can give up completing the utterance 20 because the purpose has been achieved. For example, using Figure 2B-2D For example, if all user photos in the Greece album include the user's parents, the second sub-action 26b and the third sub-action 26c will produce the same result. In this case, the user 10 can recognize that the device 110 has successfully displayed photos of him or her and his or her parents during the Greece trip. Based on this recognition, the user 10 can end the speech 20 in advance (i.e., before saying the part "and my parents").

[0044] Figure 2E-2H An example interface 200 is shown when an executor 230 performs multiple tasks in parallel based on an utterance 20 spoken by a user 10. In this example, the user 10 speaks an utterance 20 directed to the device 110, the utterance 20 including the query 22 "play rock music playlist on Google Play music". When the user 10 speaks the utterance 20, the interface receives audio data 14 corresponding to the utterance 20 and forwards the audio data 14 to the speech recognizer 210. For a first portion 14, 14a of the audio data 14 including the word "play", the speech recognizer 210 provides a first sequence 212, 212a of intermediate speech recognition results, such as Figure 2EAs shown. The interpreter 220 receives the first sequence 212a of speech recognition results and performs partial query interpretation thereon to obtain first interpretations 222, 222a. Based on the word "play", the first interpretation 222a indicates that the audio data 14 includes a media-related command for an application 118 of a media type, and the executor 230 executes a first set of four sub-actions 26, 26a-d that launch four different applications 118, 118a-d on the device 110. The first application 118a includes a first music streaming application Google Play. The second application 118b includes a second music streaming application Spotify. The third application 118c includes a video streaming application YouTube, and the fourth application 118d includes a local general video player application called "video player". As the sequence of sub-actions 26 continues and the query / command spoken in the utterance 20 becomes more specific, these four applications 118a-d can be launched to execute in parallel on the device 110. When the executor 230 launches these applications 118a-d, these applications 118a-d may appear in one or more windows (e.g., a cascade of windows or a panel of windows) on the device 110. Because the first portion of the utterance 20 lacks a clear and identifiable application 118 to "play" something, when the interface begins to execute the action 24 corresponding to the command 22 of the utterance 20, the interpreter 220 generates a plurality of candidate sub-actions 26a-d, which may include applications 118 of all media types, rather than specific applications 118. In addition, because the execution process of each sub-action 26 can be displayed to the user 10 on the display 116 of the device 110, the user 10 (e.g., while speaking the utterance 20) may engage the display 116 to further facilitate or modify the sub-action 26. For example, for the four applications 118 displayed to the user 10, the user 10 may close one or more of the applications 118 by interacting with a GUI element of the GUI 240 of the application 218, which has the function of terminating the selected application 118. Conversely, the user 10 may additionally or alternatively select which applications 118 to retain rather than terminate. In another example, the user 10 may select to retain the application 118 or to terminate the application 118.

[0045] As the user 10 continues to speak utterance 20, Figure 2FThe speech recognizer 210 is depicted performing speech recognition on a second portion 14b of the audio data 14 associated with the word "rock music" to generate a second sequence 212, 212b of intermediate speech recognition results. The interpreter 220 interprets the second sequence 212b of intermediate speech recognition results to obtain two different interpretations 222, 222b-c. A first interpretation 222b, in which "rock music" refers to a type of music (i.e., a rock music genre), or a second interpretation 222, in which "rock music" refers to a playlist of the user 10 (e.g., a playlist title). Because both of these interpretations 222b-c can have a confidence level that satisfies the confidence threshold, the executor 230 can generate parallel processes to execute these two interpretations 222b-c on each application 118 launched and opened from the first set of sub-actions 26a-d.

[0046] The executor 230 may also use the second interpretation 222b and the third interpretation 222c as validity checks to determine whether any of the sub-actions 26 in the first set of sub-actions 26a-d should be rolled back (e.g., terminated). Since the two applications 118c-d are applications for video (e.g., streaming video or a local video player), performing the validity check on the second interpretation 222b and the third interpretation 222c causes the executor 230 to terminate the video-based applications 118c-d that were launched based on the third and fourth sub-actions 26, 26c-d. In this case, only the two interpretations 222b-c need to be executed at the first application 118a and the second application 118b. Therefore, the execution process forms a second set of sub-actions 26e-h, in which four parallel action sequences appear, such as Figure 2F shown.

[0047] exist Figure 2G, as the user 10 continues to speak the utterance 20, the speech recognizer 210 performs speech recognition on the third portion 14c of the audio data 14 associated with the word "playlist" to generate a third sequence 212, 212c of intermediate speech recognition results, on which the interpreter 220 performs partial query interpretation to determine a fourth interpretation 222, 222d. At this step of the processing, the interpreter 220 may be able to recognize that the sequence 212, 212c of intermediate speech recognition results clarifies some ambiguity between the second and third interpretations 222b-c. That is, "rock music" should not be interpreted as a search for genres of music, but rather as a playlist called "rock music". Based on these additional contextual clues, the executor 230 has terminated any processes that were being executed under the premise that "rock music" may correspond to a genre. After these processes are terminated, this leaves two remaining action sequences, one for the first streaming music application 118 of Google Play Music and one for the second streaming music application 118 of Spotify. At this point in the sequence of actions, each application 118 has been launched and the playlist "Rock Music" has been searched according to the sub-action 261-j corresponding to the fourth interaction 222d.

[0048] exist Figure 2G , user 10 speaks the last part of utterance 20. When user 10 speaks the last part of utterance 20 and at the end 214 ( Figure 2H )(or speech end condition), the speech recognizer 210 performs speech recognition on the fourth portion 14, 14d of the audio data 14 related to the utterance "on Google Play music". Here, the speech recognizer 210 generates a fourth sequence 212, 212d of intermediate speech recognition results for the fourth portion 14d of the audio data 14. The speech recognizer 210 forwards the fourth sequence 212d of intermediate speech recognition results to the interpreter 220, and the interpreter 220 performs partial query interpretation to generate a fifth interpretation 222, 222e, which identifies the fourth sequence 212d of intermediate speech recognition results as uniquely specifying a specific application Google Play music. In other words, although the specific application 118 is defined at the end of the utterance 20, due to the context of the utterance 20 before this portion of the utterance "on Google Play music", the application 118 was already executing when the user 10 was speaking. Based on the fifth interpretation 222e, the executor 230 stops the execution of the Spotify application 118. Additionally, when end 214 of utterance 20 occurs, interpreter 220 may compare the sequence of sub-actions with action 24 to complete the complete speech recognition result from speech recognizer 210 .

[0049] Figure 3 Flowchart of an example arrangement of operations based on a partially hypothetical streaming action implementation. In operation 302, the method 300 receives audio data 14 corresponding to an utterance 20 spoken by a user 10 of a user device 110, wherein the utterance 20 includes a query 22 to perform an action 24, wherein the query 22 requires a sequence of sub-actions 26 to be performed in order to complete the action 24. Operation 304 includes three sub-operations 304, 304a-c, which occur when the method 300 receives the audio data 14, but before receiving the speech end condition 214. In operation 304a, the method 300 processes the first portion 14, 14a of the received audio data 14 using the speech recognizer 210 to generate a first sequence 212 of intermediate automatic speech recognition (ASR) results. In operation 304b, the method 300 performs a partial query interpretation on the first sequence 212 of the intermediate ASR results to determine whether the first sequence 212 of the intermediate ASR results identifies the application type required to perform the action. In operation 304c, when the first sequence 212 of intermediate ASR results identifies the specific application type, a first sub-action 26 in the sequence of sub-actions 26, 26a-n is performed by launching a first application 118 to be executed on the user device 110, wherein the first application 118 is associated with the specific application type. In operation 306, the method 300 includes completing the performance of action 24 in response to receiving the speech end condition 214.

[0050] Figure 4 2 is a schematic diagram of an example computing device 400 that can be used to implement the systems (e.g., interface 200) and methods (e.g., method 300) described in this document. Computing device 400 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are merely exemplary and are not meant to limit the implementation of the inventions described and / or claimed in this document.

[0051] The computing device 400 includes a processor 410, a memory 420, a storage device 430, a high-speed interface / controller 440 connected to the memory 420 and a high-speed expansion port 450, and a low-speed interface / controller 460 connected to a low-speed bus 470 and the storage device 430. Each of the components 410, 420, 430, 440, 450, and 460 are interconnected using various buses and can be installed on a common motherboard or in other appropriate ways. The processor 410 may include the data processing hardware 112d of the user device 110 or the data processing hardware 134 of the remote system 130. Therefore, the data processing hardware 410 can process instructions for execution within the computing device 400, including instructions stored in the memory 420 or on the storage device 430, to display graphical information of a graphical user interface (GUI) on an external input / output device (e.g., a display 480 coupled to the high-speed interface 440). In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and multiple memory types, can be used appropriately. Furthermore, multiple computing devices 400 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0052] The memory 420 stores information non-transitorily within the computing device 400. The memory 420 may include the memory hardware 112m of the user device 110 or the memory hardware 136 of the remote system 130. The memory hardware 420 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transitory memory hardware 420 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0053] Storage device 430 can provide mass storage for computing device 400. In some embodiments, storage device 430 is a computer readable medium. In various embodiments, storage device 430 can be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state storage device, or a device array, including devices in a storage area network or other configuration. In other embodiments, a computer program product is tangibly contained in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as a memory 420, a storage device 430, or a memory on a processor 410.

[0054] The high-speed controller 440 manages bandwidth-intensive operations of the computing device 400, while the low-speed controller 460 manages less bandwidth-intensive operations. This allocation of responsibilities is exemplary only. In some embodiments, the high-speed controller 440 is coupled to the memory 420, the display 480 (e.g., through a graphics processor or accelerator), and is coupled to a high-speed expansion port 450 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 460 is coupled to the storage device 430 and the low-speed expansion port 490. The low-speed expansion port 490, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router, through, for example, a network adapter.

[0055] As shown, computing device 400 can be implemented in many different forms. For example, it can be implemented as a standard server 400a or multiple times in a group of such servers 400a, can be implemented as a laptop 400b, or can be implemented as part of a rack server system 400c.

[0056] Various implementations of the systems and techniques described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs executable and / or interpretable on a programmable system, the programmable system comprising at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0057] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-volatile computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0058] The processes and logic flows described in this specification can be performed by one or more programmable processors, executing one or more computer programs to perform functions by operating input data and generating output. The processes and logic flows can also be performed by dedicated logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits). For example, processors suitable for executing computer programs include general-purpose and dedicated microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic components of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more large-capacity storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data or transfer data to them, or both. However, a computer does not require such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor storage devices, such as EPROMs, EEPROMs, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0059] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) display, or touch screen for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; the user's input may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0060] Many implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A method for implementing a streaming action, characterized in that: include: At data processing hardware, receiving audio data corresponding to an utterance spoken by a user of a user device, the utterance comprising a query to perform an action, the query requiring a sequence of sub-actions to be performed in order to complete the action; When the audio data is received before a speech end condition is received: processing, by the data processing hardware, a first portion of the received audio data using a speech recognizer to generate a first sequence of intermediate automatic speech recognition (ASR) results; performing, by the data processing hardware, a partial query interpretation on the first sequence of intermediate ASR results to determine whether the first sequence of intermediate ASR results identifies an application type required to perform the action; When the first sequence of intermediate ASR results identifies a specific application type, performing, by the data processing hardware, a first sub-action in the sequence of sub-actions by launching a first application to be executed on the user device, the first application being associated with the specific application type; and After launching the first application: processing, by the data processing hardware, a second portion of the received audio data using the speech recognizer to generate a second sequence of the intermediate ASR results; performing, by the data processing hardware, the partial query interpretation on a second sequence of the intermediate ASR results to identify a second sub-action in the sequence of sub-actions; determining, by the data processing hardware, a rollback feasibility score associated with the second sub-action, the rollback feasibility indicating a likelihood that a user experience will be degraded if execution of the second sub-action must be rolled back; as well as When the rollback feasibility score satisfies a rollback feasibility threshold, executing the second sub-action by the data processing hardware; and In response to receiving the speech end condition, the data processing hardware completes the execution of the action.

2. The method according to claim 1, characterized in that performing, by the data processing hardware, the partial query interpretation on a second sequence of the intermediate ASR results to identify a search query for content in the first application; as well as The second sub-action in the sequence of sub-actions performed by the data processing hardware includes instructing the first application to perform the search query for the content in the first application.

3. The method according to claim 2, characterized in that Further comprising, in response to launching the first application to be executed on the user device: The data processing hardware displays an initial screen in a graphical user interface of the user device, the initial screen having a search area and / or a plurality of graphical elements, each graphical element representing a different category of content in the first application, Instructing the first application to execute the search query for the content includes: entering text representing the search query for the content in the first application into the search area of ​​the initial screen; or The graphical element representing a content category including the content specified by the search query is selected from the plurality of graphical elements of the initial screen.

4. The method according to claim 3, characterized in that Further included, after the first application executes the search query for the content, displaying, by the data processing hardware, a second screen in the graphical user interface, the second screen presenting one or more results including the content specified by the search query.

5. The method according to claim 4, characterized in that Further including: receiving, at the data processing hardware, a user input indication indicating a selection of one of the results presented in the second screen; The end-of-speech condition is detected by the data processing hardware in response to receiving the user input indication.

6. The method according to claim 2, characterized in that Further comprising, after the first application performs the search query for the content: processing, by the data processing hardware using the speech recognizer, a third portion of the received audio data to generate a third sequence of intermediate ASR results; performing, by the data processing hardware, the partial query interpretation on a third sequence of the intermediate ASR results to determine whether to update the search query to more specific content; as well as When the partial query interpretation performed on the third sequence of intermediate search results determines to update the search query for more specific content, the data processing hardware performs a third sub-action in a sequence of sub-actions by instructing the first application to update the search query for the more specific content.

7. The method according to claim 1, characterized in that performing the partial query interpretation on the first sequence of intermediate ASR results to determine that the first sequence of intermediate ASR results identifies the specific application type and fails to specify a slot value associated with naming a specific application for completing the action; as well as Executing the first sub-action in the sequence of sub-actions includes launching the first application as a default application associated with the specific application type.

8. The method according to claim 7, characterized in that performing, by the data processing hardware, a partial query interpretation on the second sequence of intermediate ASR results to determine whether the second sequence of intermediate ASR results identifies a second application for completing the action; and The method further comprises, when receiving the audio data before receiving the voice end condition and after starting the first application: When the second sequence of intermediate ASR results identifies the second application for completing the action: Rolling back the execution of the first sub-action by stopping the execution of the first application on the user device, by the data processing hardware; as well as The data processing hardware re-executes the first sub-action in the sequence of sub-actions by launching the second application to execute on the user device.

9. The method according to claim 1, characterized in that: Further comprising, when receiving the audio data before receiving the voice end condition: When the first sequence of intermediate ASR results identifies the specific application type, performing, by the data processing hardware, the first sub-action in the sequence of sub-actions by launching the first application and a second application for parallel execution on the user device, the first application and the second application both being associated with the specific application type; as well as The data processing hardware simultaneously displays a first graphic element and a second graphic element in a graphical user interface of the user device, wherein the first graphic element represents the first application executed on the user device, and the second graphic element represents the second graphic element executed on the user device.

10. The method according to claim 9, characterized in that Further including: receiving, at the data processing hardware, a user input indication indicating a selection of one of the first graphical element or the second graphical element displayed in the graphical user interface; as well as In response to receiving the user input indication: maintaining, by the data processing hardware, execution of one of the first application or the second application represented by a selected one of the first graphical element or the second graphical element; as well as The data processing hardware stops executing the other of the first application or the second application represented by one of the first graphic element or the second graphic element that is not selected by the user input indication.

11. The method according to claim 1, characterized in that: Further comprising, when the rollback feasibility score does not satisfy the rollback feasibility threshold, delaying, by the data processing hardware, execution of the second sub-action until the speech end condition is received.

12. The method according to claim 1, characterized in that Further including: determining, by the data processing hardware, a confidence score for the second sub-action identified by performing the partial query interpretation on a second sequence of the intermediate ASR results; as well as When the confidence score of the second sub-action does not meet a confidence threshold, the data processing hardware prompts the user to confirm whether the second sub-action is correctly identified.

13. The method according to claim 1, characterized in that Determining the end-of-speech condition includes detecting, using the speech recognizer, at least a minimum duration of non-speech in the received audio.

14. A system for implementing streaming actions, characterized in that: include: Data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising: receiving audio data corresponding to an utterance spoken by a user of a user device, the utterance comprising a query to perform an action, the query requiring performance of a sequence of sub-actions in order to complete the action; When the audio data is received before receiving the speech end condition: processing a first portion of the received audio data using a speech recognizer to generate a first sequence of intermediate automatic speech recognition (ASR) results; performing a partial query interpretation on a first sequence of intermediate automatic speech recognition (ASR) results to determine whether the first sequence of intermediate automatic speech recognition (ASR) results identifies an application type required to perform the action; and When the first sequence of intermediate ASR results identifies a specific application type, performing a first sub-action in the sequence of sub-actions by initiating a first application to execute on the user device, the first application being associated with the specific application type; and After launching the first application: processing a second portion of the received audio data using the speech recognizer to generate a second sequence of intermediate ASR results; performing the partial query interpretation on a second sequence of the intermediate ASR results to identify a second sub-action in the sequence of sub-actions; determining a rollback feasibility score associated with the second sub-action, the rollback feasibility indicating a likelihood that a user experience will be degraded if execution of the second sub-action must be rolled back; and When the rollback feasibility score meets the rollback feasibility threshold, executing the second sub-action; and In response to receiving the speech end condition, the data processing hardware completes the execution of the action.

15. The system according to claim 14, characterized in that performing the partial query interpretation on a second sequence of the intermediate ASR results to identify a search query for content in the first application; as well as Performing the second sub-action in the sequence of sub-actions includes instructing the first application to perform the search query for the content in the first application.

16. The system according to claim 15, characterized in that The operations further include, in response to launching the first application to be executed on the user device: The data processing hardware displays an initial screen in a graphical user interface of the user device, the initial screen having a search area and / or a plurality of graphical elements, each graphical element representing a different category of content in the first application, Wherein instructing the first application to execute the search query for the content includes: entering text representing the search query for the content in the first application into the search area of ​​the initial screen; or The graphical element representing a content category including the content specified by the search query is selected from the plurality of graphical elements of the initial screen.

17. The system according to claim 16, characterized in that The operations further include, after the first application executes the search query for the content, displaying, by the data processing hardware, a second screen in the graphical user interface, the second screen presenting one or more results including the content specified by the search query.

18. The system according to claim 17, characterized in that The operations further include: receiving a user input indication indicating a selection of one of the results presented in the second screen; In response to receiving the user input indication, the speech end condition is detected.

19. The system according to claim 15, characterized in that The operations further include, after the first application performs the search query for the content: processing, by the data processing hardware using the speech recognizer, a third portion of the received audio data to generate a third sequence of intermediate ASR results; performing, by the data processing hardware, the partial query interpretation on a third sequence of the intermediate ASR results to determine whether to update the search query to more specific content; as well as When the partial query interpretation performed on the third sequence of intermediate search results determines to update the search query for more specific content, a third sub-action in a sequence of sub-actions is performed by the data processing hardware by instructing the first application to update the search query for the more specific content.

20. The system according to claim 14, characterized in that performing the partial query interpretation on the first sequence of intermediate ASR results to determine that the first sequence of intermediate ASR results identifies the specific application type and fails to specify a slot value associated with naming a specific application for completing the action; as well as Executing the first sub-action in the sequence of sub-actions includes launching the first application as a default application associated with the specific application type.

21. The system according to claim 20, characterized in that performing a partial query interpretation on the second sequence of intermediate ASR results to determine whether the second sequence of intermediate ASR results identifies a second application for completing the action; and The operation further includes, when receiving the audio data before receiving the voice end condition and after starting the first application: When the second sequence of intermediate ASR results identifies the second application for completing the action: rolling back the execution of the first sub-action by stopping execution of the first application on the user device; and The first sub-action in the sequence of sub-actions is re-executed by launching the second application to be executed on the user equipment.

22. The system according to claim 14, characterized in that When the audio data is received before the voice end condition is received, the operation further includes: When the first sequence of intermediate ASR results identifies a specific application type, performing the first sub-action in the sequence of sub-actions by launching the first application and a second application for parallel execution on the user device, the first application and the second application both being associated with the specific application type; and A first graphical element and a second graphical element are simultaneously displayed in a graphical user interface of the user device, the first graphical element representing the first application executed on the user device, and the second graphical element representing the second graphical element executed on the user device.

23. The system according to claim 22, characterized in that Further including: receiving, at the data processing hardware, a user input indication indicating a selection of one of the first graphical element or the second graphical element displayed in the graphical user interface; as well as In response to receiving the user input indication: maintaining, by the data processing hardware, execution of one of the first application or the second application represented by a selected one of the first graphical element or the second graphical element; as well as The data processing hardware stops executing the other of the first application or the second application represented by one of the first graphic element or the second graphic element that is not selected by the user input indication.

24. The system according to claim 14, characterized in that The operation further includes, when the rollback feasibility score does not meet the rollback feasibility threshold, delaying execution of the second sub-action until the speech end condition is received.

25. The system according to claim 14, characterized in that The operations further include: determining a confidence score for a second sub-action identified by performing the partial query interpretation on a second sequence of intermediate ASR results; and When the confidence score of the second sub-action does not meet the confidence threshold, the user is prompted to confirm whether the second sub-action is correctly identified.

26. The system according to claim 14, characterized in that Determining the end-of-speech condition includes detecting, using the speech recognizer, at least a minimum duration of non-speech in the received audio.

Citation Information

Patent Citations

  • Initiating actions based on partial hotwords

    CN105793921A

  • Headless task completion within digital personal assistants

    CN107111516A

  • Automatic speech recognition (ASR) processing using confidence measures

    CN1115902A

  • Configurable speech recognition system using multiple recognizers

    US20120179457A1

  • Method for operating speech recognition service and electronic device for supporting the same

    US20190066674A1