Alternative interface proposal in predicting that environmental interference prevents specific interaction of automatic assistant
By detecting environmental interference and suggesting alternative interfaces, the automated assistant reduces incorrect actions and resource waste in noisy environments, enhancing interaction reliability.
Patent Information
- Application Number
- JP2024134997
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-11-20
- Filing Date
- 2024-08-13
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-10-12
AI Technical Summary
Users interacting with automated assistants face interference in noisy environments, leading to repeated inputs and incorrect actions, wasting computational resources.
The automated assistant determines environmental interference and suggests alternative interfaces, such as a keyboard, to avoid input modalities affected by noise, reducing the need for repeated inputs and incorrect actions.
This approach conserves computational resources by minimizing the number of incorrect actions and repeated inputs, improving interaction reliability in noisy conditions.
Smart Images

Figure 2024167237000001_ABST
Abstract
Description
[Technical field]
[0001] and suggesting alternative interfaces when environmental interference is predicted to impede a particular interaction of the automated assistant. [Background technology]
[0002] Humans can engage in human-computer interactions using interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "conversational agents," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) can provide commands and / or requests using verbal natural language input (i.e., speech) and / or by providing textual (e.g., typed) natural language input, which may in some cases be converted to text and then processed.
[0003] Users interacting with automated assistants may be forced to invoke their respective automated assistants under circumstances where the modality for receiving user input is subject to interference. Such circumstances may include crowded public places and / or other areas where background noise is evident. Thus, the input modality subject to interference may not then be a trusted interface for the automated assistant. As a result, a user providing input to a temporarily untrusted interface may have to repeat their input. In some cases, when a user is attempting to control an automated assistant with their input, the automated assistant may request that the user repeat the input and / or initiate the performance of an erroneous action. Repeating an input and / or performing an erroneous action may result in wasted computational resources if the repeated input and / or unintended action is to be processed at the client computing device and / or communicated to a server over a network. Summary of the Invention [Means for solving the problem]
[0004] Implementations described herein relate to an automated assistant that can determine whether an ongoing or anticipated interaction between a user and an automated assistant through a particular interface is predicted to be affected by interference. When the automated assistant determines that an interaction in a particular interface is determined to be affected by interference, the automated assistant can provide an indication that the user should interact with the automated assistant through a separate interface. In this manner, the automated assistant can conserve computational resources by reducing the number of times a user must repeat an input that may have been affected by some amount of environmental interference. Additionally, by providing an indication, the number of instances in which the automated assistant initiates an incorrect action as a result of an interaction being affected by a particular interference may be reduced.
[0005] In some cases, a user may be located in an environment with some amount of background noise caused by the interaction of other people in the environment. At this time, the user may hold the portable computing device in hopes of invoking his / her automated assistant to answer a particular query (e.g., "Assistant, what time am I meeting Joe tomorrow?"). In some implementations, before the user provides input to his / her automated assistant, or at the same time as the user provides input to the automated assistant, the automated assistant may determine whether the input will be or is being occluded by background noise. For example, when the user input is embodied in a verbal utterance, background noise caused by the remaining people may be captured in the audio that also captures the verbal utterance. The automated assistant may cause an indicator to be rendered to the user when the automated assistant determines that the input is expected to be affected by background noise and / or other interference. The indicator may serve to inform the user that his / her input may not be fully received by the automated assistant and / or that the user may have another input modality available to provide input. For example, in some implementations, the indicia may be a rendering of a keyboard interface that a user may use to type input to the automated assistant instead of speaking to the automated assistant.
[0006] In some implementations, one or more features of the user's environment can be characterized by data used to determine whether one or more features of the environment will affect the user input to some extent. This data can be processed to generate one or more scores, which can be compared to a threshold that, when met, indicates that the user input is predicted to be affected by one or more features of the environment. For example, the score can be based on the number of people in the environment, the number of people speaking in the environment, one or more previous instances in which the user engaged with an automated assistant in a similar environment, the type of background noise, the volume level of the background noise, one or more images of the environment, and / or any other data that can be used to characterize the environment.
[0007] In some implementations, the score and / or additional score can be based on the availability of a separate modality for interacting with the automated assistant. For example, when a user is in an environment where the automated assistant is accessible through a stand-alone speaker device that does not include a graphical user interface, the additional score can be generated to reflect the limited availability of any other interface (e.g., a keyboard interface rendered in a graphical user interface). Alternatively, when a user is in another environment where the automated assistant is accessible through a stand-alone speaker device and a mobile computing device that is in the user's pocket, the additional score can be generated to reflect the rich interface with which the user can interact with the automated assistant. In some implementations, the additional score can be generated to reflect the availability of an automated assistant interface that is not expected to be interfered with in the environment. The automated assistant can determine whether to provide an indication that another interface is available and / or that a particular type of input is expected to be interfered with based on the interference score for the environment and, optionally, the interface availability score. In some implementations, one or more scores can be updated over time based on environmental changes and / or changes in the availability of a particular automated assistant interface.
[0008] In some implementations, the one or more scores can be based on processing data from one or more different sources using one or more trained machine learning models. The trained machine learning models can be trained using training data based on prior cases in which one or more different types of interferences affected and / or did not affect the interaction between the user and the automated assistant. For example, one or more data sources can characterize the environment in which the user is present, and the data can be processed using one or more trained machine learning models to generate an embedding. This embedding can be compared to other embeddings in the latent space to determine whether the environment exhibits interferences that will affect the interaction between the user and the automated assistant. Alternatively or in addition, the one or more trained machine learning models can also be used to classify and / or otherwise determine the probability that the user is understood. In some implementations, the distance from this embedding to another embedding in the latent space can be compared to a threshold or another distance to determine whether to suggest that the user should interface with the automated assistant via a particular interface modality. For example, a comparison of embeddings in a latent space and / or through heuristics may result in one or more interfaces being designated as being affected by interference and / or one or more other interfaces being designated as being less affected by interference.
[0009] The above description has been provided as a summary of some implementations of the disclosure. Further descriptions of these and other implementations are set forth in more detail below.
[0010] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) for performing a method, such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include a system of one or more computers including one or more processors operable to execute stored instructions for performing a method, such as one or more of the methods described above and / or elsewhere herein.
[0011] It should be understood that any combination of the foregoing concepts, and additional concepts described in more detail herein, is contemplated as being part of the present subject matter disclosed herein, for example, any combination of claimed subject matter appearing at the end of this disclosure is contemplated as being part of the present subject matter disclosed herein. [Brief description of the drawings]
[0012] [Figure 1A] FIG. 13 illustrates a view in which a user receives a suggestion to type input to an automated assistant instead of speaking the input in an environment that may interfere with spoken input. [Figure 1B] FIG. 13 illustrates a view in which a user receives a suggestion to type input to an automated assistant instead of speaking the input in an environment that may interfere with spoken input. [Diagram 2] FIG. 1 illustrates a system for suggesting alternative interface modalities when an automated assistant and / or a user is predicted not to understand a particular interaction between a user and an automated assistant. [Diagram 3]FIG. 1 illustrates a method for providing an indication of whether a particular input to an automated assistant will be affected by interference in the environment and for providing a separate interface for providing input to the automated assistant. [Figure 4] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] 1A and 1B respectively show views 100 and 120 in which a user 102 receives a suggestion to type input to an automated assistant instead of speaking the input in an environment 110 that may interfere with verbal input. For example, the user 102 may be present in the environment 110, such as the exterior of his / her apartment building, and may ask his / her automated assistant a question that he / she would like to ask. To access the automated assistant, the user 102 may have a computing device 106 that allows the user 102 to interact with the automated assistant through one or more interfaces. For example, the computing device 106 may include a display interface 104, which may be a touch-enabled display panel, and an audio interface 108, which may include a speaker and / or a microphone. In some implementations, the computing device 106 may also include a camera to provide another interface for interacting with the automated assistant.
[0014] In some implementations, the automated assistant can, with prior permission from the user 102, determine whether one or more characteristics of the environment 110 will affect the interaction between the user 102 and the automated assistant. For example, the automated assistant can determine that the user 102 is on a crowded sidewalk based on audio and / or image data available to the computing device 106. In some implementations, data from one or more sources can be processed to determine whether input to a particular interface of the computing device 106 will be subject to interference from the environment 110 and / or other sources. Based on this processing, the automated assistant can choose to provide the user 102 with an indication that input to a particular interface will be subject to interference while present in the environment 110. For example, initially, the display interface 104 may not have any indication that the particular interface will be subject to interference, such as background noise (e.g., multiple people talking as shown in FIGS. 1A and 1B) and / or other interference. However, as presented in view 120 of FIG. 1B, the automated assistant and / or other application may cause a keyboard 124 to be rendered in the display interface 104 based on determining that a particular interface will be subject to a particular interference.
[0015] In some implementations, one or more characteristics of the keyboard 124 can be based at least in part on the extent to which interference is predicted to affect input to a particular interface (e.g., a microphone) of the computing device 106. For example, the size of the keyboard 124 can be adjusted according to the extent to which interference is predicted to affect input to a particular interface. In some implementations, when interference is not predicted to affect verbal input to the automated assistant, the automated assistant can optionally cause the input field 112 to be rendered in the display interface 104. However, when interference is predicted to affect input to the automated assistant, the automated assistant can maintain or remove the input field 112 from the display interface 104 and cause the keyboard 124 to be rendered in the display interface 104. This can allow the user 102 to use their hands 122 to type input to the automated assistant without subjecting the verbal input to audible interference.
[0016] FIG. 2 illustrates a system 200 for suggesting an automated assistant and / or alternative interface modalities when a user is predicted not to understand a particular interaction between a user and an automated assistant. The automated assistant 204 can operate as part of an assistant application provided in one or more computing devices, such as the computing device 202 and / or a server device. A user can interact with the automated assistant 204 through an assistant interface 220, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. By way of example, a user can initialize the automated assistant 204 by providing verbal, textual, and / or graphical input to the assistant interface 220 to cause the automated assistant 204 to initialize one or more actions (e.g., provide data, control a peripheral device, access an agent, generate input and / or output, etc.). Alternatively, the automated assistant 204 may be initialized based on processing the context data 236 using one or more trained machine learning models. The contextual data 236 may characterize one or more characteristics of an environment in which the automated assistant 204 is accessible and / or one or more characteristics of a user who is expected to be interacting with the automated assistant 204. The computing device 202 may include a display device, which may be a display panel including a touch interface for receiving touch inputs and / or gestures that enable a user to control the applications 234 of the computing device 202 via the touch interface.In some implementations, the computing device 202 may lack a display device and therefore provide an audible user interface output without providing a graphical user interface output. Additionally, the computing device 202 may provide a user interface such as a microphone for receiving verbal natural language input from a user. In some implementations, the computing device 202 may include a touch interface, and the computing device 202 may not have a camera, although the computing device 202 may optionally include one or more other sensors.
[0017] The computing device 202 and / or other third party client devices can be in communication with a server device via a network such as the Internet. In addition, the computing device 202 and any other computing devices can be in communication with each other via a local area network (LAN) such as a Wi-Fi network. The computing device 202 can offload computational tasks to the server device to conserve computational resources at the computing device 202. As an example, the server device can host the automated assistant 204 and / or the computing device 202 can send inputs received at one or more assistant interfaces 220 to the server device. However, in some implementations, the automated assistant 204 can be hosted at the computing device 202 and various processes that can be associated with the operation of the automated assistant can be implemented at the computing device 202.
[0018] In various implementations, all or some aspects of the automated assistant 204 can be implemented on the computing device 202. In some of those implementations, aspects of the automated assistant 204 can be implemented by the computing device 202 and interface with a server device that can implement other aspects of the automated assistant 204. The server device can optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or some aspects of the automated assistant 204 are implemented by the computing device 202, the automated assistant 204 can be an application separate from (e.g., installed "on top of") the operating system of the computing device 202, or alternatively, can be implemented directly by (e.g., integral with but considered an operating system application of) the operating system of the computing device 202.
[0019] In some implementations, the automated assistant 204 can include an input processing engine 206 that can process input and / or output of the computing device 202 and / or the server device using a number of different modules. As an example, the input processing engine 206 can include a speech processing engine 208 that can process audio data received at the assistant interface 220 to identify text embodied in the audio data. To conserve computational resources at the computing device 202, the audio data can be transmitted from the computing device 202 to a server device, for example. Additionally or alternatively, the audio data can be processed solely at the computing device 202.
[0020] The process for converting the audio data to text can include a speech recognition algorithm, which can use neural networks and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data can be parsed by the data parsing engine 210 and made available to the automated assistant 204 as text data that can be used to generate and / or identify command phrases, intents, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data parsing engine 210 can be provided to the parameter engine 212 to determine whether the user has provided input that corresponds to a particular intent, action, and / or routine that can be performed by the automated assistant 204 and / or an application or agent that can be accessed via the automated assistant 204. For example, assistant data 238 can be stored at the server device and / or the computing device 202, and can include data defining one or more actions that can be performed by the automated assistant 204, as well as parameters necessary to perform the action. The parameter engine 212 can generate one or more parameters for the intent, action, and / or slot value and provide the one or more parameters to the output generation engine 214. The output generation engine 214 can use the one or more parameters to communicate with the assistant interface 220 to provide output to the user and / or to communicate with the one or more applications 234 to provide output to the one or more applications 234.
[0021] In some implementations, the automated assistant 204 can be an application that can be installed “on” the operating system of the computing device 202 and / or can itself form part (or the entirety) of the operating system of the computing device 202. The automated assistant application can include and / or access on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition can be implemented using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model stored locally in the computing device 202. On-device speech recognition generates recognized text for the spoken utterances (if any) present in the audio data. Also, for example, on-device natural language understanding (NLU) can be implemented using an on-device NLU module that processes the recognized text generated using on-device speech recognition and, optionally, contextual data to generate NLU data.
[0022] The NLU data can include an intent corresponding to the verbal utterance and, optionally, parameters for the intent (e.g., slot values). On-device fulfillment can be implemented using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and, optionally, other local data to determine actions to take to resolve the intent (and, optionally, parameters for the intent) of the verbal utterance. This can include determining local and / or remote responses (e.g., answers) to the verbal utterance, interactions with locally installed applications to be implemented based on the verbal utterance, commands to be sent (directly or via a corresponding remote system) to an Internet of Things (IoT) device based on the verbal utterance, and / or other resolution actions to be implemented based on the verbal utterance. The on-device fulfillment can then initiate local and / or remote implementation / execution of the determined actions to resolve the verbal utterance.
[0023] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment can be utilized at least selectively. For example, recognized text can be at least selectively sent to a remote automated assistant component for remote NLU and / or remote fulfillment. By way of example, recognized text can be optionally sent for remote fulfillment in parallel with on-device fulfillment or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution can be prioritized at least due to the reduced latency they provide when resolving spoken utterances (since no client-server round trips are required to resolve spoken utterances). Additionally, on-device functionality can be the only functionality available in situations without network connectivity or with limited network connectivity.
[0024] In some implementations, the computing device 202 may include one or more applications 234 that may be provided by a third party entity different than the entity that provided the computing device 202 and / or the automated assistant 204. The application state engine of the automated assistant 204 and / or the computing device 202 may access the application data 230 to determine one or more actions that may be performed by the one or more applications 234, as well as the state of each application of the one or more applications 234 and / or the state of each device associated with the computing device 202. The device state engine of the automated assistant 204 and / or the computing device 202 may access the device data 232 to determine one or more actions that may be performed by the computing device 202 and / or one or more devices associated with the computing device 202. Additionally, the application data 230 and / or any other data (e.g., device data 232) can be accessed by the automated assistant 204 to generate context data 236, which can characterize the context in which a particular application 234 and / or device is running and / or the context in which a particular user is accessing the computing device 202, the application 234, and / or any other device or module.
[0025] While one or more applications 234 are executing on the computing device 202, the device data 232 can characterize a current operational state of each application 234 executing on the computing device 202. Additionally, the application data 230 can characterize one or more features of the executing applications 234, such as the content of one or more graphical user interfaces being rendered at the direction of the one or more applications 234. Alternatively or in addition, the application data 230 can characterize action schemas that can be updated by the respective applications and / or by the automated assistant 204 based on the respective applications' current operational state. Alternatively or in addition, one or more action schemas for one or more applications 234 can remain fixed but can be accessed by the application state engine to determine appropriate actions to be initiated via the automated assistant 204.
[0026] The computing device 202 may further include an assistant invocation engine 222, which may use one or more trained machine learning models to process the application data 230, the device data 232, the context data 236, and / or any other data accessible to the computing device 202. The assistant invocation engine 222 may process this data to determine whether the data should be considered to indicate the user's intent to invoke an automated assistant instead of waiting for the user to explicitly speak an invocation phrase to invoke the automated assistant 204 or requesting the user to explicitly speak an invocation phrase. For example, the one or more trained machine learning models may be trained using training data instances based on a scenario in which a user is present in an environment in which multiple devices and / or applications exhibit various operating conditions. The training data instances may be generated to capture training data that characterizes contexts in which a user invokes an automated assistant and other contexts in which a user does not invoke an automated assistant.
[0027] When one or more trained machine learning models are trained according to these training data instances, the assistant invocation engine 222 can cause the automated assistant 204 to detect or limit the detection of a spoken invocation phrase from a user based on context and / or environmental features. Additionally or alternatively, the assistant invocation engine 222 can also cause the automated assistant 204 to detect or limit the detection of one or more assistant commands from a user based on context and / or environmental features. In some implementations, the assistant invocation engine 222 can be disabled or limited based on the computing device 202 detecting an assistant suppression output from another computing device. In this way, when the computing device 202 detects an assistant suppression output, the automated assistant 204 will not be invoked based on the context data 236 that would otherwise cause the automated assistant 204 to be invoked if the assistant suppression output had not been detected.
[0028] In some implementations, the automated assistant 204 can include an environmental feature engine 218 that can detect one or more features of an environment in which the computing device 202 and / or another computing device are operating. The environmental feature engine 218 can process data characterizing the one or more features to determine whether an interaction between the user and the automated assistant 204 will be influenced by the environment. This determination can be based on one or more heuristic processes and / or one or more trained machine learning models that can be trained based on prior cases in which one or more users interacted with the automated assistant in a similar environment. For example, data characterizing prior cases in which a user interacted with the automated assistant in a crowded environment can be used to identify a threshold for an interference score. The interference score can be generated for a particular environment using one or more trained machine learning models. When an interference score threshold is met by a particular interference score, the environmental feature engine 218 can communicate with the interference signature engine 226 of the automated assistant 204.
[0029] The interference sign engine 226 can be used by the automated assistant 204 to provide the user with an indication that an interaction between the user and the automated assistant 204 may be affected by one or more features of the environment in which they reside. In some implementations, the type of indication provided to the user can be based on one or more scores generated by the environmental features engine 218. As an example, when a score is generated indicating that audio interference will affect the audio interface, the interference sign engine 226 can cause a visual indication to be rendered for the user. In some implementations, the characteristics of the indication can also be selected by the interference sign engine 226. For example, the interference sign engine 226 can cause the size, shape, brightness, content, and / or other characteristics of the indication to be adjusted according to the degree of interference predicted to affect the interaction between the user and the automated assistant 204.
[0030] In some implementations, the automated assistant 204 can include an interface selection engine 228 that can select an interface to promote to the user when interference is predicted to affect another interface. For example, when it is determined that the computing device 202 is experiencing an amount of glare that makes typing in a touch interface difficult, the interface selection engine 228 can score the touch interface below other available interfaces. In some cases, the interface selection engine 228 can designate the touch interface as a less optimal interface in a particular environment than, for example, an audio interface. In some implementations, the interference indicator engine 226 can be informed of the rankings from the interface selection engine 228 and generate an indicator that identifies the optimal interface for a particular environment. As an example, the indicator provided by the interference indicator engine 226 can include a rendering of a keyboard to which the user can provide touch input to type input directly into the automated assistant.
[0031] FIG. 3 illustrates a method 300 for providing an indication of whether a particular input to an automated assistant will be subject to interference in an environment and for providing a separate interface for providing input to the automated assistant. The method 300 can be implemented by one or more computing devices, applications, and / or any other devices or modules that can be associated with an automated assistant. The method 300 can include an operation 302 of determining whether a user is available to provide input to the automated assistant. In some implementations, the determination of whether a user is available can be based on data from one or more sensors and / or any other interfaces of one or more computing devices. For example, the automated assistant can determine that one or more users are within a threshold distance of the automated assistant interface based on audio data and / or image data. When the automated assistant determines that the user is available to provide input to the automated assistant, the method 300 can move to operation 304. Otherwise, the automated assistant can continue to determine whether the user is available to provide input to the automated assistant.
[0032] Operation 304 may include determining whether one or more characteristics of the environment are predicted to affect the input from the user. In some implementations, the one or more characteristics of the environment may include audio characteristics of the environment. The audio characteristics may include the number of people speaking, the source of a particular sound, the volume and / or frequency of a particular sound, ambient sounds, overall volume level, the distance from the source of the sound to a particular interface, and / or any other audio characteristics that may interfere with the input to the computing device. In some implementations, the one or more characteristics of the environment may include objects, people, location, available power, weather, movement, lighting, distance between particular objects, layout of the area, temperature, and / or any other characteristics of the environment that may affect the interaction between the user and the automated assistant. In some implementations, the one or more characteristics of the environment may be determined and used to generate a score. The automated assistant may determine whether the score meets one or more different thresholds. Alternatively or additionally, one or more different scores may be generated for each interface of the computing device present in the environment. When the score for a particular interface does not meet a threshold for that particular interface, the automated assistant can provide an indication that the particular interface may be subject to interference during an interaction between the user and the automated assistant. For example, a score for an audio interface may fail to meet a threshold for the audio interface, and as a result, the automated assistant can cause a keyboard interface to be rendered at the computing device.
[0033] When the automated assistant determines that one or more characteristics of the environment are predicted to affect the interaction between the user and the automated assistant, the method 300 can move from operation 304 to operation 306. Otherwise, the automated assistant can continue to determine whether the user has provided input to the particular interface or another interface to facilitate interaction with the automated assistant. Operation 306 can include having the automated assistant provide an indication that an input provided or being provided through the particular interface may be affected by the environment. In some implementations, the indication can be provided through an audio interface, a graphical interface, a haptic interface, a wireless interface, and / or any other interface that can be used to provide an indication to the user. In some implementations, when the computing device includes a touch display panel, the indication can be rendered in a keyboard interface on the touch display panel. In this manner, when the user is about to provide a verbal utterance that is determined to be affected by interference in the environment, the user can recognize the keyboard interface and select to provide a touch input to the touch display panel instead of the verbal utterance.
[0034] From operation 306, the method 300 may move to optional operation 308, which may include receiving, by the automated assistant, a separate input from the user via a separate interface (i.e., a separate interface separate from the particular interface determined to be affected by interference in the environment). For example, the user may type a shorter input, such as "good morning" into a keyboard interface as an input to the automated assistant, instead of the user providing a verbal utterance, such as "assistant, good morning," to initiate the performance of the "good morning" routine. From operation 308, the method 300 may move to optional operation 310, which may include having the automated assistant initiate the performance of one or more actions based on the separate input. For example, the separate input may include natural language content typed by the user using the keyboard interface rendered in operation 306. The natural language content may be "good morning," which may cause the automated assistant to perform one or more actions to facilitate the completion of the "good morning" routine (e.g., reading out the calendar entries for the day, turning on the house lights, playing relaxing music). The automated assistant can reduce the number of misinterpreted inputs from a user by providing an indication as to whether a verbal or other type of input is likely to be subject to interference, thereby preserving computational resources that would otherwise be consumed performing erroneous actions and / or processing duplicate inputs.
[0035] 4 is a block diagram 400 of an exemplary computer system 410. The computer system 410 typically includes at least one processor 414 that communicates with several peripheral devices via a bus subsystem 412. These peripheral devices may include, for example, a storage subsystem 424 including a memory 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices enable user interaction with the computer system 410. The network interface subsystem 416 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.
[0036] The user interface input devices 422 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways for inputting information into the computer system 410 or onto a communications network.
[0037] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and ways to output information from the computer system 410 to a user or to another machine or computer system.
[0038] Storage subsystem 424 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 424 may include logic for performing selected aspects of method 300 and / or for implementing one or more of system 200, computing device 106, and / or any other applications, devices, apparatus, and / or modules discussed herein.
[0039] These software modules are generally executed by the processor 414 alone or in combination with other processors. The memory 425 used within the storage subsystem 424 may include several memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution, and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 may provide persistent storage for program files and data files, and may include hard disk drives, floppy disk drives and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of a particular implementation may be stored within the storage subsystem 424 or within other machines accessible by the processor 414 via the file storage subsystem 426.
[0040] Bus subsystem 412 provides a mechanism for allowing the various components and subsystems of computer system 410 to communicate with each other as intended. Although bus subsystem 412 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.
[0041] The computer system 410 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 410 depicted in Figure 4 is intended only as a specific example intended to illustrate some implementations. Many other configurations of the computer system 410 are possible, having more or fewer components than the computer system depicted in Figure 4.
[0042] In situations where the systems described herein may collect or use personal information about users (or, as is often referred to herein, "participants"), the user may be given the opportunity to control whether a program or function collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how to receive content from a content server that may be more relevant to the user. Also, certain data may be handled in one or more ways such that personally identifiable information is removed before it is stored or used. For example, a user's identification information may be handled such that personally identifiable information cannot be identified for that user, or if geographic location information is obtained, the user's geographic location may be generalized (such as to the city level, zip code level, or state level) such that the user's specific geographic location cannot be identified. Thus, the user may be able to control how information is collected and / or used about the user.
[0043] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining the results and / or one or more of the advantages described herein can be utilized, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, any parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the teachings are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. Thus, it should be understood that the implementations described above are presented by way of example only, and that within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.
[0044] In some implementations, a method implemented by one or more processors is described as including operations such as processing first data characterizing audio characteristics of an environment in which a computing device resides, the computing device providing access to an automated assistant through one or more interfaces of the computing device. The method may further include an operation of determining whether the audio characteristics correspond to interference predicted to affect an interaction between a user and the automated assistant based on the first data, the interaction being through an audio interface of the one or more interfaces. The method may further include an operation of processing second data indicating that a user is providing or is predicted to provide a verbal utterance to the automated assistant to facilitate the interaction. The method may further include an operation of causing the computing device or another computing device to render a keyboard interface based on the first data and the second data when the interference is determined to affect the interaction, receiving a user input at the keyboard interface to facilitate the interaction between the user and the automated assistant, and causing the automated assistant to initialize the implementation of one or more actions in response to receiving the user input at the keyboard interface.
[0045] In some implementations, the method may further include an operation of receiving a separate user input via an audio interface to facilitate an interaction between the user and the automated assistant when it is determined that the interference does not affect the interaction. In some implementations, determining whether the audio characteristics correspond to interference predicted to affect the interaction between the user and the automated assistant includes identifying a number of people located within a threshold distance of one or more interfaces. In some implementations, determining whether the audio characteristics correspond to interference predicted to affect the interaction between the user and the automated assistant includes determining a score indicating whether the automated assistant is predicted to correctly interpret a verbal utterance that the user is providing or predicted to provide to facilitate the interaction, and determining that the interference is predicted to affect the interaction when the score meets a threshold.
[0046] In some implementations, determining whether the audio characteristics correspond to interference predicted to affect the interaction between the user and the automated assistant includes determining an additional score characterizing the ease of accessing a keyboard interface in an environment in which the computing device is present, and determining that rendering of the keyboard interface is further based on the additional score. In some implementations, causing the computing device to render the keyboard interface is performed simultaneously with the user providing a verbal utterance to the automated assistant to facilitate the interaction. In some implementations, causing the computing device to render the keyboard interface is performed before the user provides a verbal utterance to the automated assistant to facilitate the interaction.
[0047] In another implementation, a method implemented by one or more processors is described as including an operation such as processing first data characterizing one or more features of an environment in which a computing device resides, where the computing device provides access to an automated assistant via one or more interfaces of the computing device. The method may further include an operation of determining, based on the first data, whether the one or more features correspond to interference predicted to affect an interaction between a user and the automated assistant, where the interaction occurs via one or more interfaces of the computing device. The method may further include an operation of processing second data indicating that a user is providing, or is predicted to provide, input to the automated assistant to facilitate the interaction. The method may further include the operations of, when it is determined that the interference affects the interaction, causing the computing device or another computing device to render an indication indicating that the interaction between the user and the automated assistant will be affected by the interference based on the first data and the second data, receiving user input at one or more other interfaces of the computing device or other computing device to facilitate the interaction between the user and the automated assistant, and in response to receiving the user input at the one or more other interfaces, causing the automated assistant to initialize performance of one or more actions.
[0048] In some implementations, the method may further include an operation of receiving a separate user input at one or more interfaces of the computing device to facilitate the interaction between the user and the automated assistant when it is determined that the interference does not affect the interaction. In some implementations, the one or more interfaces include an audio interface, and the indicator is rendered at a graphical user interface of the computing device or another computing device. In some implementations, determining whether the one or more features correspond to interference predicted to affect the interaction between the user and the automated assistant includes identifying a number of people located within a threshold distance of the one or more interfaces. In some implementations, determining whether the one or more features correspond to interference predicted to affect the interaction between the user and the automated assistant includes identifying a number of people speaking within a threshold distance of the one or more interfaces.
[0049] In some implementations, determining whether the one or more features correspond to interference predicted to affect the interaction between the user and the automated assistant includes determining a score indicating whether the automated assistant is predicted to correctly interpret an input that the user is providing or is predicted to provide to facilitate the interaction, where when the score meets a threshold, the interference is predicted to affect the interaction. In some implementations, determining whether the one or more features correspond to interference predicted to affect the interaction between the user and the automated assistant includes determining an additional score that characterizes the ease of accessing one or more other interfaces in an environment in which the computing device is present, where the indicator is further based on the additional score. In some implementations, causing the computing device to render the indicator is performed simultaneously with the user providing an input to the automated assistant to facilitate the interaction. In some implementations, causing the computing device to render the indicator is performed before the user provides an input to the automated assistant to facilitate the interaction. In some implementations, the one or more interfaces include a keyboard interface, and the indicator includes a keyboard interface that is rendered in a graphical user interface of the computing device. In some implementations, the indicator further includes a text field, the text field including suggested content, the suggested content being based on the content embodied in the input and affected by the interference.
[0050] In yet another implementation, a method implemented by one or more processors is described as including an operation such as processing context data characterizing audio characteristics of an environment in which a computing device is present, the computing device providing access to an automated assistant through an audio interface of the computing device. The method may further include an operation of determining whether the audio characteristics correspond to interference predicted to affect an interaction between a user and an automated assistant, the interaction being through an audio interface of the one or more interfaces, based on the processing of the context data. When it is determined that the interference affects the interaction, the method may further include an operation of causing the computing device or another computing device to render a keyboard interface for receiving user input to facilitate the interaction between the user and the automated assistant, based on the context data. In some implementations, processing the context data characterizing audio characteristics of the environment in which the computing device is present includes processing historical interaction data characterizing one or more prior instances in which a user provided a verbal input to an automated assistant and the automated assistant did not fulfill one or more requests embodied in the verbal input. [Explanation of symbols]
[0051] 100 Views 102 users 104 Display Interface 106 Computing Devices 108 Audio Interface 110 Environment 112 Input Field 120 Views 122 Hands 124 Keyboard 200 Systems 202 Computing Devices 204 Automated Assistants 206 Input Processing Engine 208 Audio Processing Engine 210 Data Parsing Engine 212 Parameter Engine 214 Output Generation Engine 218 Environmental Feature Engine 220 Assistant Interface 222 Assistant Call Engine 226 Interference Beacon Engine 228 Interface Selection Engine 230 Application Data 232 Device Data 234 Applications 236 Context Data 238 Assistant Data 300 ways 400 Block Diagram 410 Computer Systems 412 Bus Subsystem 414 processor 416 Network Interface Subsystem 420 User Interface Output Device 422 User Interface Input Devices 424 Storage Subsystem 425 Memory 426 File Storage Subsystem 430 Main Random Access Memory (RAM) 432 Read-Only Memory (ROM)
Claims
1. 1. A method implemented by one or more processors, comprising: Processing first data characterizing an audio characteristic of an environment in which the computing device resides, comprising: the computing device providing access to an automated assistant via one or more interfaces of the computing device; Steps and Determining whether the audio characteristics correspond to interference predicted to affect an interaction between a user and the automated assistant based on the first data, the interaction occurs through an audio interface of the one or more interfaces; Steps and Processing second data indicating that the user is providing, or is expected to provide, a verbal utterance to the automated assistant to facilitate the interaction; when it is determined that the interference affects the interaction, causing the computing device or another computing device to render a keyboard interface based on the first data and the second data; receiving user input at the keyboard interface to facilitate the interaction between the user and the automated assistant; In response to receiving the user input at the keyboard interface, causing the automated assistant to initiate performance of one or more actions; A method comprising:
2. when it is determined that the interference does not affect the interaction, receiving a separate user input via the audio interface to facilitate the interaction between the user and the automated assistant; The method of claim 1, further comprising:
3. Determining whether the audio characteristics correspond to the interference predicted to affect the interaction between the user and the automated assistant, determining a number of people located within a threshold distance of the one or more interfaces.
3. The method of claim 1 or claim 2, comprising:
4. Determining whether the audio characteristics correspond to the interference predicted to affect the interaction between the user and the automated assistant, determining a score indicative of whether the automated assistant is predicted to correctly interpret the verbal utterance that the user is providing or is predicted to provide to facilitate the interaction, When the score meets a threshold, the interference is predicted to affect the interaction.
3. The method of claim 1 or claim 2, comprising the steps of:
5. Determining whether the audio characteristics correspond to the interference predicted to affect the interaction between the user and the automated assistant, determining an additional score characterizing the ease of accessing the keyboard interface within the environment in which the computing device resides, and wherein the step of rendering the keyboard interface is further based on the additional score. The method of claim 4, comprising the steps of:
6. 6. The method of claim 1, wherein the step of causing the computing device to render the keyboard interface is performed simultaneously with the user providing the spoken utterance to the automated assistant to facilitate the interaction.
7. 6. The method of claim 1, wherein the step of causing the computing device to render the keyboard interface is performed before the user provides the spoken utterance to the automated assistant to facilitate the interaction.
8. 1. A method implemented by one or more processors, comprising: Processing first data characterizing one or more features of an environment in which the computing device resides, comprising: the computing device providing access to an automated assistant via one or more interfaces of the computing device; Steps and Determining whether the one or more features correspond to an interference predicted to affect an interaction between a user and the automated assistant based on the first data, the interaction occurs through the one or more interfaces of the computing device; Steps and Processing second data indicating that the user is providing, or is expected to provide, input to the automated assistant to facilitate the interaction; when it is determined that the interference affects the interaction, Based on the first data and the second data, causing the computing device or another computing device to render an indication indicating that the interaction between the user and the automated assistant will be affected by the interference; receiving user input at one or more other interfaces of the computing device or the other computing device to facilitate the interaction between the user and the automated assistant; In response to receiving the user input at the one or more other interfaces, causing the automated assistant to initiate performance of one or more actions; A method comprising:
9. when it is determined that the interference does not affect the interaction, receiving a separate user input at the one or more interfaces of the computing device to facilitate the interaction between the user and the automated assistant; 9. The method of claim 8, further comprising:
10. the one or more interfaces include an audio interface; the indicia is rendered in a graphical user interface of the computing device or the other computing device; The method according to claim 8 or claim 9.
11. Determining whether the one or more features correspond to the interference predicted to affect the interaction between the user and the automated assistant, determining a number of people located within a threshold distance of the one or more interfaces.
11. The method according to any one of claims 8 to 10, comprising:
12. Determining whether the one or more features correspond to the interference predicted to affect the interaction between the user and the automated assistant, determining a number of people speaking within a threshold distance of said one or more interfaces; 11. The method according to any one of claims 8 to 10, comprising:
13. Determining whether the one or more features correspond to the interference predicted to affect the interaction between the user and the automated assistant, determining a score indicative of whether the automated assistant is predicted to correctly interpret the input that the user is providing or predicted to provide to facilitate the interaction, When the score meets a threshold, the interference is predicted to affect the interaction.
11. The method according to any one of claims 8 to 10, comprising the steps:
14. Determining whether the one or more features correspond to the interference predicted to affect the interaction between the user and the automated assistant, determining an additional score characterizing the ease of accessing the one or more other interfaces within the environment in which the computing device resides; the indication is further based on the additional score. The method of claim 13, comprising the steps of:
15. 15. The method of claim 8, wherein the step of causing the computing device to render the indication is performed simultaneously with the user providing the input to the automated assistant to facilitate the interaction.
16. 11. The method of claim 8, wherein the step of causing the computing device to render the indicator is performed before the user provides the input to the automated assistant to facilitate the interaction.
17. the one or more interfaces include a keyboard interface; the indicia including the keyboard interface rendered in a graphical user interface of the computing device; 17. The method according to any one of claims 8 to 16.
18. 20. The method of claim 17, wherein the indicator further comprises a text field, the text field comprising suggested content, the suggested content being based on content embodied in the input and affected by the interference.
19. 1. A method implemented by one or more processors, comprising: processing context data characterizing audio characteristics of an environment in which the computing device resides; the computing device providing access to an automated assistant via an audio interface of the computing device; Steps and Based on processing of the context data, determining whether the audio characteristics correspond to interference predicted to affect an interaction between a user and the automated assistant, the interaction is via an audio interface of one or more interfaces; Steps and when it is determined that the interference affects the interaction, Based on the context data, causing the computing device or another computing device to render a keyboard interface for receiving user input to facilitate the interaction between the user and the automated assistant; A method comprising:
20. processing the contextual data characterizing an audio characteristic of the environment in which the computing device resides, processing historical interaction data characterizing one or more prior instances in which the user provided verbal input to the automated assistant and the automated assistant failed to fulfill one or more requests embodied in the verbal input; 20. The method of claim 19, comprising:
21. At least one processor; a memory storing instructions that, when executed, cause the at least one processor to perform the operations of any one of claims 1 to 20; A system comprising:
22. 21. A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor to perform operations corresponding to any one of claims 1 to 20.
Citation Information
Patent Citations
Method and device for data input
JP2004054080A
Speech recognition device for vehicle
JP2005352397A
Character input device, character input method and character input program
JP2012063921A
Automated adaptation of user interfaces for hands-free interaction
JP2015501022A
Speech-to-text device, speech-to-text method, and speech-to-text program
JP6735392B1
