Contextual assistance using mouse pointing or touch cues

Abstract

The method (400) includes receiving audio data (202) corresponding to a query (104) spoken by a user, accepting a user input indication in a graphical user interface (300) displayed on the screen indicating a spatial input (212) applied at a first location (114) on the screen, and processing the audio data to determine a transcription (214) of the query. The method also includes performing a query interpretation on the transcription to determine that the query references an object (116) displayed on the screen without uniquely identifying the object and requests information about the object. The method further includes disambiguating the query to uniquely identify the object to which the query refers using the user input indication indicating the spatial input applied at the first location on the screen, obtaining information about the object requested by the query, and providing a response (252) to the query.

Description

[Technical field]

[0001] The present disclosure relates to a contextual assistant that uses mouse pointing or touch cues. [Background technology]

[0002] In a voice-enabled environment, a user can speak a query aloud and the digital assistant performs an action to obtain an answer to the query. Digital assistants are particularly effective at providing precise answers to queries of general topics, and the query itself generates the information the digital assistant needs to obtain an answer to the query. However, when a query is ambiguous, the digital assistant needs additional context before it can obtain an answer to the query. In some cases, identifying the user's interests at the time the user spoke the query aloud provides the additional context needed to obtain an answer to the query. Thus, a digital assistant receiving a query must have some way of identifying the additional context of the user who spoke the query. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method that, when executed by data processing hardware, causes the data processing hardware to perform operations, the operations including receiving audio data corresponding to a query spoken by a user and captured by an assistant-enabled device associated with the user. The operations also include accepting a user input indication in a graphical user interface (GUI) displayed on a screen in communication with the data processing hardware, indicating a spatial input applied at a first position on the screen, and processing the audio data to determine a transcription of the query using a speech recognition model. The operations also include performing a query interpretation on the transcription of the query to determine that the query refers to an object displayed on the screen without uniquely identifying the object and requests information about the object displayed on the screen. The operations also include disambiguating the query to uniquely identify an object to which the query refers using the user input indication indicating the spatial input applied at the first position on the screen, and retrieving information about the object requested by the query in response to uniquely identifying the object. The operations also include providing a response to the query including the retrieved information about the object.

[0004] The embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the operation also includes detecting a trigger event, and in response to detecting the trigger event, activating a GUI displayed on the screen to enable detection of spatial input and activating a speech recognition model to enable performance of speech recognition on the incoming audio data captured by the assistant-enabled device. In these embodiments, detecting the trigger event includes detecting the presence of a hot word in the received audio data by a hot word detector. Alternatively, detecting the trigger event may include one of: accepting a user input indication indicating a selection of a graphic element in a GUI displayed on the screen, accepting a user input indication indicating a selection of a physical button disposed on the assistant-enabled device, detecting a predefined gesture made by a user, or detecting a predefined movement / pose of the assistant-enabled device.

[0005] In some examples, accepting a user input indication indicative of the spatial input to be applied at the first location includes one of: detecting a cursor position displayed in the GUI at the first location when the user speaks the query, detecting a touch input accepted in the GUI at the first location when the user speaks the query, or detecting a circling selection action performed in the GUI at the first location when the user speaks the query. In these examples, disambiguating the query to uniquely identify an object includes receiving image data including a plurality of candidate objects displayed in the GUI and corresponding locations of the plurality of candidate objects displayed in the GUI, and identifying a candidate object from the plurality of candidate objects having a corresponding location closest to the first location as an object to which the query refers.

[0006] In an additional example, accepting a user input representation indicative of the spatial input applied at the first location includes accepting a user input representation indicative of the spatial input applied at the first location, and disambiguating the query to uniquely identify the object includes uniquely identifying a string underlined by the underline action as the object to which the query refers. In another example, accepting a user input representation indicative of the spatial input applied at the first location includes detecting a highlight action taken in the GUI that highlights a string displayed in the GUI at the first location, and disambiguating the query to uniquely identify the object includes uniquely identifying a string highlighted by the highlight action as the object to which the query refers.

[0007] In some implementations, obtaining information about the object requested by the query includes querying a search engine using the uniquely identified object and one or more terms in the transcription of the query to obtain a list of results responsive to the query, and displaying the list of results responsive to the query in a GUI displayed on the screen, where displaying the list of results responsive to the query may further include generating a graphical element representing a highest ranked result in the list of results responsive to the query, and displaying the list of results responsive to the query in the GUI displayed on the screen at a first location on the screen. Optionally, the operations may further include determining that the uniquely identified object includes text in a first language, such that obtaining information about the object requested by the query includes obtaining a translation of the text in a second language different from the first language.

[0008] Another aspect of the disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving audio data corresponding to a query spoken by a user and captured by an assistant-enabled device associated with the user. The operations also include accepting a user input indication in a graphical user interface (GUI) displayed on a screen in communication with the data processing hardware, the user input indication indicating a spatial input applied at a first position on the screen, and processing the audio data to determine a transcription of the query using a speech recognition model. The operations also include performing a query interpretation on the transcription of the query to determine that the query refers to an object displayed on the screen without uniquely identifying the object and requests information about the object displayed on the screen. The operations also include disambiguating the query to uniquely identify an object to which the query refers using the user input indication indicating the spatial input applied at the first position on the screen, and retrieving information about the object requested by the query in response to uniquely identifying the object. The operations also include providing a response to the query including the retrieved information about the object.

[0009] This aspect may include one or more of the following optional features. In some implementations, the operation also includes detecting a trigger event, and in response to detecting the trigger event, activating a GUI displayed on the screen to enable detection of spatial input and activating a speech recognition model to enable execution of speech recognition on the incoming audio data captured by the assistant-enabled device. In these implementations, detecting the trigger event includes detecting the presence of a hot word in the received audio data by a hot word detector. Alternatively, detecting the trigger event may include one of: accepting a user input indication indicating a selection of a graphic element in a GUI displayed on the screen, accepting a user input indication indicating a selection of a physical button disposed on the assistant-enabled device, detecting a predefined gesture made by a user, or detecting a predefined movement / pose of the assistant-enabled device.

[0010] In some examples, accepting a user input indication indicative of the spatial input to be applied at the first location includes one of: detecting a cursor position displayed in the GUI at the first location when the user speaks the query, detecting a touch input accepted in the GUI at the first location when the user speaks the query, or detecting a circling selection action performed in the GUI at the first location when the user speaks the query. In these examples, disambiguating the query to uniquely identify an object includes receiving image data including a plurality of candidate objects displayed in the GUI and corresponding locations of the plurality of candidate objects displayed in the GUI, and identifying a candidate object from the plurality of candidate objects having a corresponding location closest to the first location as an object to which the query refers.

[0011] In an additional example, accepting a user input representation indicative of the spatial input applied at the first location includes accepting a user input representation indicative of the spatial input applied at the first location, and disambiguating the query to uniquely identify the object includes uniquely identifying a string underlined by the underline action as the object to which the query refers. In another example, accepting a user input representation indicative of the spatial input applied at the first location includes detecting a highlight action taken in the GUI that highlights a string displayed in the GUI at the first location, and disambiguating the query to uniquely identify the object includes uniquely identifying a string highlighted by the highlight action as the object to which the query refers.

[0012] In some implementations, obtaining information about the object requested by the query includes querying a search engine using the uniquely identified object and one or more terms in the transcription of the query to obtain a list of results responsive to the query, and displaying the list of results responsive to the query in a GUI displayed on the screen, where displaying the list of results responsive to the query may further include generating a graphical element representing a highest ranked result in the list of results responsive to the query, and displaying the list of results responsive to the query in the GUI displayed on the screen at a first location on the screen. Optionally, the operations may further include determining that the uniquely identified object includes text in a first language, such that obtaining information about the object requested by the query includes obtaining a translation of the text in a second language different from the first language.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of an example system including a context assistant using mouse pointing or touch cues. [Diagram 2] FIG. 2 is a schematic diagram of example components of a context assistant. [Figure 3A] 1 is an exemplary graphical user interface (GUI) rendered on a screen of a user device that includes a context assistant. [Figure 3B] 1 is an exemplary graphical user interface (GUI) rendered on a screen of a user device that includes a context assistant. [Figure 3C] 1 is an exemplary graphical user interface (GUI) rendered on a screen of a user device that includes a context assistant. [Figure 4] 4 is a flowchart of an exemplary sequence of operations for a method of query disambiguation using mouse pointing or touch cues. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] The manner in which a user interacts with an assistant-enabled device is designed to be primarily, but not exclusively, through voice input. While assistant-enabled devices are effective at obtaining answers to general topic queries (e.g., what is the capital of Michigan?), context-driven queries require the assistant-enabled device to obtain additional information to obtain an accurate answer. For example, an assistant-enabled device may struggle to obtain a confident / accurate answer to a query such as "show me more of this" without additional context.

[0017] In scenarios where more context is needed beyond the voice query to answer a query, it is beneficial for the assistant-enabled device to capture image data from the screen of the assistant-enabled device. For example, a user may naturally query the assistant-enabled device by speaking "show me more windows like that." Here, the voice query identifies that the user is looking for windows similar to an object, but is ambiguous because the object is unknown from the linguistic content of the query. Using image data from the screen of the assistant-enabled device allows the assistant-enabled device to narrow down the possible windows to search for from the entire screen showing the city to a separate sub-region containing a specific building in the city where user input applied at a specific location on the screen is detected in conjunction with the voice query. By capturing the input data and image data along with the query, the assistant-enabled device can generate a response to a query about buildings in a city, despite the user having to explicitly identify the building in the voice query.

[0018] 1 is an example of a system 100 including a user device 10 and / or a remote system 60 that communicates with the user device 10 via a network 40. The user device 10 and / or the remote system 60 execute a pointing assistant 200 with which a user 102 can interact through voice and spatial input, such that the pointing assistant 200 can generate responses to queries that refer to objects displayed on the screen of the user device 10, even though the queries cannot uniquely identify the objects for which information is sought. In the illustrated example, the user device 10 corresponds to a smartphone, but the user device 10 may include other computing devices having or communicating with a display screen, such as, but not limited to, a tablet, a smart display, a desktop / laptop, a smart watch, a smart appliance, a smart glasses / headset, or a vehicle infotainment device. The user device 10 includes data processing hardware 12 and memory hardware 14 that stores instructions that, when executed on the data processing hardware 12, cause the data processing hardware 12 to perform operations. The remote system 60 (e.g., a server, a cloud computing environment) also includes data processing hardware 62 and memory hardware 64 that stores instructions that, when executed on the data processing hardware 62, cause the data processing hardware 62 to perform operations. As described in more detail below, the point assistant 200 executing on the user device 10 and / or the remote system 60 includes a speech recognizer 210 and a response generator 250, and can utilize one or more information sources 240 stored in the memory hardware 14, 64. In some examples, execution of the point assistant 200 is shared across the user device 10 and the remote system 60.

[0019] The user device 10 includes an array of one or more microphones 16 configured to capture audio, such as voice, directed at the user device 10. The user device 10 also executes a graphical user interface (GUI) 300 configured to capture user input indications via any one of touch, gesture, gaze, and / or input devices (e.g., mouse, trackpad, or stylus) for controlling functions of the user device 10 for display on a screen 18 in communication with the data processing hardware 12. The GUI 300 may be an interface associated with an application 50 running on the user device 10 that presents a number of objects within the GUI 300. The user device 10 may further include or communicate with an audio output device (e.g., speaker) 19 capable of outputting audio, such as music and / or synthetic voice, from the pointing assistant 200. The user device 10 may also include a physical button 17 disposed on the user device 10 and configured to receive a tactile selection by the user 102 to invoke the pointing assistant 200.

[0020] The user device 10 may include an audio subsystem 106 for extracting audio data 202 (FIG. 2) from the query 104. For example, referring to FIG. 1, the audio subsystem 106 may receive streaming audio captured by one or more microphones 16 of the user device 10 corresponding to an utterance 106 of the query 104 spoken by the user 102 to extract audio data (e.g., acoustic frames) 202. The audio data 202 may include acoustic features such as Mel-Frequency Cepstral Coefficients (MFCCs) or filter bank energies calculated over a window of the audio signal. In the illustrated example, the query 104 spoken by the user 102 includes "Hey Google, what's this?"

[0021] The user device 10 may execute (i.e., in the data processing hardware 12) a hotword detector 20 configured to detect the presence of a hotword 105 in the streaming audio without performing semantic analysis or speech recognition processing on the streaming audio. The hotword detector 20 may execute in the audio subsystem 106. The hotword detector 202 may receive audio data 202 and determine whether an utterance 106 includes a particular hotword 105 (e.g., hey, Google) spoken by the user 102. That is, the hotword detector 20 may be trained to detect the presence of a hotword 105 (e.g., hey, Google) or one or more other variants of that hotword (e.g., okay, Google) in the audio data 202. Detecting the presence of the hot word 105 in the audio data 202 may correspond to a triggering event that invokes the pointing assistant 200 to activate the GUI 300 displayed on the screen 18 to enable detection of the spatial input 112, and to activate the speech recognizer 210 to perform speech recognition on the audio data 202 corresponding to the utterance 106 of the hot word 105 and / or one or more other terms characterizing the query 104 following the hot word. In some examples, the hot word 105 is spoken in the utterance 106 following the query 105, and a portion of the audio data 202 characterizing such query 104 is buffered and retrieved by the speech recognizer 210, which retrieves the portion of the audio data 202 upon detection of the hot word 105 in the audio data 202. In some implementations, the triggering event includes receiving a user input indication indicating a selection of a graphical element 21 (e.g., a graphical microphone) in the GUI 300. In other implementations, the triggering event includes receiving a user input indication indicating a selection of a physical button 17 located on the user device 10.In other implementations, the trigger event includes detecting a predefined gesture made by the user 102 (e.g., via image and / or radar sensors) or detecting a predefined movement / pose of the user device 10 (e.g., using one or more sensors such as an accelerometer and / or gyroscope).

[0022] The user device 10 may further include an image subsystem 108 configured to extract a location 114 (e.g., XY coordinate location) on the screen 18 of the spatial input 112 applied in the GUI 300. For example, the user 102 may provide a user input representation 110 indicative of the spatial input 112 in the GUI 300 at the location 114 on the screen. The image subsystem 108 may additionally extract image data (e.g., pixels) 204 corresponding to one or more objects 116 currently displayed on the screen 18. In the illustrated example, the GUI 300 accepts the user input representation 110 indicative of the spatial input 112 applied at a first location 114 on the screen 18, and the image data 202 includes the object (i.e., a golden retriever) 116 displayed on the screen 18 proximate to the first location 114.

[0023] Continuing with reference to the system 100 of FIG. 1 and the point assistant 200 of FIG. 2, the speech recognizer 210 executes an automatic speech recognition (ASR) model (e.g., a speech recognition model) 212 that receives the audio data 202 as an input and generates / predicts a corresponding transcription 214 of the query 104 as an output. In the illustrated example, the query 104 includes the phrase "What's this?", which requests information 246 about an object 116 displayed in an on-screen GUI 300 without uniquely identifying the object 116. The point assistant 200, which will be described in more detail below, uses spatial input 112 applied at a first location 114 on the screen 118 to disambiguate the query 104 in order to uniquely identify the object 116 to which the query 104 refers. Once the object 116 is uniquely identified, the point assistant 200 can obtain information 246 about the object and generate a response 252 to the query 104 that includes the obtained information 246 about the object 116. The response generator 250 may generate a response 252 to the query 104 as a textual representation. Here, the point assistant 200 instructs the user device 10 to display the response 252 in the GUI 300 for the user 102 to read. In the illustrated example, the point assistant 200 generates a textual representation of the response 252 for display in the GUI 300: "It's a Golden Retriever." As described in more detail below, the point assistant 200 may require additional context extracted by the image subsystem 108 (i.e., the user 102 applied the spatial input 112 at the first position 114 corresponding to the object 116) to obtain information 246 for inclusion in the response 252 so as to uniquely identify the object 116 to which the query 104 refers. In some examples, the response generator 250 uses a text-to-speech (TTS) system 260 to convert the textual representation of the response 252 into synthetic speech.In these examples, in addition to or instead of displaying a textual representation of the response 252 in the GUI 300, the point assistant 200 generates a synthesized voice to output audibly through the speaker 19 of the user device 10.

[0024] 2, the point assistant 200 further includes a natural language understanding (NLU) module 220 configured to perform query interpretation on the corresponding transcription 214 to ultimately determine the meaning behind the transcription 214. The NLU module 220 can also receive context information 201 to aid in interpreting the transcription 214. The context information 201 can indicate an application 50 (FIG. 1) currently running on the user device 10, a previous query 104 from the user 102, that a particular hot word 105 was detected, or any other information that the NLU module 220 can leverage to interpret the query 104. Continuing with the example, the context information 201 can indicate that the user is interacting with a web-based application 50 running on the user device 102, and the NLU module 230 performs query interpretation to determine that the query 104 specifies an action 232 to obtain a description / information about some object 116 displayed in the GUI 300 that the user 102 is likely viewing. However, the NLU module 230 determines that the query 104 is ambiguous because the object 116 is not explicitly identified in the transcription 214 apart from the term “this.” In other words, the query interpretation performed by the NLU module 230 determines that the query 104 references an object 116 displayed on the screen 18 without uniquely identifying the object 116, and specifies an action 232 that requests information 246 about the object 116.

[0025] To satisfy the query 104, the NLU module 220 must disambiguate the query 104 to uniquely identify the object 116 to which the query 104 refers. For example, in a scenario in which the query 104 includes a corresponding transcription 214 "show me a similar bike" while multiple bikes are currently displayed on the screen 18, the NLU module 220 can perform a query interpretation on the corresponding transcription 214 to determine that the user 102 is referring to the object (i.e., bike) 116 displayed in the GUI 300 without uniquely identifying the object 116 and without requesting information 246 about the object 116 (i.e., other objects similar to the bike 116). In this example, the NLU module 220 determines that the query 104 specifies an action 232 to retrieve an image of a bike that resembles one of the bikes displayed on the screen, but cannot satisfy the query 104 because the bike to which the query refers cannot be ascertained from the transcription 214.

[0026] The NLU module 220 can use the user input display showing the spatial input 112 applied at the first position 114 on the screen as additional context to disambiguate the query 104 to uniquely identify the object 116 to which the query refers. The NLU module 220 can additionally use image data 204 to disambiguate the query 104. Here, the image data 204 can include multiple candidate objects displayed in the GUI and corresponding positions of the multiple candidate objects displayed in the GUI. The image data 204 can be extracted by the image subsystem 108 from graphical content rendered and displayed in the GUI 300. The image data 204 can include labels identifying the candidate objects. In some examples, the image subsystem 108 performs one or more object recognition techniques on the graphical content to identify the candidate objects. By using the image data 204 and the received user input representation that is the spatial input 118 applied at the first location 114, the NLU module 220 may be able to uniquely identify the object as the object rendered for display in the GUI 300 closest to the first location 114 of the spatial input 118. In some examples, the contents of the transcription 214 may further narrow down the possibilities of the object that the query refers to by at least describing the type of object or by indicating one or more features / characteristics of the object that the query refers to. Once the object 116 is uniquely identified, the point assistant 200 adds the object 116 to perform an action 232 that obtains information 246 about the object 116 requested by the query 104. Once the point assistant 200 obtains information 246 about the object 116 requested by the query 104, the response generator 250 provides a response 252 to the query 104 that includes the obtained information 246 about the object 116.

[0027] 3A, in some implementations, accepting the user input indication 110 indicating the spatial input 112 at the first location 114 includes detecting that the position of the cursor 310 is displayed in the GUI 300a at the first location 114 when the user 102 speaks the query 104. In these implementations, the NLU module 220 further receives image data 204 including a plurality of candidate objects 320, 320a-320c displayed in the GUI 300. Each candidate object 320 of the plurality of candidate objects 320 includes a corresponding location 322, 322a-322c in the GUI 300a displayed on the screen. These locations may be quantified or otherwise characterized using one or more coordinate systems, such as, for example, Cartesian coordinates using a pixel coordinate system whose origin is defined by the bottom left of the GUI 300a, or a polar coordinate system.

[0028] Additionally, each of the candidate objects 320 may be spatially defined by a bounding box 330a, 330a-330c, or a box having the smallest dimension within which all of the candidate objects 320 are contained. The NLU module 220 may identify the candidate object 320c from the plurality of candidate objects 320 having a corresponding position 322c closest to the first position 114 as the object 116 to which the query 104 refers. In some examples, if the bounding boxes 330 of two or more candidate objects 320 overlap, the NLU module 220 may use a best intersection technique to calculate the overlap between the two or more bounding boxes 330 to identify the object 116 to which the query 104 refers. In the illustrated example, the position of the cursor 310 indicates that the spatial input 112 is being applied at a position 114 where the object 116 including the sun is displayed.

[0029] In another implementation (not shown), the user input display 110 illustrating the spatial input 112 at the first location 114 includes detecting a touch input received within the GUI 300 at the first location 114 when the user 102 speaks the query 104. Alternatively, the user input display 110 illustrating the spatial input 112 at the first location 114 includes detecting a circling selection action performed within the GUI 300 at the first location 114 when the user 102 speaks the query 104.

[0030] 3B , in some implementations, accepting the user input display 110 indicating the spatial input 112 at the first location 114 includes detecting a circling selection action occurring within the GUI 300b at the first location 114. In response to detecting the circling selection action, the NLU module 220 uses the first location 114 within the image data 204 to crop a subset of the image data 204 contained within the area identified by the circling selection action and located at the first location 114 to uniquely identify the object 116 to which the query 104 refers. In the illustrated example, the objects within the area of ​​the circling selection action include a building.

[0031] 3C, in some implementations, accepting the user input display 110 indicating the spatial input 112 at the first location includes detecting an underline action occurring within the GUI 300c at the first location 114. In these implementations, the query 104 may be directed to a string of characters (e.g., "Bienvenue au cours de francais!") displayed within the GUI 300c at the first location 114. For example, the query 104 may include the phrase "What does this mean?" As in FIG. 3A, the NLU module 220 may identify a candidate object 320 (e.g., the underlined string of characters) having a corresponding location 322 closest to the first location 114 as the object 116 to which the query 104 refers. In other implementations (not shown), the user input display 110 showing the spatial input 112 at the first location 114 includes detecting a highlighting action performed in the GUI 300c that highlights a character string (e.g., "Bienvenue au cours de francais!") at the first location 114. In these implementations, the disambiguation model 230 disambiguates the query 104 to uniquely identify the object 116 to which the query refers as the character string highlighted by the highlighting action.

[0032] 2, once the NLU 220 disambiguates the query 104 and uniquely identifies the object 116, the NLU module 220 performs an action 232 to insert the object 116 into the missing object slot of the action 232 and obtain information 246 about the uniquely identified object 116 requested by the query 104. In some implementations, the point assistant 200 performs the identified action 232 to obtain the information 246 about the object 116 requested by the query 104 by querying an information source 240. In these implementations, the information source 240 can include a search engine 242, and the point assistant 200 queries the search engine 242 using the uniquely identified object 116 and one or more terms in the transcription 214 of the query 104 to obtain the information 246 about the object 116 requested by the query 104. For example, the point assistant 200 queries a search engine to obtain information 238 including one or more words in the transcription 214 "What's this?" plus a description of a golden retriever uniquely identified as the object 116 requested by the query. The information source may include an object recognition engine 244 that applies image processing techniques to detect and recognize a pattern (i.e., a golden retriever) in the image data 204 to classify the object 116 as a golden retriever and obtain information 238 providing information about golden retrievers. The information may include a link to a content source (e.g., a web page). That is, the information source 240 may use the image data 204 along with the transcription 214 of the query 104 to obtain the information 246 requested by the query 104. The response generator 250 receives the information 246 requested by the query 104 and generates a response 252 "It's a golden retriever." As described above, the answer generator 250 may generate an answer 252 to the query 104 as a text representation 19 that is displayed in a GUI 300 on the screen of the user device 10 .

[0033] In other examples, the point assistant 200 queries the search engine 242 to obtain a list of results responsive to the query 104. In these examples, the query 104 may be a similarity query 104, where the user 102 seeks a list of results that are visually similar to the object 116 in the GUI 300 on the screen of the user device 10. When the information source 240 returns information 246 including a list of results, the response generator 250 may generate a response 252 to the query 104 as a textual representation 19 including the list of results displayed in the GUI 300 on the screen of the user device 10. When the point assistant 200 displays the response 252, the point assistant 200 may further generate a graphical element representing the highest ranked result in the list of results responsive to the query 104, where the highest ranked result is displayed more prominently (e.g., in the first position 114, in a larger font, in an emphasized color) than the remaining results in the list of ranked results.

[0034] In some implementations, the point assistant 200 determines that the uniquely identified object 116 includes text in a first language (e.g., French), where the user 102 who spoke the query 104 may only speak a second language (e.g., English) that is different from the first language. For example, as shown in FIG. 3C, the uniquely identified object 116 includes the text "Bienvenue au cours de francais!" in the first language. When the point assistant queries the information source 240 for information 246 about the object, the information source 240 may obtain a translation of the uniquely identified object 116 in the second language: "Welcome to French class!". For example, the information source 240 may include a text-to-text machine translation model.

[0035] 4 is a flowchart of an example sequence of operations of a method 400 for context assistant using mouse pointing or touch cues. The method 400 includes, at operation 402, receiving audio data 202 corresponding to a query 104 spoken by a user 102 and captured by an assistant-enabled device (e.g., a user device) 10 associated with the user 102. The method 400 further includes, at operation 404, accepting a user input display 110 indicating a spatial input 112 applied at a first position 114 on the screen in a graphical user interface 300 displayed on a screen in communication with the data processing hardware 12. At operation 406, the method 400 includes processing the audio data 202 using a speech recognition model 212 to determine a transcription 214 of the query 104.

[0036] At operation 408, the method 400 also includes performing a query interpretation on the transcription 214 of the query 104 to determine that the query 104 references an object 116 displayed on the screen without uniquely identifying the object 116 and requests information 256 about the object 116 displayed on the screen. At operation 410, the method 400 further includes disambiguating the query 104 to uniquely identify the object 116 to which the query 104 refers using a user input display 110 indicative of a spatial input 112 applied at a first position 114 on the screen. At operation 412, in response to uniquely identifying the object 116, the method 400 includes obtaining information 246 about the object 116 requested by the query 104. At operation 414, the method 400 further includes providing a response 252 to the query 104 including the obtained information 246 about the object 116.

[0037] 5 is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described herein. Computing device 500 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0038] Computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 that connects to memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or may exist in other ways as desired. Processor 510 (e.g., data processing hardware 12 or data processing hardware 62 of FIG. 1) may process instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, and display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and memory types. Also, multiple computing devices 500 may be connected, each performing some of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0039] The memory 520 (e.g., memory hardware 14 or memory hardware 64 of FIG. 1) stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0040] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0041] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages low-bandwidth-intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and can connect, via a network adapter or the like, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0042] The computing device 500 can be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0043] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general purpose, coupled to receive data and instructions from the storage system, and to transmit data and instructions to the storage system.

[0044] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0045] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0046] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0047] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, for example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0048] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen, for displaying information to the user, and, optionally, a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user, for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, speech input, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0049] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method that, when executed by data processing hardware (510), causes the data processing hardware (510) to perform operations, the operations comprising: Receiving audio data (202) corresponding to a query (104) spoken by a user and captured by an assistant-enabled device associated with the user; receiving, in a graphical user interface (GUI (300)) displayed on a screen in communication with said data processing hardware (510), a user input representation (110) indicative of a spatial input (112) to be applied at a first location (114) on said screen; processing the audio data (202) using a speech recognition model (212) to determine a transcription (214) of the query (104); performing a query interpretation on the transcription (214) of the query (104); referencing an object (116) displayed on the screen without uniquely identifying said object (116); requesting information about the object (116) displayed on the screen; and and determining disambiguating the query (104) to uniquely identify the object (116) to which the query (104) refers using the user input representation (110) indicative of the spatial input (112) applied at the first position (114) on the screen; retrieving the information regarding the object (116) requested by the query (104) in response to uniquely identifying the object (116); providing a response (252) to the query (104) that includes the retrieved information regarding the object (116); A method comprising:

2. The operation includes detecting a trigger event; and in response to detecting the trigger event, the GUI (300) displayed on the screen to enable detection of a spatial input (112); The speech recognition model (212) for enabling the speech recognition to be performed on input audio data (202) captured by the assistant-enabled device; To start the The method of claim 1 , further comprising:

3. 3. The method (400) of claim 2, wherein detecting the trigger event comprises detecting the presence of a hot word (105) in the received audio data (202) by a hot word detector.

4. Detecting the trigger event includes: accepting a user input indication (110) indicating a selection of a graphical element (21) in the GUI (300) displayed on the screen; accepting a user input indication (110) indicating a selection of a physical button (17) located on the assistant-enabled device; detecting a predefined gesture made by said user; or Detecting a predefined movement / pose of the assistant-enabled device; The method of claim 2 , further comprising one of:

5. Receiving the user input representation (110) indicative of the spatial input (112) to be applied at the first location (114) includes: detecting when the user speaks the query (104), a position of a cursor (310) is displayed on the GUI (300) at the first position (114); detecting a touch input received at the GUI (300) at the first location (114) when the user speaks the query (104); or detecting a selection action (232) made in the GUI (300) at the first location (114) when the user speaks the query (104); Disambiguating the query (104) to uniquely identify the object (116) comprises: receiving image data (204) including a plurality of candidate objects (116) to be displayed on the GUI (300) and corresponding locations of the plurality of candidate objects (116) displayed on the GUI (300); identifying a candidate object (116) from the plurality of candidate objects (116) having a corresponding location (114) closest to the first location (114) as the object (116) to which the query (104) refers; The method (400) of any of claims 1 to 4, comprising:

6. Receiving the user input representation (110) indicative of the spatial input (112) applied at the first location (114) includes detecting an underline action (232) performed on the GUI (300) to underline a character string displayed on the GUI (300) at the first location (114); 6. The method (400) of claim 1, wherein disambiguating the query (104) to uniquely identify the object (116) comprises uniquely identifying the character string underlined by the underline action (232) as the object (116) to which the query (104) refers.

7. Receiving the user input representation (110) indicative of the spatial input (112) applied at the first location (114) includes detecting a highlighting action (232) taken on the GUI (300) that highlights a character string displayed on the GUI (300) at the first location (114); 7. The method (400) of claim 1, wherein disambiguating the query (104) to uniquely identify the object (116) comprises uniquely identifying the string highlighted by the highlighting action (232) as the object (116) to which the query (104) refers.

8. Obtaining the information about the object (116) requested by the query (104) comprises: querying a search engine (242) using the uniquely identified object (116) and one or more terms in the transcription (214) of the query (104) to obtain a list of results responsive to the query (104); A method (400) according to any preceding claim, wherein providing the response (252) to the query (104) including the retrieved information comprises displaying in the GUI (300) displayed on the screen a list of the results responsive to the query (104).

9. Displaying the list of results in response to the query (104) includes: generating a graphical element (21) representing a highest ranked result in said list of results responsive to said query (104); displaying, in the GUI (300) displayed on the screen, the list of results responsive to the query (104) at the first location (114) on the screen; 10. The method of claim 8, further comprising:

10. 10. The method (400) of claim 1, wherein the operations further include determining that the uniquely identified object (116) includes text in a first language, and wherein obtaining the information about the object (116) requested by the query (104) includes obtaining a translation of the text in a second language different from the first language.

11. Data processing hardware (510); Memory hardware (520) in communication with the data processing hardware (510), storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations, the operations including: Receiving audio data (202) corresponding to a query (104) spoken by a user and captured by an assistant-enabled device associated with the user; receiving, in a graphical user interface (GUI (300)) displayed on a screen in communication with said data processing hardware (510), a user input representation (110) indicative of a spatial input (112) to be applied at a first location (114) on said screen; processing the audio data (202) using a speech recognition model (212) to determine a transcription (214) of the query (104); performing a query interpretation on the transcription (214) of the query (104); referencing an object (116) displayed on the screen without uniquely identifying said object (116); requesting information about the object (116) displayed on the screen; and and determining disambiguating the query (104) to uniquely identify the object (116) to which the query (104) refers using the user input representation (110) indicative of the spatial input (112) applied at the first position (114) on the screen; retrieving the information regarding the object (116) requested by the query (104) in response to uniquely identifying the object (116); providing a response (252) to the query (104) that includes the retrieved information regarding the object (116); A memory hardware (520) including: A system (100).

12. The operation includes detecting a trigger event; and in response to detecting the trigger event, the GUI (300) displayed on the screen to enable detection of a spatial input (112); The speech recognition model (212) for enabling the speech recognition to be performed on input audio data (202) captured by the assistant-enabled device; To start the The system (100) of claim 11, further comprising:

13. 13. The system (100) of claim 12, wherein detecting the trigger event comprises detecting the presence of a hot word (105) in the received audio data (202) by a hot word detector.

14. Detecting the trigger event includes: accepting a user input indication (110) indicating a selection of a graphical element (21) in the GUI (300) displayed on the screen; accepting a user input indication (110) indicating a selection of a physical button (17) located on the assistant-enabled device; detecting a predefined gesture made by said user; or Detecting a predefined movement / pose of the assistant-enabled device; The system (100) of claim 12, comprising one of:

15. Receiving the user input representation (110) indicative of the spatial input (112) to be applied at the first location (114) comprises: detecting when the user speaks the query (104), a position of a cursor (310) is displayed on the GUI (300) at the first position (114); detecting a touch input received at the GUI (300) at the first location (114) when the user speaks the query (104); or detecting a selection action (232) made in the GUI (300) at the first location (114) when the user speaks the query (104); Disambiguating the query (104) to uniquely identify the object (116) comprises: receiving image data (204) including a plurality of candidate objects (116) to be displayed on the GUI (300) and corresponding locations of the plurality of candidate objects (116) displayed on the GUI (300); identifying a candidate object (116) from the plurality of candidate objects (116) having a corresponding location (114) closest to the first location (114) as the object (116) to which the query (104) refers; The system (100) according to any one of claims 11 to 14, comprising:

16. Receiving the user input representation (110) indicative of the spatial input (112) applied at the first location (114) includes detecting an underline action (232) performed on the GUI (300) to underline a character string displayed on the GUI (300) at the first location (114); The system (100) of any one of claims 11 to 15, wherein disambiguating the query (104) to uniquely identify the object (116) comprises uniquely identifying the string underlined by the underline action (232) as the object (116) to which the query (104) refers.

17. Receiving the user input representation (110) indicative of the spatial input (112) applied at the first location (114) includes detecting a highlighting action (232) taken on the GUI (300) that highlights a character string displayed on the GUI (300) at the first location (114); The system (100) of any one of claims 11 to 16, wherein disambiguating the query (104) to uniquely identify the object (116) comprises uniquely identifying the string highlighted by the highlight action (232) as the object (116) to which the query (104) refers.

18. Obtaining the information about the object (116) requested by the query (104) comprises: querying a search engine (242) using the uniquely identified object (116) and one or more terms in the transcription (214) of the query (104) to obtain a list of results responsive to the query (104); The system (100) of any one of claims 11 to 17, wherein providing the response (252) to the query (104) including the retrieved information comprises displaying a list of the results responsive to the query (104) within the GUI (300) displayed on the screen.

19. Displaying the list of results in response to the query (104) includes: generating a graphical element (21) representing a highest ranked result in said list of results responsive to said query (104); displaying, in the GUI (300) displayed on the screen, the list of results responsive to the query (104) at the first location (114) on the screen; The system (100) of claim 18, further comprising:

20. 20. The system (100) of claim 11, wherein the operations further include determining that the uniquely identified object (116) contains text in a first language, and wherein obtaining the information about the object (116) requested by the query (104) includes obtaining a translation of the text in a second language different from the first language.

Images