Contextual assistant using mouse pointing or touch cues

Abstract

A method (400) includes receiving audio data (202) corresponding to a query (104) spoken by a user, receiving, in a graphical user interface (300) displayed on a screen, a user input indication indicating a spatial input (212) applied at a first location (114) on the screen, and processing the audio data to determine a transcription (214) of the query. The method also includes performing a query interpretation on the transcription to determine that the query is referring to an object (116) displayed on the screen without having to uniquely identify the object and is requesting information about the object. The method further includes disambiguating the query using a user input indication indicative of a spatial input applied at a first location on the screen to uniquely identify an object that the query is referred to, obtaining information about the object requested by the query, and providing a response to the query (252).

Description

Technical Field

[0001] The present disclosure relates to contextual assistants using mouse pointing or touch cues. Background Art

[0002] A voice-enabled environment permits a user to speak a query out loud, and the digital assistant will perform actions to obtain an answer to the query. Digital assistants are particularly effective in providing accurate answers to general topic queries, where the query itself generates the information necessary for the digital assistant to obtain an answer to the query. However, in cases where the query is ambiguous, the digital assistant requires additional context to obtain an answer to the query. In some cases, recognizing the user's attention as the user speaks the query out loud provides the additional context needed to obtain an answer to the query. Therefore, a digital assistant receiving a query must have some way to recognize the additional context of the user speaking the query. Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed by data processing hardware, causes the data processing hardware to perform operations that include receiving audio data corresponding to a query spoken by a user and captured by a device associated with the user that supports an assistant. The operations also include: receiving a user input indication indicating a spatial input applied at a first position on the screen in a graphical user interface (GUI) displayed on a screen in communication with the data processing hardware, and processing the audio data using a speech recognition model to determine a transcription of the query. The operations also include performing query interpretation on the transcription of the query to determine that the query is referring to an object displayed on the screen without uniquely identifying the object, and is requesting information about the object displayed on the screen. The operations also include: using the user input indication indicating the spatial input applied at the first position on the screen to disambiguate the query to uniquely identify the object to which the query is referring, and in response to uniquely identifying the object, obtaining information about the object requested by the query. The operations also include providing a response to the query, the response including the obtained information about the object.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations also include: detecting a trigger event, and in response to detecting the trigger event, activating: a GUI displayed on the screen to enable detection of spatial input; and a speech recognition model to enable speech recognition to be performed on incoming audio data captured by the assistant-enabled device. In these implementations, detecting the trigger event includes detecting the presence of hot words in the received audio data via a hot word detector. Alternatively, detecting the trigger event may include one of the following: receiving a user input indication indicating a selection of a graphical element in a GUI displayed on the screen; receiving a user input indication indicating a selection of a physical button set on the assistant-enabled device; detecting a predefined gesture performed by a user; or detecting a predefined movement / posture of the assistant-enabled device.

[0005] In some examples, receiving a user input indication indicating a spatial input applied at a first location includes one of: detecting a position of a cursor displayed in a GUI at the first location when the user speaks the query; detecting a touch input received in the GUI at the first location when the user speaks the query; or detecting a lassoing action performed in the GUI at the first location when the user speaks the query. In these examples, disambiguating the query to uniquely identify the object includes: receiving image data including a plurality of candidate objects displayed in the GUI and corresponding locations of the plurality of candidate objects displayed in the GUI; and identifying a candidate object of the plurality of candidate objects having a corresponding location closest to the first location as the object to which the query is referring.

[0006] In additional examples, receiving a user input indication indicating a spatial input applied at a first location includes receiving a user input indication indicating a spatial input applied at the first location, and disambiguating the query to uniquely identify the object includes uniquely identifying a sequence of characters underlined by an underlining action as the object to which the query is referring. In other examples, receiving a user input indication indicating a spatial input applied at a first location includes detecting a highlighting action performed in a GUI that highlights a sequence of characters displayed at the first location in the GUI, and disambiguating the query to uniquely identify the object includes uniquely identifying the sequence of characters highlighted by the highlighting action as the object to which the query is referring.

[0007] In some implementations, obtaining information about the object requested by the query includes: querying a search engine using the uniquely identified object and one or more terms in the transcription of the query to obtain a result list responsive to the query; and displaying the result list responsive to the query in a GUI displayed on the screen. Here, displaying the result list responsive to the query may further include: generating a graphical element representing a highest-ranked result in the result list responsive to the query, and displaying the result list responsive to the query at a first position on the screen in the GUI displayed on the screen. Optionally, the operation may further include determining that the uniquely identified object includes text in a first language, so that obtaining information about the object requested by the query includes obtaining a translation of the text in a second language different from the first language.

[0008] Another aspect of the present disclosure provides a system that includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations that include: receiving audio data corresponding to a query spoken by a user and captured by a device associated with the user that supports an assistant. The operations also include: receiving a user input indication indicating a spatial input applied at a first position on the screen in a graphical user interface (GUI) displayed on a screen that communicates with the data processing hardware, and processing the audio data using a speech recognition model to determine a transcription of the query. The operations also include performing query interpretation on the transcription of the query to determine that the query is referring to an object displayed on the screen without uniquely identifying the object, and is requesting information about the object displayed on the screen. The operations also include: using the user input indication indicating the spatial input applied at the first position on the screen to disambiguate the query to uniquely identify the object to which the query is referring, and in response to uniquely identifying the object, obtaining information about the object requested by the query. The operations also include providing a response to the query, the response including the obtained information about the object.

[0009] This aspect may include one or more of the following optional features. In some implementations, the operations also include: detecting a trigger event, and in response to detecting the trigger event, activating: a GUI displayed on the screen to enable detection of spatial input; and a speech recognition model to enable speech recognition to be performed on incoming audio data captured by the assistant-enabled device. In these implementations, detecting the trigger event includes detecting the presence of hot words in the received audio data via a hot word detector. Alternatively, detecting the trigger event may include one of: receiving a user input indication indicating a selection of a graphical element in a GUI displayed on the screen; receiving a user input indication indicating a selection of a physical button set on the assistant-enabled device; detecting a predefined gesture performed by a user; or detecting a predefined movement / posture of the assistant-enabled device.

[0010] In some examples, receiving a user input indication indicating a spatial input applied at a first location includes one of: detecting a position of a cursor displayed in a GUI at the first location when the user speaks the query; detecting a touch input received in the GUI at the first location when the user speaks the query; or detecting a lassoing action performed in the GUI at the first location when the user speaks the query. In these examples, disambiguating the query to uniquely identify the object includes: receiving image data including a plurality of candidate objects displayed in the GUI and corresponding locations of the plurality of candidate objects displayed in the GUI; and identifying a candidate object of the plurality of candidate objects having a corresponding location closest to the first location as the object to which the query is referring.

[0011] In additional examples, receiving a user input indication indicating a spatial input applied at a first location includes receiving a user input indication indicating a spatial input applied at the first location, and disambiguating the query to uniquely identify the object includes uniquely identifying a sequence of characters underlined by an underlining action as the object to which the query is referring. In other examples, receiving a user input indication indicating a spatial input applied at a first location includes detecting a highlighting action performed in a GUI that highlights a sequence of characters displayed at the first location in the GUI, and disambiguating the query to uniquely identify the object includes uniquely identifying the sequence of characters highlighted by the highlighting action as the object to which the query is referring.

[0012] In some implementations, obtaining information about the object requested by the query includes: querying a search engine using the uniquely identified object and one or more terms in the transcription of the query to obtain a result list responsive to the query; and displaying the result list responsive to the query in a GUI displayed on the screen. Here, displaying the result list responsive to the query may further include: generating a graphical element representing a highest-ranked result in the result list responsive to the query, and displaying the result list responsive to the query at a first position on the screen in the GUI displayed on the screen. Optionally, the operation may further include determining that the uniquely identified object includes text in a first language, so that obtaining information about the object requested by the query includes obtaining a translation of the text in a second language different from the first language.

[0013] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1is a schematic diagram of an example system including a contextual assistant using mouse pointing or touch cues.

[0015] Figure 2 is a diagram of example components of a context assistant.

[0016] FIG. 3A to FIG. 3C is an example graphical user interface (GUI) rendered on a screen of a user device that includes a context assistant.

[0017] Figure 4 is a flow diagram of an example arrangement of operations of a method for disambiguating queries using mouse pointing or touch cues.

[0018] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0019] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0020] The way a user interacts with an assistant-enabled device is designed to be primarily, if not exclusively, through voice input. While an assistant-enabled device is effective at obtaining answers to general topic queries (e.g., what's the capital of Michigan?), context-driven queries require the assistant-enabled device to obtain additional information to obtain an accurate answer. For example, without more context, the assistant-enabled device may have difficulty obtaining a confident / accurate answer to the query "show me more of these."

[0021] In scenarios where a verbal query requires additional context to answer the query, an assistant-enabled device benefits from including image data derived from the screen of the assistant-enabled device. For example, a user may query the assistant-enabled device in a natural manner by saying “Show me more windows like that.” Here, the verbal query identifies that the user is looking for windows that are similar to an object, but the query is ambiguous because the object is unknown in the linguistic content of the query. Using image data from the screen of the assistant-enabled device may allow the assistant-enabled device to narrow down the potential windows to be searched from the entire screen showing the city to a distinct sub-area including a specific building in the city, in which a user input applied at a specific location on the screen has been detected in conjunction with the verbal query. By including input data and image data in conjunction with the query, the assistant-enabled device is able to generate a response to a query about buildings in the city, even if the user needs to explicitly identify the building in the verbal query.

[0022] Figure 1 4 is an example of a system 100 including a user device 10 and / or a remote system 60 in communication with the user device 10 via a network 40. The user device 10 and / or the remote system 60 executes a point assistant 200, with which the user 102 can interact through voice and spatial input, so that the point assistant 200 can generate a response to a query referring to an object displayed on a screen of the user device 10, although the query cannot uniquely identify the object for which the query seeks information. In the example shown, the user device 10 corresponds to a smart phone, however, the user device 10 may include other computing devices having a display screen or communicating with a display screen, such as, but not limited to, a tablet computer, a smart display, a desktop / laptop computer, a smart watch, a smart appliance, a smart glasses / head-mounted device, or a vehicle infotainment device. The user device 10 includes data processing hardware 12 and memory hardware 14, the memory hardware storing instructions that, when executed on the data processing hardware 12, cause the data processing hardware 12 to perform operations. The remote system 60 (e.g., a server, a cloud computing environment) also includes data processing hardware 62 and memory hardware 64, the memory hardware storing instructions that, when executed on the data processing hardware 62, cause the data processing hardware 62 to perform operations. As described in more detail below, the pointing assistant 200 executed on the user device 10 and / or the remote system 60 includes a speech recognizer 210 and a response generator 250, and has access to one or more information sources 240 stored on the memory hardware 14, 64. In some examples, execution of the pointing assistant 200 is shared across the user device 10 and the remote system 60.

[0023] The user device 10 includes an array of one or more microphones 16 configured to capture acoustic sounds directed toward the user device 10, such as speech. The user device 10 also executes a graphical user interface (GUI) 300 for display on a screen 18 in communication with the data processing hardware 12, the graphical user interface being configured to capture user input indications for controlling the functionality of the user device 10 via any of touch, gesture, gaze, and / or an input device (e.g., a mouse, a trackpad, or a stylus). The GUI 300 may be an interface associated with an application 50 executing on the user device 10 that presents a plurality of objects in the GUI 300. The user device 10 may further include or communicate with an audio output device (e.g., a speaker) 19 that may output audio from the pointing assistant 200, such as music and / or synthesized speech. The user device 10 may also include a physical button 17 disposed on the user device 10 and configured to receive a tactile selection by the user 102 to invoke the pointing assistant 200.

[0024] The user device 10 may include a method for extracting audio data 202 ( Figure 2 ) of the audio subsystem 106. For example, reference Figure 1 , the audio subsystem 106 may receive streaming audio corresponding to the utterance 106 of the query 104 spoken by the user 102 captured by one or more microphones 16 of the user device 10, and extract audio data (e.g., acoustic frames) 202. The audio data 202 may include acoustic features, such as Mel-frequency cepstral coefficients (MFCCs) or filter bank energies calculated over a window of the audio signal. In the example shown, the query 104 spoken by the user 102 includes “Hey Google, what is this?”

[0025] The user device 10 may execute (i.e., on the data processing hardware 12) a hot word detector 20 configured to detect the presence of a hot word 105 in the streaming audio without performing semantic analysis or speech recognition processing on the streaming audio. The hot word detector 20 may be executed on the audio subsystem 106. The hot word detector 202 may receive the audio data 202 to determine whether the utterance 106 includes a particular hot word 105 (e.g., Hey Google) spoken by the user 102. That is, the hot word detector 20 may be trained to detect the presence of a hot word 105 (e.g., Hey Google) or one or more other variations of the hot word (e.g., Ok Google) in the audio data 202. Detecting the presence of hotword 105 in audio data 202 may correspond to a triggering event that invokes pointing assistant 200 to activate GUI 300 displayed on screen 18 to enable detection of spatial input 112, and activates speech recognizer 210 to perform speech recognition on audio data 202 corresponding to utterance 106 of hotword 105 and / or one or more other terms representing query 104 following the hotword. In some examples, hotword 105 is spoken in utterance 106 following query 105, causing a portion of audio data 202 representing query 104 to be buffered and retrieved by speech recognizer 210, and when hotword 105 is detected in audio data 202, a portion of audio data 202 is retrieved. In some implementations, the triggering event includes receiving a user input indication indicating a selection of a graphical element 21 (e.g., a graphical microphone) in GUI 300. In other implementations, the triggering event includes receiving a user input indication indicating a selection of a physical button 17 disposed on user device 10. In other implementations, the triggering event includes detecting a predefined gesture performed by user 102 (e.g., via an image and / or radar sensor), or detecting a predefined movement / posture of user device 10 (e.g., using one or more sensors such as an accelerometer and / or gyroscope).

[0026] The user device 10 may further include an image subsystem 108 configured to extract a location 114 (e.g., an XY coordinate location) on the screen 18 of a spatial input 112 applied in the GUI 300. For example, the user 102 may provide a user input indication 110 indicating a spatial input 112 at a location 114 on the screen in the GUI 300. The image subsystem 108 may additionally extract image data (e.g., pixels) 204 corresponding to one or more objects 116 currently displayed on the screen 18. In the example shown, the GUI 300 receives a user input indication 110 indicating a spatial input 112 applied at a first location 114 on the screen 18, wherein the image data 202 includes an object (i.e., a golden retriever) 116 displayed on the screen 18 near the first location 114.

[0027] Continue to refer Figure 1 System 100 and Figure 2In the pointing assistant 200 of the present invention, a speech recognizer 210 executes an automatic speech recognition (ASR) model (e.g., a speech recognition model) 212 that receives audio data 202 as input and generates / predicts a corresponding transcription 214 of a query 104 as output. In the example shown, the query 104 includes the phrase "what is this?", which requests information 246 about an object 116 displayed in a GUI 300 on a screen without uniquely identifying the object 116. As described in more detail below, the pointing assistant 200 uses the spatial input 112 applied at a first position 114 on the screen 118 to disambiguate the query 104 to uniquely identify the object 116 to which the query 104 is referring. Once the object 116 is uniquely identified, the pointing assistant 200 can obtain information 246 about the object and generate a response 252 to the query 104 that includes the obtained information 246 about the object 116. Response generator 250 may generate a response 252 to query 104 as a textual representation. Here, pointing assistant 200 instructs user device 10 to display response 252 in GUI 300 for user 102 to read. In the example shown, pointing assistant 200 generates a textual representation of response 252 "That is a golden retriever" for display in GUI 300. As will be discussed in further detail below, pointing assistant 200 may require additional context extracted by image subsystem 108 (i.e., user 102 applied spatial input 112 at first location 114 corresponding to object 116) in order to uniquely identify object 116 to which query 104 is referring in order to obtain information 246 for inclusion in response 252. In some examples, response generator 250 employs text-to-speech (TTS) system 260 to convert the textual representation of response 252 into synthesized speech. In these examples, in addition to or in lieu of displaying a textual representation of response 252 in GUI 300 , pointing assistant 200 generates synthesized speech for audible output from speaker 19 of user device 10 .

[0028] See also Figure 2 , pointing assistant 200 further includes a natural language understanding (NLU) module 220, which is configured to perform query interpretation on the corresponding transcription 214 to ultimately determine the meaning behind the transcription 214. NLU module 220 can also receive context information 201 to assist in interpreting the transcription 214. Context information 201 can indicate the application 50 ( Figure 1), previous queries 104 from the user 102, detection of particular hot words 105, or any other information that the NLU module 220 may utilize to interpret the query 104. Continuing with this example, the context information 201 may indicate that the user is interacting with a web-based application 50 executing on the user device 102, and the NLU module 230 performs query interpretation to determine that the query 104 specifies an action 232 to obtain a description / information about a certain object 116 displayed in the GUI 300 that the user 102 may be viewing. However, the NLU module 230 determines that the query 104 is ambiguous because the object 116 is not explicitly identified in the transcription 214, but is instead directed to the term "this". In other words, the query interpretation performed by the NLU module 230 determines that the query 104 refers to the object 116 displayed on the screen 18 without uniquely identifying the object 116, and specifies the action 232 to request information 246 about the object 116.

[0029] To satisfy the query 104, the NLU module 220 needs to disambiguate the query 104 to uniquely identify the object 116 to which the query 104 is referring. For example, in a scenario where the query 104 includes the corresponding transcription 214 "show me similar bicycles" when multiple bicycles are currently displayed on the screen 18, the NLU module 220 can perform query interpretation on the corresponding transcription 214 to identify that the user 102 is referring to the object (i.e., bicycle) 116 displayed in the GUI 300 without uniquely identifying the object 116, and is requesting information 246 about the object 116 (i.e., other objects similar to the bicycle 116). In this example, the NLU module 220 determines that the query 104 specifies the action 232 to retrieve an image of a bicycle similar to one of the bicycles displayed on the screen, but the query 104 cannot be satisfied because the bicycle to which the query is referring cannot be determined from the transcription 214.

[0030] The NLU module 220 may use the user input indication indicating the spatial input 112 applied at the first position 114 on the screen as additional context for disambiguating the query 104 to uniquely identify the object 116 to which the query is referring. The NLU module 220 may additionally use the image data 204 to disambiguate the query 104. Here, the image data 204 may include a plurality of candidate objects displayed in the GUI and corresponding positions of the plurality of candidate objects displayed in the GUI. The image data 204 may be extracted by the image subsystem 108 from the graphical content rendered to be displayed in the GUI 300. The image data 204 may include a label identifying the candidate object. In some examples, the image subsystem 108 performs one or more object recognition techniques on the graphical content to identify the candidate object. By using the image data 204 and the received user input indication of the spatial input 118 applied at the first position 114, the NLU module 220 may be able to uniquely identify the object as the object rendered to be displayed in the GUI 300 closest to the first position 114 of the spatial input 118. In some examples, the content of transcription 214 can further narrow the possibilities of the object to which the query is referring by at least describing the type of the object or indicating one or more features / characteristics of the object to which the query is referring. Once object 116 is uniquely identified, pointing assistant 200 adds object 116 to perform action 232 of obtaining information 246 about object 116 requested by query 104. Once pointing assistant 200 obtains information 246 about object 116 requested by query 104, response generator 250 provides response 252 to query 104 that includes the obtained information 246 about object 116.

[0031] refer to Figure 3A In some implementations, receiving the user input indication 110 indicating the spatial input 112 at the first location 114 includes detecting that the location of the cursor 310 is displayed in the GUI 300a at the first location 114 when the user 102 speaks the query 104. In these implementations, the NLU module 220 further receives the image data 204 including a plurality of candidate objects 320 (320a to 320c) displayed in the GUI 300. Each candidate object 320 in the plurality of candidate objects 320 includes a corresponding location 322 (322a to 322c) in the GUI 300a displayed on the screen. For example, the locations may be quantified or otherwise characterized using one or more coordinate systems, such as Cartesian coordinates using a pixel coordinate system with the origin defined by the lower left corner of the GUI 300a, or a polar coordinate system.

[0032] In addition, each of the candidate objects 320 may be spatially defined by a bounding box 330a (330a to 330c) or a box with a minimum metric, within which all the candidate objects 320 are located. The NLU module 220 may identify a candidate object 320c having a corresponding position 322c closest to the first position 114 among the plurality of candidate objects 320 as the object 116 to which the query 104 is referring. In some examples, in the case where the bounding boxes 330 of two or more candidate objects 320 overlap, the NLU module 220 may employ an optimal intersection technique to calculate the overlap between the two or more bounding boxes 330 to identify the object 116 to which the query 104 is referring. In the example shown, the position of the cursor 310 indicates that the spatial input 112 is applied at the position 114 where the object 116 including the sun is displayed.

[0033] In other implementations (not shown), the user input indication 110 indicating the spatial input 112 at the first location 114 includes detecting a touch input received at the first location 114 in the GUI 300 when the user 102 speaks the query 104. Alternatively, the user input indication 110 indicating the spatial input 112 at the first location 114 includes detecting a lasso action performed at the first location 114 in the GUI 300 when the user 102 speaks the query 104.

[0034] refer to Figure 3B In some implementations, receiving the user input indication 110 indicating the spatial input 112 at the first location 114 includes detecting a lasso action performed in the GUI 300b at the first location 114. In response to detecting the lasso action, the NLU module 220 uses the first location 114 in the image data 204 to crop a subset of the image data 204 contained within the region identified by the lasso action and located at the first location 114 to uniquely identify the object 116 to which the query 104 is referring. In the example shown, the objects within the region of the lasso action include a building.

[0035] refer to Figure 3C In some implementations, receiving a user input indication 110 indicating a spatial input 112 at a first location includes detecting an underline action performed at a first location 114 in the GUI 300c. In these implementations, the query 104 can be for a character sequence displayed at the first location 114 in the GUI 300c (e.g., "Bienvenue au cours de français! (Welcome to French class!)"). For example, the query 104 can include the phrase "What does this say? (What does this mean?)" and Figure 3ASimilarly, the NLU module 220 can identify the candidate object 320 (e.g., the underlined character sequence) having the corresponding position 322 closest to the first position 114 as the object 116 to which the query 104 is referring. In other implementations (not shown), the user input indication 110 indicating the spatial input 112 at the first position 114 includes detecting a highlight action performed in the GUI 300c that highlights the character sequence at the first position 114 (e.g., "Bienvenue au cours de français!"). In these implementations, the disambiguation model 230 disambiguates the query 104 to uniquely identify the object 116 to which the query is referring as the character sequence highlighted by the highlight action.

[0036] Return to reference Figure 2 Once the NLU 220 disambiguates the query 104 to uniquely identify the object 116, the NLU module 220 inserts the object 116 into the missing object slot of the action 232 and performs the action 232 of obtaining the information 246 about the uniquely identified object 116 requested by the query 104. In some implementations, the pointing assistant 200 performs the identified action 232 to obtain the information 246 about the object 116 requested by the query 104 by querying an information source 240. In these implementations, the information source 240 can include a search engine 242, where the pointing assistant 200 queries the search engine 242 using the uniquely identified object 116 and one or more terms in the transcription 214 of the query 104 to obtain the information 246 about the object 116 requested by the query 104. For example, the pointing assistant 200 queries the search engine to obtain information 238 that includes a description of the golden retriever uniquely identified as the object 116 requested by the query in addition to the one or more terms in the transcription 214 of the query 104. The information source may include an object recognition engine 244 that applies image processing techniques to detect and recognize patterns (i.e., golden retrievers) in the image data 204 to obtain information 238 that classifies the object 116 as a golden retriever and provides information about golden retrievers. The information may include a link to a content source (e.g., a web page). That is, the information source 240 may use the image data 204 and the transcription 214 of the query 104 to obtain the information 246 requested by the query 104. The response generator 250 receives the information 246 requested by the query 104 and generates a response 252 "That is a golden retriever". As discussed above, the response generator 250 may generate the response 252 to the query 104 as a text representation 19 in the GUI 300 displayed on the screen of the user device 10.

[0037] In other examples, the pointing assistant 200 queries the search engine 242 for a list of results that is responsive to the query 104. In these examples, the query 104 may be a similarity query 104 in which the user 102 seeks a list of results that have a visual similarity to the object 116 in the GUI 300 on the screen of the user device 10. Once the information source 240 returns information 246 including the list of results, the response generator 250 may generate a response 252 to the query 104 as a textual representation 19 including the list of results that is displayed in the GUI 300 on the screen of the user device 10. When the pointing assistant 200 displays the response 252, it may further generate a graphical element representing the highest ranked result in the list of results that is responsive to the query 104, wherein the highest ranked result is displayed more prominently (e.g., at the first position 114, in a larger font, in a highlighted color) than the rest of the results in the ranked list of results.

[0038] In some implementations, pointing assistant 200 determines that uniquely identified object 116 includes text in a first language (e.g., French). Here, user 102 speaking query 104 may only speak a second language (e.g., English) that is different from the first language. Figure 3C As shown, the uniquely identified object 116 includes the text "Bienvenue au coursde français!" in the first language. When the pointing assistant queries the information source 240 for information 246 about the object, the information source 240 can obtain the translation of the uniquely identified object 116 in the second language, "Welcome to French class! (Welcome to French class!)". For example, the information source 240 can include a text-to-text machine translation model.

[0039] Figure 4 4 is a flow chart of an exemplary arrangement of operations of a method 400 for contextual assistant using mouse pointing or touch cues. At operation 402, the method 400 includes receiving audio data 202 corresponding to a query 104 spoken by a user 102 and captured by an assistant-enabled device (e.g., a user device) 10 associated with the user 102. At operation 404, the method 400 further includes receiving a user input indication 110 indicating a spatial input 112 applied at a first position 114 on the screen in a graphical user interface 300 displayed on the screen in communication with the data processing hardware 12. At operation 406, the method 400 includes processing the audio data 202 using a speech recognition model 212 to determine a transcription 214 of the query 104.

[0040] At operation 408, the method 400 further includes performing query interpretation on the transcription 214 of the query 104 to determine that the query 104 is referring to the object 116 displayed on the screen without uniquely identifying the object 116, and is requesting information 256 about the object 116 displayed on the screen. At operation 410, the method 400 further includes disambiguating the query 104 using the user input indication 110 indicating the spatial input 112 applied at the first position 114 on the screen to uniquely identify the object 116 to which the query 104 is referring. At operation 412, in response to uniquely identifying the object 116, the method 400 includes obtaining the information 246 about the object 116 requested by the query 104. At operation 414, the method 400 further includes providing a response 252 to the query 104, the response including the obtained information 246 about the object 116.

[0041] Figure 5 is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0042] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or in other ways as appropriate. The processor 510 (e.g., Figure 1 The data processing hardware 12 or data processing hardware 62 of the computing device 500 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or stored on the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 580 coupled to the high-speed interface 540). In other implementations, multiple processors and / or multiple buses as well as multiple memories and multiple types of memories can be used as appropriate. Moreover, multiple computing devices 500 can be connected, with each device providing a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).

[0043] Memory 520 (e.g., Figure 1The memory hardware 14 or memory hardware 64 of the computing device 500 stores information non-temporarily. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0044] Storage device 530 is capable of providing mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as memory 520, storage device 530, or a memory on processor 510.

[0045] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or a router, for example, through a network adapter.

[0046] The computing device 500 may be implemented in a variety of different forms, as shown. For example, the computing device may be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0047] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0048] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0049] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0050] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0051] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, a processor suitable for executing a computer program includes both a general-purpose microprocessor and a special-purpose microprocessor, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0052] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0053] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method, which, when executed by data processing hardware (510), causes the data processing hardware (510) to perform operations comprising: receiving audio data (202) corresponding to a query (104) spoken by a user and captured by an assistant-supporting device associated with the user; receiving, in a graphical user interface (GUI (300)) displayed on a screen in communication with the data processing hardware (510), a user input indication (110) indicating a spatial input (112) applied at a first location (114) on the screen; processing the audio data (202) using a speech recognition model (212) to determine a transcription (214) of the query (104); Query interpretation is performed on the transcription (214) of the query (104) to determine that for the query (104): referring to an object (116) displayed on the screen without uniquely identifying the object (116); and requesting information about said object (116) displayed on said screen; disambiguating the query (104) using the user input indication (110) indicating the spatial input (112) applied at the first location (114) on the screen to uniquely identify the object (116) to which the query (104) is referring; Responsive to uniquely identifying the object (116), obtaining the information about the object (116) requested by the query (104); and A response (252) to the query (104) is provided, the response including the obtained information about the object (116).

2. The method (400) of claim 1, wherein: The operations further include detecting a triggering event, and in response to detecting the triggering event, activating: the GUI (300) displayed on the screen to enable detection of spatial input (112); as well as The speech recognition model (212) enables speech recognition to be performed on incoming audio data (202) captured by the assistant-enabled device.

3. The method (400) of claim 2, wherein: Detecting the trigger event includes detecting, by a hot word detector, the presence of a hot word (105) in the received audio data (202).

4. The method (400) of claim 2, wherein: Detecting the trigger event includes one of the following: receiving a user input indication (110) indicating a selection of a graphical element (21) in the GUI (300) displayed on the screen; receiving a user input indication (110) indicating a selection of a physical button (17) provided on the assistant-supporting device; detecting a predefined gesture performed by the user; or A predefined movement / gesture of the assistant-enabled device is detected.

5. The method (400) of any one of claims 1 to 4, wherein: Receiving the user input indication (110) indicating the spatial input (112) applied at the first location (114) includes one of: detecting that a position of a cursor (310) was displayed in the GUI (300) at the first position (114) when the user spoke the query (104); detecting touch input received at the first location (114) in the GUI (300) when the user speaks the query (104); or detecting a lasso action (232) performed in the GUI (300) at the first location (114) when the user speaks the query (104); and Disambiguating the query (104) to uniquely identify the object (116) includes: receiving image data (204), the image data comprising a plurality of candidate objects (116) displayed in the GUI (300) and corresponding positions of the plurality of candidate objects (116) displayed in the GUI (300); as well as A candidate object (116) having the corresponding position (114) closest to the first position (114) is identified from the plurality of candidate objects (116) as the object (116) to which the query (104) is referring.

6. The method (400) of any one of claims 1 to 5, wherein: Receiving the user input indication (110) indicating the spatial input (112) applied at the first position (114) includes detecting an underline action (232) performed in the GUI (300), the underline action underlining a sequence of characters displayed in the GUI (300) at the first position (114); and Disambiguating the query (104) to uniquely identify the object (116) includes uniquely identifying the sequence of characters underlined by the underlining action (232) as the object (116) to which the query (104) is referring.

7. The method (400) of any one of claims 1 to 6, wherein: Receiving the user input indication (110) indicating the spatial input (112) applied at the first location (114) includes detecting a highlighting action (232) performed in the GUI (300), the highlighting action causing a sequence of characters displayed in the GUI (300) at the first location (114) to be highlighted; and Disambiguating the query (104) to uniquely identify the object (116) includes uniquely identifying the sequence of characters highlighted by the highlighting action (232) as the object (116) to which the query (104) is referring.

8. The method (400) of any one of claims 1 to 7, wherein: Obtaining the information about the object (116) requested by the query (104) includes: querying a search engine (242) using the uniquely identified object (116) and one or more terms in the transcription (214) of the query (104) to obtain a list of results responsive to the query (104); and Providing the response (252) to the query (104) including the obtained information includes displaying the list of results responsive to the query (104) in the GUI (300) displayed on the screen.

9. The method (400) of claim 8, wherein: Displaying the list of results in response to the query (104) further comprises: generating a graphical element (21) representing a highest ranked result in the list of results responsive to the query (104); and The list of results responsive to the query (104) is displayed in the GUI (300) displayed on the screen at the first position (114) on the screen.

10. The method (400) according to any one of claims 1 to 9, wherein: The operations further include determining that the uniquely identified object (116) includes text in a first language, wherein obtaining the information about the object (116) requested by the query (104) includes obtaining a translation of the text in a second language different from the first language.

11. A system (100), comprising: Data processing hardware (510); as well as Memory hardware (520) in communication with the data processing hardware (510), the memory hardware (520) storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations including: receiving audio data (202) corresponding to a query (104) spoken by a user and captured by an assistant-supporting device associated with the user; receiving, in a graphical user interface (GUI (300)) displayed on a screen in communication with the data processing hardware (510), a user input indication (110) indicating a spatial input (112) applied at a first location (114) on the screen; processing the audio data (202) using a speech recognition model (212) to determine a transcription (214) of the query (104); Query interpretation is performed on the transcription (214) of the query (104) to determine that for the query (104): referring to an object (116) displayed on the screen without uniquely identifying the object (116); and requesting information about said object (116) displayed on said screen; disambiguating the query (104) using the user input indication (110) indicating the spatial input (112) applied at the first location (114) on the screen to uniquely identify the object (116) to which the query (104) is referring; Responsive to uniquely identifying the object (116), obtaining the information about the object (116) requested by the query (104); and A response (252) to the query (104) is provided, the response including the obtained information about the object (116).

12. The system (100) of claim 11, wherein: The operations further include detecting a triggering event, and in response to detecting the triggering event, activating: the GUI (300) displayed on the screen to enable detection of spatial input (112); as well as The speech recognition model (212) enables speech recognition to be performed on incoming audio data (202) captured by the assistant-enabled device.

13. The system (100) of claim 12, wherein: Detecting the trigger event includes detecting, by a hot word detector, the presence of a hot word (105) in the received audio data (202).

14. The system (100) of claim 12, wherein: Detecting the trigger event includes one of the following: receiving a user input indication (110) indicating a selection of a graphical element (21) in the GUI (300) displayed on the screen; receiving a user input indication (110) indicating a selection of a physical button (17) provided on the assistant-supporting device; detecting a predefined gesture performed by the user; or A predefined movement / gesture of the assistant-enabled device is detected.

15. The system (100) of any one of claims 11 to 14, wherein: Receiving the user input indication (110) indicating the spatial input (112) applied at the first location (114) includes one of: detecting that a position of a cursor (310) was displayed in the GUI (300) at the first position (114) when the user spoke the query (104); detecting touch input received at the first location (114) in the GUI (300) when the user speaks the query (104); or detecting a lasso action (232) performed in the GUI (300) at the first location (114) when the user speaks the query (104); and Disambiguating the query (104) to uniquely identify the object (116) includes: receiving image data (204), the image data comprising a plurality of candidate objects (116) displayed in the GUI (300) and corresponding positions of the plurality of candidate objects (116) displayed in the GUI (300); as well as A candidate object (116) having the corresponding position (114) closest to the first position (114) is identified from the plurality of candidate objects (116) as the object (116) to which the query (104) is referring.

16. The system (100) of any one of claims 11 to 15, wherein: Receiving the user input indication (110) indicating the spatial input (112) applied at the first position (114) includes detecting an underline action (232) performed in the GUI (300), the underline action underlining a sequence of characters displayed in the GUI (300) at the first position (114); and Disambiguating the query (104) to uniquely identify the object (116) includes uniquely identifying the sequence of characters underlined by the underlining action (232) as the object (116) to which the query (104) is referring.

17. The system (100) of any one of claims 11 to 16, wherein: Receiving the user input indication (110) indicating the spatial input (112) applied at the first location (114) includes detecting a highlighting action (232) performed in the GUI (300), the highlighting action causing a sequence of characters displayed in the GUI (300) at the first location (114) to be highlighted; and Disambiguating the query (104) to uniquely identify the object (116) includes uniquely identifying the sequence of characters highlighted by the highlighting action (232) as the object (116) to which the query (104) is referring.

18. The system (100) according to any one of claims 11 to 17, wherein: Obtaining the information about the object (116) requested by the query (104) includes: querying a search engine (242) using the uniquely identified object (116) and one or more terms in the transcription (214) of the query (104) to obtain a list of results responsive to the query (104); and Providing the response (252) to the query (104) including the obtained information includes displaying the list of results responsive to the query (104) in the GUI (300) displayed on the screen.

19. The system (100) of claim 18, wherein: Displaying the list of results in response to the query (104) further comprises: generating a graphical element (21) representing a highest ranked result in the list of results responsive to the query (104); and The list of results responsive to the query (104) is displayed in the GUI (300) displayed on the screen at the first position (114) on the screen.

20. The system (100) according to any one of claims 11 to 19, wherein: The operations further include determining that the uniquely identified object (116) includes text in a first language, wherein obtaining the information about the object (116) requested by the query (104) includes obtaining a translation of the text in a second language different from the first language.

Images