Resolving automated assistant requests based on image and / or other sensor data

Abstract

Techniques are provided for resolving automated assistance requests that are based on imagery and / or other sensor data. [Solution] Methods, apparatus, and computer-readable media are described that relate to causing processing of sensor data in response to a determination of a request related to an environmental object that may be captured by the sensor data. Some implementations further relate to determining whether the request is solvable based on the processing of the sensor data. If the request is determined to be unsolvable, a prompt is determined and provided as a user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In these implementations, further input (e.g., additional sensor data and / or user interface input) received in response to the prompt can then be utilized to resolve the request.

Description

[Technical field]

[0001] The present disclosure relates to resolving automated assistant requests that are based on image and / or other sensor data. [Background technology]

[0002] Image processing can be utilized to resolve attributes for objects in an image. For example, some image processing techniques utilize an image processing engine to resolve classifications for objects captured in an image. For example, for an image capturing a sailboat, image processing can be performed to resolve classification values ​​of "boat" and / or "sailing ship" for the image. Image processing can be utilized to resolve additional or alternative attributes. For example, optical character recognition (OCR) can be utilized to resolve text within an image. Also, for example, some image processing techniques can be utilized to determine a more detailed classification of an object in an image (e.g., a particular make and / or model of sailboat).

[0003] Some image processing engines utilize one or more machine learning models, such as deep neural network models, that accept an image as input and utilize learning parameters to generate, as an image-based output, a measure indicating which of a plurality of corresponding attributes are present in the image. If the measure indicates that a particular attribute is present in the image (e.g., if the measure meets a threshold), then the attribute can be considered "solved" for the image (i.e., the attribute can be considered to be present in the image). However, it will often be the case that image processing of an image is unable to resolve one or more (e.g., any) attributes. Moreover, it may even be the case that the resolved attributes for an image do not allow for the definition of an object in the image with a desired specificity. For example, the resolved attributes of an image may allow for the determination that a "shirt" is present in the image and that the shirt is "red," but may not allow for the determination of the shirt's manufacturer, whether the shirt is "short-sleeved" or "long-sleeved," etc.

[0004] Alternatively, humans may engage in human-computer interactions with interactive software applications referred to herein as "automated assistants" (also referred to as "interactive personal assistants," "intelligent personal assistants," "personal voice assistants," "conversational agents," etc.). Automated assistants often receive natural language input (utterances) from a user. Natural language input may in some cases be received as audio input (e.g., streaming audio) and converted to text, and / or may be received as textual (e.g., typed) natural language input. Automated assistants respond to the natural language input with a response (e.g., visual and / or audible natural language output). However, it will often be the case that an automated assistant will not accept and / or respond to requests that are based on sensor data (e.g., images) capturing one or more properties of environmental objects. Summary of the Invention [Means for solving the problem]

[0005] Implementations described herein relate to causing processing of sensor data in response to a determination of a request related to an environmental object that may be captured by the sensor data. For example, image processing may be performed on an image in response to a request determined based on natural language input provided by a user in conjunction with the capture of at least one image (e.g., natural language input received immediately before, after, and / or during the capture of at least one image). For example, a user may provide a voice input of "What's wrong with my device" via an automated assistant interface of a client device. The voice input may be determined to be related to an environmental object, and as a result, image processing may be performed on an image captured by a camera of the client device. The image may be captured by the camera based on a separate user interface input (e.g., selection of a "capture image" interface element) or may be automatically captured in response to a determination that the voice input is related to an environmental object.

[0006] Some implementations described herein further relate to determining whether the request is solvable based on processing of sensor data. For example, the request can be determined to be unsolvable based on a determination that the attributes (if any) resolved based on image processing of at least one image cannot define the object with a target specificity. If the request is determined to be unsolvable, a prompt is determined and provided as a user interface output (e.g., audible and / or graphical), where the prompt provides a pointer to further input that would enable the request to be resolved. The prompt can instruct the user to obtain additional sensor data (e.g., images, audio, temperature sensor data, weight sensor data) for the object and / or move the object (and / or other objects) to enable acquisition of additional sensor data for the object. For example, the prompt can be modified to enable acquisition of additional images that enable resolution of one or more attributes that are unresolved based on image processing of at least one image. The prompt can additionally or alternatively ask the user to provide a user interface input (e.g., natural language input) directed to the unresolved attributes of the object.

[0007] In those implementations, additional sensor data (e.g., additional images) and / or user interface input received in response to the prompt can then be leveraged to resolve the request. For example, image processing can be performed on the additional images received in response to the prompt, and additional attributes resolved from the image processing can be leveraged to resolve the request. For example, the request can be resolved by submitting an agent request to one or more agents (e.g., a search system and / or other agents), where the agent request is generated based on the additional attributes resolved from image processing of the additional images, and optionally based on attributes determined based on processing of prior sensor data (e.g., determined based on image processing of prior images). As another example, additional attributes can be resolved based on natural language input or other user interface input received in response to the prompt, and such additional attributes can be leveraged to resolve the request. It is noted that in some implementations and / or situations, multiple prompts can be provided, additional attributes can be determined from additional sensor data and / or user interface input responsive to those prompts, and such additional attributes can be leveraged in resolving the request.

[0008] As will be appreciated, the methods and systems described herein may assist a user in effecting the performance of a particular task through a guided human-machine interaction process. A variety of different tasks may be performed, which may include, for example, enabling a user to obtain information required to perform a further task. For example, as described with reference to Figures 6 and 7, the obtained information may facilitate the performance of maintenance on another device.

[0009] As described above, a request can be determined to be unresolvable based on a determination that the attributes (if any) resolved based on processing of the sensor data cannot define the object with a target specificity. In some implementations, the target specificity for the object can be a target classification of the object in a taxonomy. For example, a target classification of a car can be a classification at a level that defines the make and model of the car, or at a level that defines the make, model, and year of the car. In some implementations, the target specificity for the object can be defined in terms of one or more items to be defined, where the items for the object can depend on the classification (general or specific) of the object. For example, for a bottle of wine, the items can be defined for a particular brand, wine type, and / or vintage, and the target specificity is the resolution of the attributes for all of those items. In some implementations, the target specificity can additionally or alternatively be determined based on an initial natural language input provided by the user, feedback provided by the user, the interaction history of the user and / or other users, and / or location and / or other context signals.

[0010] As also described above, the determined prompt can provide guidance for further input that would enable resolution of the request related to the environmental object. In some implementations, the prompt is determined based on one or more attributes for the resolved object. For example, classification attributes for the environmental object can be leveraged, such as classification attributes resolved based on image processing of a previously acquired image of the object. For example, a prompt for a "car" classification can be specialized to the car classification (e.g., "take a picture of the car from a different angle"). And, for example, a prompt for a "jacket" classification can be specialized to the "jacket" classification (e.g., "take a picture of the logo or tag"). Also, for example, as described above, classification attributes can be associated with one or more items to be defined. In some of these situations, the prompt can be generated based on items not defined by the already resolved attributes (if any). For example, if a vintage item for a bottle of wine has not yet been defined, the prompt can be "take a picture of the age" or "how old is it?".

[0011] Some implementations described herein may provide the described prompts for presentation to the user only when (1) there is a request related to the environmental object (e.g., a request for additional information) and / or (2) it is determined that the request cannot be resolved based on processing of previously collected sensor data. In this way, computational resources are not wasted through the provision of unnecessary prompts and / or the processing of further inputs that would be responsive to such unnecessary prompts. For example, if a user captures an image of a bottle of wine and provides a natural language input of "send this picture to Bob", a prompt requesting the user to take an additional image of the bottle of wine would not be provided based on the request (i.e., sending the image to Bob) being resolvable based solely on the captured image and / or based on the request not being a request for additional information related to the bottle of wine. If the natural language input is instead "how much does this cost", a prompt may be provided requesting the user to take an additional image if image processing of the initial image cannot resolve sufficient attributes to resolve the request (i.e., determine the price). For example, a prompt "take a picture of the label" may be provided if the brand, wine type, and / or vintage cannot be resolved based on image processing. As yet another example, when a user is at a "retail" location, a request related to an environmental object in an acquired image can be inferred, whereas if the user instead acquired the same environmental object at a park, the request would not be inferred (under the assumption that the user may be looking for shopping information while at the retail location, such as prices, reviews, etc.). As yet another example, if the initially acquired images all capture multiple objects at a long distance, the request may not be inferred, whereas the request would have been inferred if the images instead captured only one object at a close distance.

[0012] Some implementations described herein can also determine prompts that are modified for attributes already resolved (e.g., classification attributes), for items yet to be resolved, and / or otherwise modified to enable resolution of the request. In this manner, prompts can be modified to increase the likelihood that inputs (e.g., images and / or user interface inputs) responsive to the prompts will enable resolution of the request, thereby mitigating the need for further prompts in resolving the request and / or the processing of further inputs that would be responsive to such further prompts.

[0013] As an explicit example of some implementations, assume that a user provides voice input of "What type of reviews is this getting?" while pointing the camera of a client device (e.g., smartphone, tablet, wearable device) at a bottle of wine. It can be determined that the voice input includes a request related to an object in the environment of the client device. In response to the voice input including the request, an image of the wine bottle can be processed to resolve one or more attributes of the object in the image. For example, the image can be processed to determine a classification of "bottle" and / or "wine bottle." In some implementations, the image can be obtained based on user interface input or can be obtained automatically based on a determination that the voice input includes the request.

[0014] Further, it may be determined that the request is not resolvable based on the resolved "bottle" and / or "wine bottle" classification. For example, based on speech input, it may be determined that the request is for a review on a particular wine (e.g., a particular brand, wine type, and / or vintage) in a bottle - and not on "wine bottles" in general. Thus, resolution of the request requires resolution of sufficient attributes to enable determination of the particular wine in the bottle. The general classification of "bottle" and "wine bottle" may not enable such a determination.

[0015] In response to a determination that the request is not resolvable, a prompt may then be provided. For example, the prompt may be "Can you take a picture of the label?" or "Can you make the barcode visible to the camera?", etc. In response to the prompt, the user may move the wine bottle and / or the electronic device (and thus the camera) and additional images are acquired following such movement. Processing of the additional images may then be performed to determine attributes for the selected parameters. For example, OCR processing may be performed to determine text values ​​for the label, such as text including the brand name, wine type, and vintage. If processing of the additional images still does not enable resolution of the request (e.g., if the required attributes are still unresolved), a further prompt may be generated and image processing of further images received after the further prompt is performed.

[0016] Based on the additional attributes, the additional content can then be generated. For example, the additional content can be received in response to issuing a search based on the additional attributes and / or verbal input. For example, the text value "Vineyard A Cabernet Sauvignon 2012" may have been determined, a query "Reviews for Vineyard A Cabernet Sauvignon 2012" is submitted (e.g., to a search system agent), and the additional content is received in response to the query.

[0017] As described herein, one or more prompts may additionally or alternatively ask the user to provide a responsive user interface input to enable resolution of attributes for the unresolved item. For example, assume that processing of one or more images is utilized to determine a "wine bottle" classification value, but is unable to resolve enough text to unambiguously identify the wine. For example, text identifying a particular brand was identified, but text identifying a wine type and age was not identified. Instead of or in addition to prompting the user to obtain additional images and / or move the bottle of wine, a prompt may request that the user identify the wine type and age (e.g., "Can you tell me the wine type and age for brand X wine?"). A responsive user interface input provided by the user may then be utilized to resolve the wine type and age. In some implementations, the prompt may be generated to include one or more candidate attributes determined based on image processing. For example, assume that OCR image processing techniques are utilized to determine candidate vintages of "2017" and "2010". For example, an image processing technique might identify "2017" and "2010" as candidates, but neither with sufficient confidence to allow resolution of a specific vintage. In such a situation, the prompt might be "Is this a 2010 or a 2017 vintage?" or might provide selectable options of "2010" and "2017."

[0018] In some implementations, multiple processing engines and / or models may be operated in parallel and each may be specifically configured for one or more particular items. For example, a first image processing engine may be a general classification engine configured to determine general entities in an image, a second image processing engine may be a logo processing engine configured to determine the brand of a logo in an image, a third image processing engine may be an OCR or other character recognition engine configured to determine text and / or numbers in an image, etc. In some of these implementations, a prompt may be generated based on which image processing engine was unable to resolve attributes for the corresponding item. Further, in some of these implementations, in response to additional images received in response to the prompt, only a subset of the engines may be utilized to process such additional images. For example, only engines configured to resolve unresolved items / parameters may be utilized, thereby saving various computational resources by not utilizing the entire set of engines for such images.

[0019] In some implementations, a method performed by one or more processors is provided, comprising receiving a voice input provided by a user via an automated assistant interface of a client device, and determining that the voice input includes a request related to an object in an environment of the client device. The method further comprises, in response to determining that the voice input includes a request related to the object, causing processing to be performed on initial sensor data acquired by at least one sensor. The at least one sensor is of the client device or an additional electronic device in the environment, and the initial sensor data captures one or more characteristics of the object. The method further comprises determining that the request is not resolvable based on the initial sensor data based on one or more initial attributes of the object resolved based on processing of the initial sensor data. The method further comprises, in response to determining that the request is not resolvable, providing a prompt for presentation to the user via the automated assistant interface of the client device, instructing the user to acquire additional sensor data or move the object. The method further comprises receiving additional sensor data acquired by the client device or the additional electronic device after the prompt is presented to the user, causing processing to be performed on the additional sensor data, and resolving the request based on at least one additional attribute resolved based on processing of the additional sensor data.

[0020] In some implementations, a method performed by one or more processors is provided, comprising receiving at least one image captured by a camera of a client device and determining that the at least one image is associated with a request related to an object captured by the at least one image. The method further comprises, in response to determining that the image is associated with a request related to the object, causing image processing to be performed on the at least one image. The method further comprises, based on the image processing of the at least one image, determining that at least one parameter required to resolve the request is not resolvable based on the image processing of the at least one image. The method further comprises, in response to determining that the at least one parameter is not resolvable, providing a prompt for presentation via the client device or an additional client device, the prompt being modified for the at least one parameter. The method further comprises, in response to the prompt, receiving an additional image captured by the camera and / or a user interface input, resolving a given attribute for the at least one parameter based on the additional image and / or the user interface input, and resolving the request based on the given attribute.

[0021] In some implementations, a method performed by one or more processors is provided, comprising receiving a natural language input provided by a user via an automated assistant interface of a client device, and determining that the natural language input includes a request related to an object in an environment of the client device. The method further comprises, in response to determining that the natural language input includes a request related to the object, causing processing to be performed on initial sensor data acquired by a sensor of the client device or an additional electronic device in the environment. The method further comprises, based on one or more initial attributes of the object resolved based on processing of the initial sensor data, determining that the request is not resolvable based on the initial sensor data. In response to determining that the request is not resolvable, the method further comprises providing a prompt for presentation to the user via the automated assistant interface of the client device. The method further comprises, in response to the prompt, receiving a natural language input or an image, and resolving the request based on the natural language input or the image.

[0022] In some implementations, a method performed by one or more processors is provided, and includes processing at least one image acquired by a camera of an electronic device to resolve one or more attributes for an object in the at least one image; selecting one or more items for the object that are undefined by the attributes resolved by the processing of the at least one image; providing, via the electronic device or an additional electronic device, a prompt that is modified to fit at least one of the selected one or more items; receiving, in response to the prompt, at least one of an additional image acquired by the camera and user interface input; resolving given attributes for the selected one or more items based on the at least one of the additional image and the user interface input; determining additional content based on the resolved given attributes; and providing, via the electronic device, the additional content for presentation to a user.

[0023] In some implementations, a method performed by one or more processors is provided, comprising: processing at least one image acquired by a camera of an electronic device to resolve one or more attributes for an object in the at least one image; selecting one or more items for the object that are undefined by the attributes resolved by the processing of the at least one image; providing a prompt for presentation to a user via the electronic device or an additional electronic device; receiving at least one additional image acquired after providing the prompt; and selecting a subset of available image processing engines for processing the at least one additional image. The available image processing engines of the subset are selected based on being associated with the resolution of the one or more items. The method further comprises resolving one or more additional attributes for the one or more items based on application of the at least one additional image to the selected subset of available image processing engines. The step of resolving the one or more additional attributes occurs without any application of the at least one additional image to the remainder of the available image processing engines that are not included in the selected subset.

[0024] Additionally, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the methods described above. Some implementations also include one or more non-transitory computer-readable storage media having stored thereon computer instructions executable by the one or more processors to perform any of the methods described above.

[0025] It should be appreciated that all combinations of the above concepts, and additional concepts described in more detail herein, are contemplated as part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a block diagram of an example environment in which the techniques disclosed herein can be implemented. [Figure 2A] Following are examples of implementations that illustrate how the techniques described herein can be utilized. [Figure 2B] Following are examples of implementations that illustrate how the techniques described herein can be utilized. [Diagram 3] Following are examples of implementations that illustrate how the techniques described herein can be utilized. [Figure 4] Following are examples of implementations that illustrate how the techniques described herein can be utilized. [Figure 5A] Following are examples of implementations that illustrate how the techniques described herein can be utilized. [Figure 5B] Following are examples of implementations that illustrate how the techniques described herein can be utilized. [Figure 6] Following are examples of implementations that illustrate how the techniques described herein can be utilized. [Figure 7] Following are examples of implementations that illustrate how the techniques described herein can be utilized. [Figure 8] 1 shows a flowchart illustrating an example method according to implementations disclosed herein. [Figure 9] 1 illustrates an example architecture for a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] 1 illustrates an example environment in which the techniques disclosed herein can be implemented. The example environment includes multiple client devices 106. 1-N and an automated assistant 120. The automated assistant 120 is connected to the client device 106. 1-N Although illustrated in FIG. 1 as separate from the client device 106, in some implementations, all or aspects of the automated assistant 120 may be implemented as part of a 1-NFor example, the client device 1061 may implement one or more instances of one or more aspects of the automated assistant 120, and the client device 106 N Each of the automated assistants 120 may implement separate instances of one or more of those aspects of the automated assistant 120. 1-N In an implementation implemented by one or more computing devices remote from the client device 106 1-N and those aspects of the automated assistant 120 may communicate over one or more networks, such as a local area network (LAN) and / or a wide area network (WAN) (e.g., the Internet).

[0028] Client device 106 1-N These may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-car communication system, an in-car entertainment system, an in-car navigation system), a standalone interactive speaker, a smart camera, and / or a user wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0029] In some implementations, a given user may communicate with an automated assistant 120 that leverages multiple client devices that collectively form a collaborative "ecosystem" of computing devices. In some such implementations, the automated assistant 120 may be considered to "serve" that given user, for example, providing the automated assistant 120 with expanded access to resources (e.g., content, documents, etc.) whose access is controlled by the "served" user. However, for simplicity, some examples described herein will focus on users operating a single client device.

[0030] Client device 106 1-N Each of the plurality of message exchange clients 107 1-N 109 1-N Each client device 106 may run a variety of different applications, such as a corresponding one of the following: A peripheral device 1131, a peripheral scale 1132, a peripheral scale 1133, a peripheral scale 1134, a peripheral scale 1135, a peripheral scale 1136, a peripheral scale 1137, a peripheral scale 1138, a peripheral scale 1139, a peripheral scale 11311, a peripheral scale 1132, a peripheral scale 1133, a peripheral scale 1134, a peripheral scale 1135, a peripheral scale 1136, a peripheral scale 1137, a peripheral scale 1138, a peripheral scale 1139, a peripheral scale 11311, a peripheral scale 11312, a peripheral scale 113 ...

[0031] Message Exchange Client 107 1-N may take a variety of forms, and the forms may include those of the client computing device 106 1-N The client computing device 106 may vary across and / or have multiple configurations. 1-N In some implementations, the messaging client 107 may be operated in a single one of the1-N One or more of the messaging clients 107 may be in the form of a short message service ("SMS") and / or multimedia message service ("MMS") client, an online chat client (e.g., an instant messenger, Internet Relay Chat or "IRC", etc.), a messaging application associated with a social network, a personal assistant messaging service dedicated to conversations with the automated assistant 120, etc. In some implementations, the messaging client 107 1-N One or more of may be implemented via a web page or other resource rendered by a web browser (not shown) or other application of the client computing device 106.

[0032] Camera Application 109 1-N The user can view the camera 111 1-N For example, the camera application 109 1-N may provide a graphical user interface with which a user may interact to capture one or more images and / or videos. In some implementations, the camera application 109 1-N The user may interact / interface with the automated assistant 120 as described herein to view the camera 111 1-N In another implementation, the camera application 109 may be able to resolve requests related to images captured by the camera application 109. 1-N One or more of the following may be used by a user to view the camera 111 1-N The automated assistant 107 may have its own built-in functionality, separate from the automated assistant 120, that enables it to resolve requests related to images acquired by the camera 111 and / or the additional sensor 113. Additionally or alternatively, in some implementations, the messaging client 107 or any other application installed on the client device 106 may include functionality that enables the application to access data acquired by the camera 111 and / or the additional sensor 113 and perform the techniques described herein.

[0033] Camera 111 1-N The client device 1061 and the client device 106 may include a monochrome camera, a stereo camera, and / or a thermographic camera. N Although each is illustrated in FIG. 1 with only a single camera, in many implementations a client device may include multiple cameras. For example, a client device may have a front and a rear monochrome camera. Also, for example, a client device may have a stereo camera and a thermographic camera. Also, for example, a client device may have a monochrome camera and a thermographic camera. Moreover, in various implementations, the sensor data utilized in the techniques described herein may include images from multiple disparate cameras (of the same client device and / or of multiple client devices). For example, images from a monochrome camera may be utilized initially to determine that the request is not solvable, and images from a separate thermographic camera may be subsequently received (e.g., in response to a prompt) and utilized in resolving the request. Additionally, in some implementations, sensor data from other visual sensors, such as point cloud sensor data from a 3D laser scanner, may be utilized.

[0034] As described in more detail herein, the automated assistant 120 may be configured to communicate with one or more client devices 106. 1-N In some implementations, the automated assistant 120 may engage in a human-computer dialogue session with one or more users via the user interface input / output devices of the client device 106. 1-N The computer may be adapted to engage in a human-computer dialogue session with a user in response to user interface input provided by the user via one or more user interface input devices of one of the computer systems.

[0035] In some of these implementations, user interface input is explicitly directed to the automated assistant 120. For example, the messaging client 107 1-NOne of the messaging services may be a personal assistant messaging service dedicated to conversations with the automated assistant 120, and user interface input provided via the personal assistant messaging service may be automatically provided to the automated assistant 120. Also, for example, the user interface input may be provided to the messaging client 107 based on a particular user interface input indicating that the automated assistant 120 is to be invoked. 1-N For example, a particular user interface input may be one or more typed characters (e.g., @AutomatedAssistant), a user interaction with a hardware button and / or a virtual button (e.g., tap, long tap), a verbal command (e.g., "Hey, automated assistant"), and / or other particular user interface input. In some implementations, the automated assistant 120 may engage in a dialogue session in response to a user interface input even when the user interface input is not explicitly addressed to the automated assistant 120. For example, the automated assistant 120 may examine the content of the user interface input and engage in a dialogue session in response to the presence of certain terms in the user interface input and / or based on other indications. In many implementations, the automated assistant 120 may engage in interactive voice response ("IVR") such that a user may issue commands, searches, etc., and the automated assistant 120 may utilize natural language processing and / or one or more grammars to convert the utterances to text and respond accordingly.

[0036] Client Computing Device 106 1-N Each of the client computing devices 106 and the automated assistant 120 may include one or more memories for storage of data and software applications, one or more processors for accessing the data and executing the applications, and other components that facilitate communication over the network. 1-NOperations performed by one or more of and / or by automated assistant 120 may be distributed across multiple computer systems. Automated assistant 120 may be implemented, for example, as a computer program running on one or more computers at one or more locations coupled together through a network.

[0037] 1, the automated assistant 120 may include a natural language processor 122, a request engine 124, a prompt engine 126, and a request resolution engine 130 (including an attribute module 132). In some implementations, one or more of the engines and / or modules of the automated assistant 120 may be omitted, combined, and / or implemented in components that are separate from the automated assistant 120. In some implementations, the automated assistant 120 may communicate with the client device 106 during a human-computer dialogue session with the automated assistant 120. 1-N The automated assistant 120 generates response content in response to various inputs from the client device 106. The automated assistant 120 provides the response content (e.g., over one or more networks when separate from the user's client device) for presentation to the user as part of a dialogue session. For example, the automated assistant 120 may 1-N , in response to free-form natural language input provided via one of the cameras 111, in response to images acquired by one of the cameras 111, and / or in response to additional sensor data acquired by one or more of the additional sensors 113. As used herein, free-form input is input that is created by a user and is not constrained to a set of options presented for selection by the user.

[0038] As used herein, a "dialog session" may include a logically independent exchange of one or more messages between a user and the automated assistant 120 (and in some cases, other human participants in the dialog). The automated assistant 120 may distinguish between multiple dialog sessions with a user based on various signals, such as the passage of time between sessions, a change in user context (e.g., location, before / during / after a scheduled meeting, etc.) between sessions, detection of one or more intervening actions between the user and the client device other than the dialog between the user and the automated assistant (e.g., the user switches applications for a while, the user walks away from a standalone voice-activated product and then returns), locking / sleeping the client device between sessions, a change in the client device used to interface with one or more instances of the automated assistant 120, etc.

[0039] In some implementations, when the automated assistant 120 provides a prompt requesting user feedback in the form of a user interface input (e.g., spoken and / or typed input), the automated assistant 120 may pre-activate one or more components of the client device (through which the prompt is provided) that are configured to process the user interface input that will be received in response to the prompt. For example, if the user interface input is to be provided via a microphone of the client device 1061, the automated assistant 120 may provide one or more commands to cause the microphone to be pre-"opened" (thereby preventing the need to open the microphone by hitting an interface element or speaking a "hot word"), the local speech-to-text processor of the client device 1061 to be pre-activated, a communication session between the client device 1061 and a remote speech-to-text processor (e.g., a speech-to-text processor remotely provided to the automated assistant 120) to be pre-established, and / or a graphical user interface to be rendered on the client device 1061 (e.g., an interface including one or more selectable elements that may be selected to provide feedback). This may allow user interface input to be provided and / or processed more quickly than if the part had not been pre-activated.

[0040] The natural language processor 122 of the automated assistant 120 is 1-Nand generate annotated output for use by one or more other components of the automated assistant 120, such as the request engine 124, the prompt engine 126, and / or the request resolution engine 130. For example, the natural language processor 122 may process natural language free-form input generated by a user via one or more user interface input devices of the client device 1061. The generated annotated output includes one or more annotations of the natural language input and optionally one or more (e.g., all) of the terms of the natural language input. In some implementations, the natural language processor 122 includes a speech processing module configured to process speech (spoken) natural language input. The natural language processor 122 can then act on the processed speech input (e.g., based on text derived from the processed speech input). For example, the speech processing module may be a speech-to-text module that receives free-form natural language speech input in the form of a streaming audio recording and converts the speech input to text utilizing one or more speech-to-text models. For example, the client device can generate a streaming audio recording in response to signals received from a microphone on the client device while a user is speaking, and can transmit the streaming audio recording to the automated assistant for processing by the speech-to-text module.

[0041] In some implementations, the natural language processor 122 is configured to identify and annotate various types of grammatical information in the natural language input. For example, the natural language processor 122 may include a part-of-speech tagger configured to annotate terms with their grammatical roles. For example, the part-of-speech tagger may tag each term with its part of speech, such as "noun," "verb," ​​"adjective," "pronoun," etc. Also, for example, in some implementations, the natural language processor 122 may additionally and / or alternatively include a dependency parser (not shown) configured to determine syntactic relationships between terms in the natural language input. For example, the dependency parser may determine which terms modify other terms, subjects and verbs of a sentence or the like (e.g., a parse tree)—and make annotations of such dependencies.

[0042] In some implementations, the natural language processor 122 may additionally and / or alternatively include an entity tagger (not shown) configured to annotate entity references in one or more sections, such as references to people (e.g., including literary figures, celebrities, public figures, etc.), organizations, places (real and fictional), etc. In some implementations, data about the entities may be stored in one or more databases, such as in a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes representing known entities (and in some cases, entity attributes), as well as edges connecting the nodes and representing relationships between the entities. For example, a "banana" node may be connected (e.g., as a child) to a "fruit" node, which may then be connected (e.g., as a child) to a "produce" and / or "food" node. As another example, a restaurant called "hypothetical cafe" may be represented by a node that also includes attributes such as its address, the type of food served, opening hours, contact information, etc. The "Hypothetical Cafe" node, in some implementations, may be connected by edges (e.g., representing parent-child relationships) to one or more other nodes, such as a "Restaurant" node, a "Business" node, a node representing a city and / or state in which the restaurant is located, etc.

[0043] The entity tagger of the natural language processor 122 may annotate entity references at a high level of granularity (e.g., allowing identification of all references of an entity class, such as people) and / or at a low level of granularity (e.g., allowing identification of all references of a particular entity, such as a particular person). The entity tagger may rely on the content of the natural language input to resolve a particular entity, and / or may optionally communicate with a knowledge graph or other entity database to resolve a particular entity.

[0044] In some implementations, the natural language processor 122 may additionally and / or alternatively include a coreference analyzer (not shown) configured to group or "cluster" references of the same entity based on one or more contextual indications. For example, a coreference analyzer may be utilized to resolve the term "there" to "the hypothetical cafe" in the natural language input "I liked the hypothetical cafe where we ate last time."

[0045] In some implementations, one or more components of the natural language processor 122 may rely on annotations from one or more other components of the natural language processor 122. For example, in some implementations, the named entity tagger may rely on annotations from the coreference analyzer and / or the dependency parser when annotating all references to a particular entity. Also, for example, in some implementations, the coreference analyzer may rely on annotations from the dependency parser when clustering references of the same entity. In some implementations, when processing a particular natural language input, one or more components of the natural language processor 122 may determine one or more annotations using prior inputs related outside of the particular natural language input and / or other relevant data.

[0046] The request engine 124 utilizes one or more signals to determine when a request is related to an object in the environment of the client device associated with the request. For example, the request engine 124 can determine a request related to an object in the environment of the client device 1061 based on natural language input provided via the client device 1061, images acquired by the camera 1111, additional sensor data from the additional sensor 1131, and / or a detected location and / or other contextual attributes of the client device 1061.

[0047] As one example, the request engine 124 can determine such requests based on natural language input (e.g., oral speech input) provided by a user via the client device 1061. For example, the request engine 124 can determine that an utterance, such as "what is this," "where can I buy this," "how much does this cost," "how much does this thing weigh," "tell me more about it," etc., relying on annotations from the natural language processor 122, may be related to an object in the environment. In some implementations, the request engine 124 can determine such requests based on an utterance that includes a substitute form (e.g., an ambiguous pronoun such as "this," "this thing," "it," "it") that cannot be coreference-resolved (e.g., by the natural language processor 122) to an antecedent in the preceding natural language input. In other words, the request engine 124 can assume that an utterance is related to an environmental object because of a request that references a substitute form that cannot be coreference-resolved to an antecedent in the preceding natural language input. In some implementations, the request engine 124 can cause images and / or other sensor data to be captured by the client device 1061 in response to determining that the utterance is related to an environmental object. For example, the request engine 124 can provide one or more commands to the camera 1111 and / or camera application 1091 to cause one or more images to be captured. In some of these implementations, the request engine 124 can cause the images to be captured without requiring a user of the client device 1061 to select a "capture image" interface element, provide a verbal "capture image" command, and / or otherwise physically interact with the client device 1061 to cause the images to be captured. In some implementations, confirmation from the user can be required prior to capturing the images. For example, the request engine 124 can cause a user interface output to be provided stating "I need to capture images to answer your request," and the images to be captured only in response to a positive user response (e.g., "Okay").

[0048] As another example, the request engine 124 can additionally or alternatively determine such a request based on an image, audio recording, and / or other sensor data being acquired via the client device 1061. For example, the request engine 124 can determine such a request based on an utterance being provided in conjunction with (e.g., immediately before, immediately after, and / or during) the acquisition of at least one image via the camera 1111. For example, an utterance of "what's this" within X seconds of acquiring an image via the camera 1111 in response to a user interface input. The image can be acquired via the camera 1111 in association with an assistant application of the client device 1061 (e.g., an application dedicated solely to interacting with the automated assistant 120) or in association with another application of the client device 1061, such as a camera application, a chat application, etc. Thus, in some implementations, the request engine 124 can determine such a request based on an image and / or an utterance acquired from any of a number of disparate applications. Also, for example, the request engine 124 can determine such a request based on sensor data being acquired, and can determine such a request independent of any utterance. For example, the request engine 124 can determine such a request based on a user acquiring an image in a certain context (e.g., when location data of the client device 1061 indicates that the user is at a retail location). As yet another example, the request engine 124 can determine such a request in response to a user causing a recording to be acquired via a microphone of the additional sensor 1131. For example, a user can cause an audio recording to be acquired, where the audio recording captures sounds being made by the user's vacuum cleaner, and then provide the utterance "Why is the vacuum cleaner making this noise?"

[0049] As another example, the request engine 124 can additionally or alternatively determine such a request based on an interface element selected by the user and / or based on images and / or other sensor data being acquired through a particular interface. For example, after acquiring an image via the camera application 1091, a "Learn More" graphical interface element can be presented as a suggestion via the camera application 1091 (e.g., based on output from the automated assistant 120), and selection of the graphical interface element can be interpreted by the request engine 124 as a request for additional information about the object captured by the image. Also, for example, if the user utilizes the messaging client 1071 and / or other applications specifically tailored for the automated assistant 120 (e.g., client applications dedicated to interacting with the automated assistant 120) to acquire images and / or other sensor data, such acquisition can be interpreted by the request engine 124 as a request related to the object captured by the sensor data.

[0050] When a request is associated with an object in the client device's environment associated with the request (e.g., as determined by the request engine 124), the request resolution engine 130 attempts to resolve the request. In attempting to resolve the request, the request resolution engine 130 may leverage natural language input associated with the request (if any), images and / or other sensor data associated with the request, and / or other content. As described in more detail herein, the request resolution engine 130 may interface with one or more agents 146, image processing engine 142, and / or additional processing engines 144 in determining whether the request is resolvable.

[0051] If the request resolution engine 130 determines that the request can be resolved, the request resolution engine 130 can interact with one or more agents 146 in resolving the request. The agents 146 can include one or more so-called first party (1P) agents controlled by the same party that controls the automated assistant 120 and / or can include one or more so-called third party (3P) agents controlled by a separate party. As one example, the agent 146 can include a search system (a 1P search system or a 3P search system), and the request resolution engine 130 can resolve the request by submitting a search to the search system, receiving response content (e.g., a single "answer"), and providing the response content for presentation via the client device 1061.

[0052] If the request resolution engine 130 determines that the request is not resolvable, the request resolution engine 130 may cause the prompt engine 126 to determine one or more prompts to provide for presentation via the client device 1061. The prompts determined by the prompt engine 126 may instruct the user to acquire additional sensor data (e.g., images, audio, temperature sensor data, weight sensor data) for the object and / or to move the object (and / or other objects) to enable acquisition of additional sensor data for the object. The prompts may additionally or alternatively ask the user to provide user interface inputs targeted to unresolved attributes of the object.

[0053] The request resolution engine 130 may then utilize additional sensor data and / or user interface input received in response to the prompts to again attempt to resolve the request. If the request is still not solvable, the request resolution engine 130 may cause the prompt engine 126 to determine one or more additional prompts to provide for presentation via the client device 1061. The additional sensor data and / or user interface input received in response to such additional prompts may then be utilized to again attempt to resolve the request. This may continue until the request is resolved, a threshold number of prompts is reached, a threshold time limit has elapsed, and / or one or more other criteria are achieved.

[0054] The request resolution engine 130 optionally includes an attribute module 132 that determines various attributes for the object indicated by the request. As described herein, the attributes can be utilized in determining whether the request is solvable and / or in determining a prompt to provide if the request is determined to be unsolvable. The attribute module 132 can interface with the image processing engines 142 and / or the additional processing engines 144 in determining the various attributes. For example, the attribute module 132 can provide acquired images to one or more of the image processing engines 142. The image processing engine 142 can perform image processing on such images and, accordingly, provide attributes (if any) that are resolvable based on the acquired images. Also, for example, the attribute module 132 can provide other acquired sensor data to one or more of the additional processing engines 144. The additional processing engines 144 can perform processing on such images and, accordingly, provide attributes (if any) that are resolvable based on the acquired sensor data. For example, one of the additional processing engines 144 may be configured to process the audio data to determine one or more attributes of the audio data, such as entities present in the audio data (e.g., particular objects that are the source of sounds in the audio data) and / or other attributes of the audio data (e.g., the number and / or frequency of “beeps” in the audio data).

[0055] In some implementations, the request resolution engine 130 determines that the request is not resolvable based on a determination that the attributes (if any) resolved by the attribute module 132 fail to define the object with a target specificity. In some implementations, the target specificity for the object can be a target classification of the object in a taxonomy. For example, a target classification of a car can be a classification at a level that defines the make and model of the car, or at a level that defines the make, model, and year of the car. Such a target classification can be optionally stored in and accessed from the resource database 148. In some implementations, the target specificity for the object can be defined with respect to one or more items to be defined, where the items for the object can depend on the classification (general or specific) of the object. For example, for a bottle of wine, the items can be defined for a particular brand, wine type, and / or vintage, and the target specificity is the resolution of the attributes for all of those items. Such items to be defined for the classification of the object can be optionally stored in and accessed from the resource database 148. In some implementations, the target specificity can be determined based on an initial natural language input provided by the user, feedback provided by the user, the interaction history of the user and / or other users, and / or location and / or other contextual signals.

[0056] In some implementations, the prompt engine 126 determines a prompt that provides a guide to further input that would enable the resolution of a request related to an environmental object. In some of those implementations, the prompt is determined based on one or more attributes for the object resolved by the attribute module 132. For example, a classification attribute for the environmental object can be determined by the attribute module 132, such as a classification attribute resolved by one of the image processing engines 142 based on the acquired image. For example, the prompt engine 126 can determine a prompt for a car classification (e.g., "take a picture of the back of the car") that is specialized for the "car" classification. Also, for example, the prompt engine 126 can determine a prompt for a "jacket" classification (e.g., "take a picture of the logo") that is specialized for the "jacket" classification. Also, for example, in some implementations, a classification attribute can be associated with one or more items to be defined. In some of those situations, a prompt can be generated based on an item that is not defined by the attributes (if any) already determined by the attribute module 132. For example, if a vintage item for a bottle of wine has not yet been defined, the prompt can be "take a picture of the vintage" or "how old is it?"

[0057] Additional description of the various components of Figure 1 will now be provided with reference to the examples of Figures 2A-8. Although some of the components of Figure 1 are not shown in Figures 2A-8, they are referenced in the following discussion when describing certain examples of the functionality of the various components.

[0058] 2A and 2B illustrate one example of how a user (not shown) can interact with an instance of an automated assistant (120 in FIG. 1) operating on and / or in conjunction with a client device 1061 according to an implementation described herein. The client device 1061 takes the form of a smartphone or tablet computer that includes a touch screen 160 and at least one camera 1111 (front and / or rear). Rendered on the touch screen 160 is a graphical user interface associated with camera functionality (e.g., camera application 109 in FIG. 1 and / or other application including an electronic viewfinder) that renders, for example, in real time, sensor data acquired by the camera 1111. The graphical user interface includes a user input field 164 and one or more graphical elements 166 that may be operable to control the operation of the camera 1111. 1、2 For example, a first graphical element 1661 may be operable to switch between the front and rear cameras, and a second graphical element 1662 may be operable to capture an image (or video (capturing multiple images in succession) depending on the settings) using the camera 1111. Other graphical elements not shown in Figures 2A and 2B may be operable to perform other actions, such as changing camera settings, switching between image capture and video capture modes, adding various effects, etc.

[0059] The user input field 164 may be operable by a user to provide various inputs, such as free-form natural language inputs, that can be provided to the automated assistant 120. The free-form natural language inputs can be typed user interface inputs (e.g., via a virtual keyboard, not shown) and / or voice inputs provided by the user (e.g., by clicking a microphone icon on the right or by speaking a "hot word"). For example, when aspects of the automated assistant 120 are implemented remotely from the client device 1061 and voice inputs are provided via the user input field 164, a streaming version of the voice inputs can be sent over one or more networks to the automated assistant 120. In various implementations, the voice inputs provided via the user input field 164 can be converted to text, for example, at the client device 1061 and / or remotely (e.g., at one or more cloud-based components of the automated assistant 120).

[0060] The camera 1111 in FIG. 2A has captured a bottle of wine 261 in its field of view. As a result, a drawing 261A of the bottle of wine 261 appears on the touch screen 160 as part of the electronic viewfinder described above. In various implementations, the user may invoke the automated assistant 120, for example, by tapping in the user input field 164 or by speaking an invocation phrase such as "hey, automated assistant." Once the automated assistant 120 is invoked, the user speaks or types a natural language input such as "how much is this?" Additionally or alternatively, the user may provide a single natural language input (e.g., "hey, automated assistant, how much is this?") that invokes the automated assistant 120 and also provides a natural language input. In some implementations, the automated assistant 120 may be automatically invoked whenever a camera application is active on the client device, or may be invoked in response to a different invocation phrase that does not otherwise invoke the automated assistant 120. For example, in some implementations, the automated assistant 120 may be invoked when the camera application 109 is active (ie, being interacted with by a user, presented as a graphical user interface, etc.).

[0061] The request engine 124 of the automated assistant 120 can determine that the natural language input, "how much does this cost," is related to an object in the environment of the client device 1061. In some implementations, the request engine 124 can cause an image of the environment to be captured in response to such a determination (e.g., an image capturing drawing 261A of FIG. 2A). In some other implementations, the user can cause an image to be captured (e.g., through selection of the second graphical element 1662) and provide the natural language input in conjunction with (e.g., immediately before, during, or after) the capture of the image. In some of these implementations, the request engine 124 can determine that the request is related to an object in the environment based on both the natural language input and the user's capture of the image.

[0062] The request engine 124 provides an indication of the request to the request resolution engine 130. The request resolution engine 130 attempts to resolve the request using the natural language input and the acquired image. For example, the request resolution engine 130 can provide the acquired image to one or more of the image processing engines 142. The image processing engine 142 can process the acquired image to determine classification attributes of "wine bottle" and return the classification attributes to the request resolution engine 130. The request resolution engine 130 can further determine that the request is for a "cost" action (e.g., based on the output provided by the natural language processor 122, which is based on "how much does this cost?"). Furthermore, for a "cost" action on an object that is a "wine bottle", the request resolution engine 130 can determine that attributes need to be resolved for brand, wine type, and vintage items to resolve the request. For example, the request resolution engine 130 can determine those items based on looking up definition items for a "cost" action for the "wine bottle" classification in the resource database 148. Also, for example, the request resolution engine 130 can determine those items based on what items are indicated as needed by an agent of the agents 146 that can handle the request (e.g., a “wine cost” agent, a more general “liquor cost” agent, or an even more general “search system” agent). For example, an agent can be associated with a “wine cost” intent and define the required slots / items of “brand”, “wine type”, and “vintage” for that “wine cost” intent.

[0063] The request resolution engine 130 may further determine that the request cannot be resolved based on the given natural language input and image processing of the acquired image. For example, the request resolution engine 130 may determine that the brand, wine type, and vintage are not resolvable, e.g., they are not resolvable from the natural language input, and the image processing engine 142 may have provided only the "wine bottle" classification attribute (e.g., they could not resolve more detailed attributes because the labels are not visible in the acquired image corresponding to drawing 261A of FIG. 2A).

[0064] Based on the request being unresolvable, the prompt engine 126 determines and provides the prompt 272A, "Can you take a picture of the label?" The prompt 272A is illustrated in FIG. 2A as an audible prompt (e.g., provided via the speaker of the client device 1061). However, in other implementations, a graphical prompt may additionally and / or alternatively be provided. When an audible prompt is provided, a text-to-speech processor module can optionally be utilized to convert the text prompt to an audible prompt. For example, the prompt engine 126 can include a text-to-speech processor to convert the text prompt to an audio format (e.g., streaming audio) and provide the audio format to the client device 1061 for audible presentation via the speaker of the client device 1061. The prompt engine 126 can determine the prompt 272A based on the "wine bottle" classification attributes determined by the request resolution engine 130 and / or based on the brand, wine type, and vintage items being unresolved by the request resolution engine 130. For example, resource database 148 may define that a prompt such as prompt 272A should be provided for the “Wine Bottles” classification and / or for open items of brand, wine type, and / or vintage (e.g., the prompt (or a portion thereof) may be stored in association with the “Wine Bottles” classification and / or the open items).

[0065] In FIG. 2B, an additional image is captured by the camera 1111, where the additional image captures the label of the bottle of wine 261. For example, the additional image can capture an image that corresponds to the drawing 261B of the bottle of wine 261 illustrated in the electronic viewfinder of FIG. 2B. To capture such an image, a user can reposition the bottle of wine 261 (e.g., rotate it so that the label is visible and / or move it closer to the camera 1111), reposition the camera 1111, and / or adjust the zoom (hardware and / or software) and / or other characteristics of the camera 1111 (e.g., via a "pinch out" gesture on the touch screen 160). In some implementations, the additional image can be captured in FIG. 2B in response to a user's selection of the second graphical element 1662. In some other implementations, the additional image can be captured automatically.

[0066] The request resolution engine 130 leverages the additional images to determine additional attributes of the object. For example, the request resolution engine 130 can provide the captured additional images to one or more of the image processing engines 142. Based on the image processing, one or more of the image processing engines 142 can determine the text attributes "Hypothetical Vineyard", "Merlot" and "2014" (e.g., using OCR) and return such text to the request resolution engine 130. Additionally or alternatively, one or more of the image processing engines 142 can provide a detailed classification that specifically identifies the 2014 Merlot of the Hypothetical Vineyard. As described herein, in some implementations, the request resolution engine 130 can invoke only a subset of the available image processing engines for sensor data (e.g., images) captured in response to a prompt to determine additional attributes based on such sensor data. For example, the request resolution engine 130 may provide additional images only to an OCR image processing engine of the engine 142 based on the prompt 272A (FIG. 2A) being modified to result in a subsequent image for which attributes will be derived utilizing OCR. Also, for example, the request resolution engine 130 may not provide additional images to a general classification engine of the engine 142 based on the general classification already being resolved. In some of these implementations, this may save various computational resources. This may be particularly beneficial, for example, when one or more of the image processing engines 142 are implemented in the client device 1061.

[0067] The request resolution engine 130 can then determine that the request is resolvable based on the additional attributes. For example, the request resolution engine 130 can submit to one of the agents 146 an agent query that is based on the natural language input and the additional attributes. For example, the request resolution engine 130 can submit an agent query of "cost of vineyard A cabernet sauvignon 2012" and / or a structured agent query such as {intent="wine_cost"; brand="vineyard a"; type="merlot"; vintage="2012"}. Additional content can be received from the agent in response to the agent query, and at least a portion of the additional content can be provided for presentation to the user. For example, the output 272B of FIG. 2B can be presented based on the additional content. The output 272B identifies a price range and also asks whether the user would like to see a link where the user can purchase a bottle of wine. If the user responds affirmatively to the output 272B (e.g., further voice input of "yes"), a link for purchase can be displayed via the interface of FIG. 2B and / or a separate interface. Both the price range and the links may be based on additional content received in response to the agent query. Output 272B is illustrated in FIG. 2B as an audible output (e.g., provided through a speaker on client device 1061). However, in other implementations, a graphical output may additionally and / or alternatively be provided.

[0068] 3 illustrates another example of how a user (not shown) can interact with an instance of an automated assistant (120 in FIG. 1, not shown in FIG. 2) operating on and / or in conjunction with a client device 1061 according to implementations described herein. FIG. 3 is similar to FIG. 2A, and like numbers refer to like parts. In FIG. 3, camera 1111 has captured a bottle of wine 261 in its field of view, and the captured image and drawing 261A of the bottle of wine 261 are the same as those in FIG. 2A.

[0069] However, while in FIG. 3 the user provided the natural language input "Text Bob a picture of this," the user instead provided the natural language input in FIG. 2A of "How much does this cost?"

[0070] The request engine 124 of the automated assistant 120 can determine that the natural language input "Text Bob a picture of this" relates to an object in the environment of the client device 1061. In some implementations, the request engine 124 can cause an image of the environment to be captured in response to such a determination (e.g., image capturing drawing 261A of FIG. 3). In some other implementations, the user can cause the image to be captured (e.g., through selection of the second graphical element 1662) and provide the natural language input in conjunction with (e.g., immediately before, during, or after) the capture of the image.

[0071] The request engine 124 provides an indication of the request to the request resolution engine 130. The request resolution engine 130 attempts to resolve the request using the natural language input and the captured image. In the situation of FIG. 3, the request resolution engine 130 determines that the request (send the photo to Bob) can be resolved based on the image itself. As a result, the request resolution engine 130 can resolve the request without prompting the user to take additional images (or provide other additional information related to the bottle of wine) and / or without providing the image for processing by the image processing engine 142. For example, the request resolution engine 130 can resolve the request by simply texting the photo to a contact of the user named "Bob." The request resolution engine 130 can optionally provide a "Sent" output 372A to inform the user that the request has been resolved.

[0072] Thus, Figure 3 provides an example of how a prompt that is modified to enable determination of additional object attributes can optionally be provided only when the request is determined to be not resolvable based on the initial sensor data and / or initial user interface input. In Figure 3, it is noted that if the user instead provided a natural language input of "Text Bob the name of this wine," a prompt would be provided (as the name of the wine would likely not be resolvable based on an image matching drawing 261A of Figure 3).

[0073] Thus, in these and other schemes, whether such a prompt is provided is based on whether the request is resolvable, which in turn is based on the specificity of the request. For example, "text Bob a picture of this" does not require any attributes of the object in the image to be known, while "text Bob the name of this wine" requires the name of the wine bottle to be known. It is also noted that in some implementations, the specificity may be based on other factors in addition to or instead of the natural language input provided with the request. For example, in some implementations, the natural language input may not be provided in conjunction with the captured image (or other captured sensor data). In those implementations, the specificity may be based on the location of the client device 1061, based on resolved attributes of the image, and / or based on other factors. For example, if the user captures an image of an object at a retail location (e.g., a grocery store), a "cost comparison" or similar request of high specificity may be inferred. On the other hand, if the user captures an image of the object at a park or other non-retail location, a request cannot be inferred (e.g., the image can only be memorized) or a request of low (or no) specificity may be inferred.

[0074] FIG. 4 illustrates another example of how a user (not shown) can interact with an instance of an automated assistant (120 in FIG. 1, not shown in FIG. 2) operating on and / or in conjunction with a client device 1061 according to implementations described herein. The interface of FIG. 4 is similar to that of FIGS. 2A, 2B, and 3, and like numbers refer to like parts. In FIG. 4, camera 1111 has captured a bottle of wine 261 in its field of view. The captured image and drawing 261C of bottle of wine 261 differ from drawing 261A (FIG. 2A, FIG. 3) and 261B (FIG. 2B). Drawing 261C captures most of the label of bottle of wine 261, but is missing part of "Vineyard" and is missing the "4" in "2014."

[0075] 4, the user has provided the natural language input "order a case of this." The request engine 124 can determine that the natural language input is a request related to an object in the environment of the client device 1061 and provide an indication of the request to the request resolution engine 130.

[0076] The request resolution engine 130 attempts to resolve the request using the natural language input and the captured image. For example, the request resolution engine 130 can provide the captured image to one or more of the image processing engines 142. Based on the image processing, the image processing engine 142 can determine a classification attribute of "wine bottle", a brand attribute of "hypothetical vineyard" (e.g., based on the observable "hypothetical" and the observable "vin"), and a type attribute of "merlot". However, the request resolution engine 130 can determine that the attributes for the required "vintage" item are not resolvable (e.g., a specific vintage cannot be resolved with a high enough confidence). As a result of the "vintage" item being unresolvable and required for the "order a case" request, the prompt engine 126 determines and provides prompt 472A: "Okay, what vintage of merlot from hypothetical vineyard would you like?" Prompt 472A is illustrated in FIG. 4 as an audible prompt, but can be graphical in other implementations. The prompt engine 126 can determine the prompt 472A based on the unresolved item (vintage) and based on the resolved attributes (by including a reference to the determined attributes for brand and type).

[0077] Prompt 472A asks the user to provide further natural language input (e.g., voice input) that can be leveraged to resolve attributes for the vintage item. For example, the user may respond to prompt 472A with the voice input of "2014," and "2014" may be leveraged as an attribute for the vintage item. In some implementations, the automated assistant 120 may activate the microphone of the client device 1061 in response to prompt 472A in anticipation of the further voice input. Alternatively, prompt 472A may be addressed by entering a second image (e.g., 261B in FIG. 2B) in which the complete era "2014" is visible.

[0078] The request resolution engine 130 can then leverage the additional attribute of "2014" and the previously determined attributes to resolve the request. For example, the request resolution engine 130 can submit an agent query to one of the agents 146 to order a case of "Hypothetical Vineyard's 2014 Merlot." Additional content can optionally be received from the agent in response to the agent query (e.g., confirmation of the order, a total price, and / or an expected delivery date), and at least a portion of the additional content can optionally be provided for presentation to the user.

[0079] 5A and 5B illustrate another example of how a user (not shown) can interact with an instance of an automated assistant (120 in FIG. 1, not shown in FIG. 2) operating on and / or in conjunction with a client device 1061 according to implementations described herein. The interface of FIG. 5A is similar to that of FIGS. 2A, 2B, 3, and 4, and like numbers refer to like parts.

[0080] In Figure 5A, the camera 1111 has captured a quarter 561, and in particular, the "back" side of the Kentucky quarter. In Figure 5A, the user has provided natural language input of "tell me more about this." The request engine 124 can determine that the natural language input is a request related to the captured image and provides an indication of the request to the request resolution engine 130.

[0081] The request resolution engine 130 attempts to resolve the request using the natural language input and the captured image. For example, the request resolution engine 130 can provide the captured image to one or more of the image processing engines 142. Based on the image processing, the request resolution engine 130 can determine that it is a "2001 Kentucky Quarter." For example, the request resolution engine 130 can determine that it is a "2001 Kentucky Quarter" based on one of the image processing engines 142 refining it as a "2001 Kentucky Quarter." Also for example, the request resolution engine 130 can additionally or alternatively determine that it is a "2001 Kentucky Quarter" based on one of the image processing engines 142 classifying it as a "quarter" and based on another one of the image processing engines 142 recognizing the text "Kentucky" and "2001" in the image. The request resolution engine 130 may further determine additional text and / or additional entities present on the quarter coin based on providing the acquired image to the image processing engines 142. For example, OCR processing by one of the processing engines 142 may also identify the text "My Old Kentucky Home" and / or image processing may identify the "house" on the quarter coin as the "My Old Kentucky Home" house.

[0082] As illustrated in FIG. 5B , the automated assistant 120 initially provides an output 572A of “It’s a 2001 Kentucky quarter.” Such an output 572A can be initially provided by the automated assistant 120 based on, for example, determining that the target specificity for the “quarter” classification is “year” and “state.” In response to the initial output 572A, the user provides a natural language input 574A of “No, the location on the back.” Based on the natural language input 574A, the request resolution engine 130 determines an adjusted target specificity of the particular venue / location cited by the quarter. In other words, the request resolution engine 130 adjusts the target specificity based on the feedback provided by the user in the natural language input 574A. In response, the request resolution engine 130 attempts to resolve the request at the adjusted target specificity. For example, the request resolution engine 130 can determine attributes to be resolved from the image related to “venue,” such as text and / or entities related to “My Old Kentucky Home.” If such attributes are not resolved based on the captured image, a prompt may be provided to allow the user to provide a user interface input indicating the "venue" and / or to request that additional images of the "venue" be captured.

[0083] However, in the example of FIG. 5B, the request resolution engine 130 has resolved the attributes of "My Old Kentucky Home" based on a previously acquired image. Thus, in response to user input 574A, the request resolution engine 130 can resolve the request with adjusted specificity and generate a response 572B based on the "My Old Kentucky Home" attributes. For example, the request resolution engine 130 can issue a search request for "My Old Kentucky Home" and receive response 572B and additional search results in response to the search request. The request resolution engine 130 can provide response 572B as output along with selectable option 572C that can be selected by the user to cause the additional search results to be displayed.

[0084] In some implementations, the target specificity for the "quarterly quarter" classification may be adapted to the user and / or other users based at least in part on the interactions of Figures 5A and 5B. For example, based on such interactions and / or similar interaction history, a learned target specificity of "specific place / location on quarterly quarter" for the "quarterly quarter" classification may be determined. Subsequent requests related to acquired images of quarterly quarters may then be adapted in light of such determined specificity. In these and other schemes, user feedback provided via interactions with the automated assistant 120 may be leveraged to learn when requests are resolved and / or to learn appropriate target specificities for various future requests.

[0085] 6 illustrates another example scenario in which the disclosed techniques may be utilized. In FIG. N The client device 106 N The client 106 takes the form of a stand-alone interactive speaker that enables the user 101 to engage in verbal human-computer dialogue with an automated assistant 120 running on and / or in conjunction with the client 106. N The client device 106 may also include one or more microphones (not shown in FIG. 6) for detecting verbal input from the user 101. N a camera 111 configured to acquire images; N Although not shown in FIG. 6, in some implementations, the client device 106 N may also include a display device.

[0086] In this example, the user 101 provides speech input 674A of "Why is my robot vacuum making this noise?"

[0087] The request engine 124 of the automated assistant 120 receives the natural language input "Why is my robot vacuum making this noise?" from the client device 106. NThe request engine 124 may determine that the sound associated with the object in the environment and the sound being made by the object is related to the object. In some implementations, the request engine 124 may cause audio (e.g., audio attempting to capture "that sound") and / or images to be acquired in response to such a determination.

[0088] The request engine 124 provides an indication of the request to the request resolution engine 130, which attempts to resolve the request using the natural language input and the captured audio and / or captured images. For example, the request resolution engine 130 can provide the captured audio to one or more of the additional processing engines 144. The additional processing engine 144 can analyze the audio and determine an attribute of "three consecutive beeps" from the audio. In other words, the audio captures the robot vacuum cleaner making three consecutive beeps.

[0089] The request resolution engine 130 may then attempt to resolve the request, for example, by submitting to a search system agent an agent query that is based on the input 674A and the resolved audio attributes (e.g., the agent query "what does three consecutive beeps mean for a robot vacuum cleaner"). In response to the agent query, the search system may not be able to return any answer or may not return an answer with a confidence level that meets a threshold. In response, the request resolution engine 130 may determine that the request cannot be resolved.

[0090] Based on the request being unresolvable, the prompt engine 126 determines and provides a prompt 672A to the client device 106, such as "Can you hold the vacuum up to the camera?" N 111 in an attempt to acquire an image that can be utilized to resolve additional attributes associated with the vacuum cleaner robot 661. N requests that the robot keep vacuum cleaner robot 661 (which is making "this kind of noise") within its field of vision.

[0091] User first selects camera 111 N An image may be captured after the user lifts the vacuum cleaner to the ground. However, the request resolution engine 130 may determine that it cannot resolve attributes from the image and / or that any resolved attributes are still insufficient to resolve the request. In response, another prompt 672B is provided instructing the user to "move it and get another image."

[0092] Additional images may be obtained after the user 101 moves the vacuum cleaner further, and the request resolution engine 130 may be able to resolve the request based on attributes resolved from the additional images. For example, the additional images may have enabled the determination of a "hypothetical vacuum cleaner" brand and / or a "3000" model of the vacuum cleaner, allowing an agent query of "what does three consecutive beeps mean for a hypothetical vacuum cleaner 3000" to be created by the request resolution engine 130. The agent query may be submitted to a search system agent, and a high-confidence answer is returned accordingly. A further output 672C may be based on the high-confidence answer. For example, the high-confidence answer may match a further output 672C of "three consecutive beeps mean the bin is full."

[0093] Figure 7 is similar to Figure 6. In particular, the natural language input 774A provided by the user 101 in Figure 7 is the same as the natural language input 674A in Figure 6. Moreover, the automated assistant 120 may similarly determine that it is unable to resolve the request based on the natural language input 774A and based on any captured initial audio and / or images. In Figure 7, a prompt 772A prompts the user to select the camera 111. NInstead of requiring the user 101 to hold up the vacuum cleaner 661 in front of the camera, the user 101 may take a picture with his or her separate smartphone. For example, as illustrated in FIG. 7, the user 101 may utilize the client device 1061 (which may be a smartphone) to capture an image of the vacuum cleaner 661 while it is on the ground. The client device 1061 and the client device 106 N can be linked to the same user via the automated assistant 120 as described herein. The automated assistant 120 can leverage images captured by the client device 1061 (in response to prompt 772A) to resolve the request. Output 772B is the same as output 672C, and includes attributes derived from the image captured by the client device 1061 as well as the attributes of the image captured by the client device 1061. N In these and other ways, the automated assistant 120 can leverage natural language input and / or sensor data from multiple devices of the user in resolving the request.

[0094] 8 is a flow chart illustrating an example method 800 according to an implementation disclosed herein. For convenience, the operations of the flow chart are described with respect to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of the automated assistant 120. Moreover, although the operations of the method 800 are illustrated in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0095] At block 802, the system determines a request related to an object in the environment of the client device. In some implementations, the system can determine the request based on natural language input provided through an automated assistant interface of the client device. The natural language input can include voice input and / or typed input. In some implementations, the system can additionally or alternatively determine the request based on images, audio recordings, and / or other sensor data being acquired through the client device. In some implementations, the system can additionally or alternatively determine the request based on an interface element selected by a user and / or based on images and / or other sensor data being acquired through a particular interface.

[0096] At block 804, the system receives initial sensor data from the client device. For example, the system can receive an initial image captured by a camera of the client device. In some additional or alternative implementations, the system additionally or alternatively receives initial sensor data from additional client devices, such as additional client devices associated with a user that is also associated with the client device.

[0097] At block 806, the system resolves initial attributes of the object based on the initial sensor data received at block 804. For example, if the initial sensor data includes an initial image, the system can resolve the initial attributes based on providing the initial image to one or more image processing engines and receiving the initial attributes from the engines in response.

[0098] At block 808, the system determines whether the request of block 802 is resolvable based on previously resolved attributes of the object. In some implementations, the system determines whether the request is resolvable based on whether previously resolved attributes define the object with a target specificity. In some of these implementations, the target specificity is a target classification of the object in a taxonomy and / or is defined with respect to one or more items to be defined, where the items for the object can depend on the classification (general or specific) of the object. In some of these implementations, the target specificity can additionally or alternatively be determined based on an initial natural language input provided by the user, feedback provided by the user, the interaction history of the user and / or other users, and / or location and / or other context signals.

[0099] If at block 808 the system determines that the request is resolvable, then the system proceeds to block 816 to resolve the request. In some implementations, blocks 808 and 816 may occur in coordination (e.g., the system may determine that the request is resolvable or not resolvable based on attempting to resolve the request at block 816).

[0100] If, at block 808, the system determines that the request is not resolvable, the system proceeds to block 810 to provide a prompt for presentation via the client device or additional client devices. The prompt may, for example, prompt the user to obtain additional sensor data (e.g., take additional images), move the object, and / or provide user interface input (e.g., natural language input). In some implementations, the system determines the prompt based on one or more attributes (e.g., classification attributes) for the resolved object and / or based on items not defined by already resolved attributes.

[0101] At block 812, after providing the prompt, the system receives further input. The system receives further input via the client device or additional client devices. The further input can include additional sensor data (e.g., additional images) and / or user interface input (e.g., natural language input).

[0102] At block 814, the system resolves the additional attributes based on the further input. The system then returns to block 808. For example, if the additional sensor data includes additional images, the system can resolve the additional attributes based on providing the additional images to one or more image processing engines and receiving the additional attributes from the engines in response.

[0103] In some implementations, block 816 includes sub-blocks 817 and 818. In sub-block 817, the system generates the additional content based on one or more of the resolved attributes. In some implementations, the system generates the additional content based on creating a request based on one or more of the resolved attributes, submitting the request to an agent, and receiving the additional content from the agent in response to the request. In some of those implementations, the system can select an agent based on the resolved attributes (e.g., attributes resolved from the sensor data and / or natural language input). In block 818, the system provides the additional content for presentation via the client device or additional client devices. For example, the system can provide the additional content for audible and / or graphical presentation. For example, the system can provide the additional content for audible presentation by providing streaming audio including the additional content to the client device or additional client devices.

[0104] 9 is a block diagram of an example computing device 910 that may be optionally utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, the automated assistant 120, and / or other components may consist of one or more components of the example computing device 910.

[0105] Computing device 910 typically includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include, for example, a storage subsystem 924, including a memory subsystem 925 and a file storage subsystem 926, user interface output devices 920, user interface input devices 922, and a network interface subsystem 916. The input / output devices enable user interaction with computing device 910. The network interface subsystem 916 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0106] The user interface input devices 922 may include a keyboard, a mouse, a trackball, a pointing device such as a touchpad or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and means of inputting information into the computing device 910 or onto a communications network.

[0107] The user interface output devices 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image (e.g., an augmented reality display associated with "smart" glasses). The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and means for outputting information from the computing device 910 to a user or to another machine or computing device.

[0108] Storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 924 may include logic to implement various components shown in FIG. 1 as well as perform selected aspects of the method of FIG.

[0109] These software modules are generally executed by the processor 914 alone or in combination with other processors. The memory 925 used in the storage subsystem 924 may include several memories including a main random access memory (RAM) 930 for storing instructions and data during program execution and a read-only memory (ROM) 932 in which fixed instructions are stored. The file storage subsystem 926 may provide persistent storage for program and data files and may include hard disk drives, floppy disk drives as well as associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of an implementation may be stored in the storage subsystem 924 by the file storage subsystem 926 or in other machines accessible by the processor 914.

[0110] The bus subsystem 912 provides a mechanism for allowing the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is illustrated generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0111] The computing device 910 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 910 shown in Figure 9 is intended merely as a specific example for purposes of illustrating some implementations. Many other configurations of the computing device 910 are possible, having more or fewer components than the computing device shown in Figure 9.

[0112] In situations where certain implementations described herein may collect or use personal information about the user (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, and user activity and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how information is collected, stored, and used about the user. That is, the systems and methods described herein collect, store, and / or use user personal information only upon explicit authorization to do so from the associated user.

[0113] For example, a user is given control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. Each user about whom personal information is to be collected is presented with one or more options that allow control of the collection of information related to that user, giving permission or authorization as to whether the information is to be collected and what portions of the information are to be collected. For example, a user may be given one or more such control options through a communication network. In addition, some data may be processed in one or more ways before it is stored or used, such that personally identifiable information is removed. As one example, a user's identification information may be processed such that personally identifiable information cannot be determined. As another example, a user's geographic location may be generalized to a larger area such that a user's specific location cannot be determined. Furthermore, some processing according to the present disclosure may be performed exclusively on the user's device such that the data and related processing are not shared to a network or other third party devices or services, and may be encrypted and / or password protected for additional privacy and security. [Explanation of symbols]

[0114] 101 users 106 1-N Client Device 107 1-N Conversation Client 109 1-N Camera application 111 1-N camera 113 1-N Additional Sensors 120 Automated Assistants 122 Natural Language Processors 124 Request Engine 126 Prompt Engine 130 Requirement Resolution Engine 132 Attribute Module 142 Image Processing Engine 144 additional processing engines 146 Agent 148 Resource Database 160 Touch Screen 164 User Input Fields 166 1,2 Graphical Elements 261 Bottle of Wine 261A Drawing 261B Drawing 261C Drawing 272A Prompt 272B Output 372A output 472A Prompt 572A Output 572B Response 572C Available Options 574A Natural Language Input 661 Vacuum Cleaner Robot 672A Prompt 672B Prompt 672C Output 674A Natural Language Input 772A Prompt 772B Output 774A Natural Language Input 910 Examples of Computing Devices 912 Bus Subsystem 914 Processor 916 Network Interface 920 User Interface Output Device 922 User Interface Input Devices 924 Memory Subsystem 925 Memory Subsystem 926 File Storage Subsystem 930 Random Access Memory 932 Read Only Memory

Claims

1. receiving a voice input provided by a user via an automated assistant interface on a client device; Based on processing the speech input, the voice input indicating a request by the user related to an object in an environment comprising the client device; a specificity of the object, which is a target classification in a taxonomy, required to resolve the request given the location of the client device; an initial natural language input provided by the user; feedback given by said user; the interaction history of said user and / or other users; Location and / or other contextual signals determining the specificity based on two or more of: responsive to determining that the request relates to the object in the environment, image processing at least one image capturing the object and acquired by a camera of the client device; determining, based on the image processing of the at least one image, that at least one parameter of the object required to resolve the request at the specificity is not resolvable; In response to determining that the at least one parameter is not resolvable, providing a prompt for presentation via the client device or an additional client device, the prompt being modified to match the at least one parameter; In response to the prompt, an additional image acquired by the camera; and Voice input and and receiving one or both of: the additional image received in response to the prompt; and the voice input received in response to the prompt. resolving a given attribute for the at least one parameter based on one or both of resolving the request based on the given attributes; causing an output reflective of said resolution of said request to be rendered on said client device or said additional client device; Including, A method implemented by one or more processors.

2. resolving the request based on the given attributes, issuing a query based on the given attributes; receiving one or more results in response to the issued query; generating the output based on at least one result of the received one or more results. The method of claim 1.

3. issuing the query based on the given attribute, transmitting the query to an agent over one or more networks, the agent including a first-party search system or a third-party search system; and receiving, in response to sending the query to the agent, from the agent, the one or more results responsive to the query. The method of claim 2.

4. selecting the agent from a plurality of available agents; sending the query to the agent based on selecting the agent from the plurality of available agents; The method according to claim 3.

5. The method of claim 1 , wherein the additional image is received in response to the prompt, and the step of resolving the given attribute is based on the additional image.

6. The method of claim 1 , wherein the speech input is received in response to the prompt, and the step of resolving the given attribute is based on the speech input.

7. The method of claim 1 , wherein the additional image and the voice input are received in response to the prompt, and the step of resolving the given attribute is based on the additional image and based on the voice input.

8. receiving at least one image captured by a camera of a client device; determining that the at least one image is associated with a need related to an object captured by the at least one image; a specificity of the object, which is a target classification level in a taxonomy, required to resolve the request based on the position of the client device when the image was captured; an initial natural language input provided by a user; feedback given by said user; the interaction history of said user and / or other users; Location and / or other contextual signals determining the specificity based on two or more of: In response to determining that the image is associated with the request related to the object for image processing the at least one image, determining, based on the image processing of the at least one image, that at least one parameter of the object required to resolve the request at the specificity is not resolvable; In response to determining that the at least one parameter is not resolvable, providing a prompt for presentation via the client device or an additional client device, the prompt being modified to match the at least one parameter; In response to the prompt, an additional image acquired by the camera; and Voice input and receiving one or both of the additional image received in response to the prompt; and the voice input received in response to the prompt. resolving a given attribute for the at least one parameter based on one or both of: resolving the request based on the given attributes; causing an output reflective of said resolution of said request to be rendered on said client device or said additional client device; Including, A method implemented by one or more processors.

9. The method of claim 8 , wherein the additional image is received in response to the prompt, and the step of resolving the given attribute is based on the additional image.

10. The method of claim 8 , wherein the speech input is received in response to the prompt, and the step of resolving the given attribute is based on the speech input.

11. The method of claim 8 , wherein the additional image and the voice input are received in response to the prompt, and the step of resolving the given attribute is based on the additional image and the voice input.

Images