Identifying foolproof products

Abstract

The method includes receiving image data of a user's region of interest from an image capture device, receiving a query from the user referencing one or more objects detected in the image data, and requesting a digital assistant to identify insights associated with the one or more objects referenced by the query. The method also includes processing the query and the image data to identify the one or more objects referenced by the query based on contextual data extracted from the image data, and determining insights associated with the identified one or more objects for the digital assistant to identify. The method also includes generating content for output from a user device associated with the user that indicates the identified insights associated with the identified one or more objects.

Description

[Technical field]

[0001] This disclosure relates to identifying fool-proofing products. [Background technology]

[0002] Product packaging is often poorly suited to clearly define what the product is and / or what it is used for. This can be the result of the use of small print and / or vague names used to describe or distinguish the product from a collection of different products that use nearly identical packaging. For example, a company may use the same type of bottle for each of their shampoo and conditioner products, and the two products have substantially identical labels with one difference being the hard-to-read term (e.g., shampoo or conditioner). A similar scenario occurs in restrooms, where soaps and lotions are placed next to a sink, packaged in identical bottles, and the text label that would normally distinguish the two products from one another is worn away. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method for identifying foolproof products. The computer-implemented method, when executed on the data processing hardware, causes the data processing hardware to perform operations including receiving image data of a user's area of ​​interest from an image capture device in communication with the data processing hardware. The operations further include receiving a query from the user referencing one or more objects detected in the image data, and requesting the digital assistant to discern insights associated with the one or more objects referenced by the query. The operations include processing the query and the image data to identify the one or more objects referenced by the query based on contextual data extracted from the image data, and determining insights associated with the identified one or more objects for the digital assistant to identify. The operations further include generating content for output from a user device associated with the user that indicates the identified insights associated with the identified one or more objects.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the query references, but does not explicitly identify, one or more objects associated with the insight the digital assistant is requested to identify. In other implementations, the contextual data extracted from the image data includes a user's hand recognized in the image data. In these implementations, processing the query and the image data to identify the one or more objects includes processing the query and the image data to identify the one or more objects based on the proximity of the user's hand recognized in the image data to at least one of the one or more objects detected in the image data.

[0005] In some implementations, the contextual data extracted from the image data includes a point of focus of the image capture device. In these implementations, processing the query and the image data to identify one or more objects referenced by the query includes processing the query and the image data to identify the one or more objects based on positions of the one or more objects detected in the image data relative to the point of focus of the image capture device.

[0006] In other embodiments, processing the query and image data to identify one or more objects associated with the insight includes performing query interpretation on the received query to identify one or more terms that convey descriptors of the one or more objects referenced by the query. These embodiments further include extracting visual features from the received image data to obtain object recognition results and determining an association between one or more of the object recognition results and the descriptors of the one or more objects. These embodiments also include identifying the one or more objects referenced by the query based on the association between one or more of the object recognition results and the descriptors of the one or more objects.

[0007] Further, the operations may include extracting text features from the received image data and combining the text features extracted from the received image data with the visual features extracted from the received image data to obtain object recognition results. The descriptors conveyed by the one or more terms identified by performing query interpretation on the received query may include at least one of an object category associated with the one or more objects, a physical trait associated with the one or more objects, or a position of the one or more objects relative to a reference object within a field of view of the image data.

[0008] Processing the query and image data to determine insights associated with one or more objects identified for the digital assistant to identify may include performing query interpretation on the received query to identify a type of insight for the digital assistant to identify. In some implementations, the type of insight identified for the digital assistant to identify includes at least one of an insight for uniquely identifying a single object, an insight for identifying multiple related objects, an insight for obtaining additional information about the object, an insight for providing personalized information about the object, an insight for distinguishing between two or more objects, or an insight for highlighting available information.

[0009] In some embodiments, the operations further include processing the query and image data to identify one or more objects and determining insights associated with the identified one or more objects for the digital assistant to identify, and then performing one or more operations to identify insights associated with the identified one or more objects, wherein generating content indicative of the identified insights associated with the identified one or more objects is based on the one or more operations performed to identify the insights.

[0010] Performing one or more operations to identify insights may include at least one of: extracting from the image data text features including detailed product information associated with at least one of the identified one or more objects; extracting from the image data text features including an object identifier that uniquely identifies at least one of the identified one or more objects; obtaining search results including product information associated with at least one of the identified one or more objects; obtaining text data including product information associated with at least one of the identified one or more objects, where the text data is uploaded by a merchant; obtaining personal information associated with at least one of the identified one or more objects; or obtaining custom information associated with at least one of the identified one or more objects.

[0011] In some implementations, generating content indicative of the identified insight includes generating graphical content indicative of the identified insight. In these implementations, the graphical content is superimposed on a graphical user interface displayed on a screen of the user device. In other implementations, generating content indicative of the identified insight includes generating audible content indicative of the identified insight. In these other implementations, the audible content is audibly output from the user device.

[0012] The operations may further include performing a personalized training process to train the visual feature recognizer with training samples to teach the visual feature recognizer to learn to uniquely identify a particular object, where each of the training samples may include training image data representing one of the particular objects and a corresponding ground truth label that uniquely identifies the particular object. Alternatively, the operations may further include performing a training process to train a content generator with the training samples to teach the content generator to generate personalized graphical content when the particular object is uniquely identified in the image data, where each of the training samples may include data representing the personalized graphical content to be generated and a corresponding ground truth label that uniquely identifies the particular object.

[0013] Another aspect of the present disclosure provides a system for identifying foolproof products. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving image data of a user's area of ​​interest from an image capture device in communication with the data processing hardware. The operations further include receiving a query from the user referencing one or more objects detected in the image data, and requesting a digital assistant to identify insights associated with the one or more objects referenced by the query. The operations include processing the query and the image data to identify the one or more objects referenced by the query based on contextual data extracted from the image data, and determining insights associated with the identified one or more objects for the digital assistant to identify. The operations further include generating content for output from a user device associated with the user indicating the identified insights associated with the identified one or more objects.

[0014] This aspect may include one or more of the following optional features: In some implementations, the query references, but does not explicitly identify, one or more objects associated with the insight the digital assistant is requested to identify. In other implementations, the contextual data extracted from the image data includes a user's hand recognized in the image data. In these implementations, processing the query and the image data to identify the one or more objects includes processing the query and the image data to identify the one or more objects based on the proximity of the user's hand recognized in the image data to at least one of the one or more objects detected in the image data.

[0015] In some implementations, the contextual data extracted from the image data includes a focus point of the image capture device. In these implementations, processing the query and the image data to identify one or more objects referenced by the query includes processing the query and the image data to identify the one or more objects based on positions of the one or more objects detected in the image data relative to the focus point of the image capture device.

[0016] In other embodiments, processing the query and image data to identify one or more objects associated with the insight includes performing query interpretation on the received query to identify one or more terms that convey descriptors of the one or more objects referenced by the query. These embodiments further include extracting visual features from the received image data to obtain object recognition results and determining an association between one or more of the object recognition results and the descriptors of the one or more objects. These embodiments also include identifying the one or more objects referenced by the query based on the association between one or more of the object recognition results and the descriptors of the one or more objects.

[0017] Further, the operations may include extracting text features from the received image data and combining the text features extracted from the received image data with the visual features extracted from the received image data to obtain object recognition results. The descriptors conveyed by the one or more terms identified by performing query interpretation on the received query may include at least one of an object category associated with the one or more objects, a physical trait associated with the one or more objects, or a position of the one or more objects relative to a reference object within a field of view of the image data.

[0018] Processing the query and image data to determine insights associated with one or more objects identified for the digital assistant to identify may include performing query interpretation on the received query to identify a type of insight for the digital assistant to identify. In some implementations, the type of insight identified for the digital assistant to identify includes at least one of an insight for uniquely identifying a single object, an insight for identifying multiple related objects, an insight for obtaining additional information about the object, an insight for providing personalized information about the object, an insight for distinguishing between two or more objects, or an insight for highlighting available information.

[0019] In some embodiments, the operations further include processing the query and image data to identify one or more objects and determining insights associated with the identified one or more objects for the digital assistant to identify, and then performing one or more operations to identify insights associated with the identified one or more objects, wherein generating content indicative of the identified insights associated with the identified one or more objects is based on the one or more operations performed to identify the insights.

[0020] Performing one or more operations to identify insights may include at least one of: extracting from the image data text features including detailed product information associated with at least one of the identified one or more objects; extracting from the image data text features including an object identifier that uniquely identifies at least one of the identified one or more objects; obtaining search results including product information associated with at least one of the identified one or more objects; obtaining text data including product information associated with at least one of the identified one or more objects, where the text data is uploaded by a merchant; obtaining personal information associated with at least one of the identified one or more objects; or obtaining custom information associated with at least one of the identified one or more objects.

[0021] In some implementations, generating content indicative of the identified insight includes generating graphical content indicative of the identified insight. In these implementations, the graphical content is superimposed on a graphical user interface displayed on a screen of the user device. In other implementations, generating content indicative of the identified insight includes generating audible content indicative of the identified insight. In these other implementations, the audible content is audibly output from the user device.

[0022] The operations may further include performing a personalized training process to train the visual feature recognizer with training samples to teach the visual feature recognizer to learn to uniquely identify a particular object, where each of the training samples may include training image data representing one of the particular objects and a corresponding ground truth label that uniquely identifies the particular object. Alternatively, the operations may further include performing a training process to train a content generator with the training samples to teach the content generator to generate personalized graphical content when the particular object is uniquely identified in the image data, where each of the training samples may include data representing the personalized graphical content to be generated and a corresponding ground truth label that uniquely identifies the particular object.

[0023] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a schematic diagram of an example system for identifying insights associated with one or more objects detected in image data. [Figure 2A] 1 is a schematic diagram of an example graphical user interface displaying graphical content illustrating identified insights associated with one or more objects detected within the image data. [Figure 2B] 1 is a schematic diagram of an example graphical user interface displaying graphical content illustrating identified insights associated with one or more objects detected within the image data. [Figure 2C]1 is a schematic diagram of an example graphical user interface displaying graphical content illustrating identified insights associated with one or more objects detected within the image data. [Figure 2D] 1 is a schematic diagram of an example graphical user interface displaying graphical content illustrating identified insights associated with one or more objects detected within the image data. [Figure 3A] FIG. 1 is a schematic diagram of an exemplary personalized training process for training a visual feature recognizer and content generator to identify particular objects in image data and generate graphical content in response to identifying the particular objects. [Figure 3B] FIG. 1 is a schematic diagram of an exemplary personalized training process for training a visual feature recognizer and content generator to identify particular objects in image data and generate graphical content in response to identifying the particular objects. [Figure 4] 1 is a flowchart of an example arrangement of operations of a method for identifying insights associated with one or more objects detected within image data. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Like reference symbols in the various drawings indicate like elements.

[0026] Product packaging is often poorly suited to clearly define what the product is and / or what it is used for. This can be the result of the use of small print and / or vague names used to describe or distinguish the product from a collection of different products that use nearly identical packaging. For example, a company may use the same type of bottle for each of their shampoo and conditioner products, and the two products have substantially identical labels with one difference being a hard-to-read term (e.g., shampoo or conditioner). This problem can be exacerbated when the distinguishing term on the label is in a foreign language. A similar scenario occurs in a restroom, where soaps and lotions are placed next to a sink, packaged in identical bottles, and the text label that would normally distinguish the two products from one another is worn away.

[0027] Embodiments herein are directed to generating augmented reality (AR) content on a user device associated with a user to communicate identified insights associated with one or more objects within the field of view of an image capture device. As will become apparent, the insights of a particular object communicated by the AR content can be used to not only provide detailed information about the product represented by the object recognized in the image data, but also to differentiate similarly packaged products that would otherwise be difficult to distinguish from one another without close inspection. Advantageously, the use of AR content can be similarly adapted to allow children who cannot read to distinguish different objects from one another by overlaying simple graphic elements on top of the object to communicate the necessary information about the object to the child. For example, if a tube of toothpaste and a tube of triple antibiotic ointment of similar size and shape are placed adjacent to one another in the field of view, AR content may be generated to provide an AR view on the display of the user device that overlays a warning graphic on the tube of triple antibiotic ointment that is intended to communicate to the child that the triple antibiotic ointment is not for use by children. In this example, the generated AR content may similarly overlay an encouraging / welcoming graphic onto the tube of toothpaste that is intended to communicate to the child that the tube contains toothpaste for the child to use to brush his / her teeth.

[0028] More specifically, embodiments herein are directed to enabling a user to interact with a digital assistant via a user device to issue a voice query that requests the digital assistant to identify insights regarding one or more objects detected in image data captured by an image capture device and related to an object referenced by the query. The image capture device may reside on the user device, and the digital assistant may run on the user device and / or a remote computing device (e.g., a cloud-based server) that communicates with the user device. In particular, the user may speak a voice query, and a microphone on the user device may capture the voice query in streaming audio, such that the user device, or the remote computing device (e.g., a cloud-based server), may perform speech recognition on the audio data, characterized by providing a textual representation (i.e., a transcription) of the voice query. In particular, the user freely issues voice queries to the digital assistant in a natural manner, as if the user were speaking to another human being. However, when naturally speaking voice queries in such a conversational manner, the digital assistant will often receive ambiguous queries that are missing parameters necessary to also uniquely identify one or more objects associated with the identifying insight and uniquely identify the scope of the identifying insight. For example, if the captured image data includes two bottles that are identical except that a respective label on one of the bottles identifies the bottle as a conditioner and a respective label on the other of the bottles identifies the bottle as a shampoo, the user may issue a voice query to the digital assistant of "What are these?" As described in more detail below, in order for the digital assistant to disambiguate these naturally spoken queries, the digital assistant may process the query and the image data to both identify one or more objects referenced by the query and to identify insights associated with the identified one or more objects.Using the above example, the digital assistant may extract context data from the image data to identify one or more objects referenced by the query. Here, the extracted context data may include a focus point (e.g., focal point) of the image capture device, such that two objects may be identified as the two bottles referenced by the query when detected in the image data relative to the focus point. The extracted context data may also include a user's hand recognized in the image data, which serves as a reference object such that two objects in proximity to the user's hand detected in the image data may be identified as the two bottles referenced by the query. For example, a user may use their hand to grasp, point, or convey some other gesture to indicate that the user is referring to two identical bottles when uttering the phrase "What are these?"

[0029] FIG. 1 illustrates an exemplary system 100 for identifying an insight associated with an object 50 detected within image data 122 and generating content 202 for output from a user device 10 indicating the identified insight to a user 102. Briefly, as described in more detail below, the user device 10 may be an assistant-enabled device configured to capture an utterance 106 spoken by the user 102, such as "Okay computer, which bottle is shampoo?", and invoke a digital assistant 105 to identify an insight associated with one or more objects detected within the image data 122 in a region of interest of the user 102. In this example, the two objects 50a, 50b detected within the image data 122 represent two bottles that are substantially identical except for their respective labels having different text in a foreign language that the user 102 cannot read. Using the query 118 and image data 122, the digital assistant 105 may identify the two objects 50a, 50b as one of two bottles / products that the user 102 cannot identify as shampoo, thereby identifying insights by recognizing text labels in the image data and distinguishing between the two objects 50 and determining which bottle is shampoo.

[0030] After identifying the insight by determining that the object 50a includes a bottle of shampoo, the digital assistant 105 generates content 202 for output from the user device 10, indicating the identified insight associated with the identified one or more objects 50. For example, the content 202 may include augmented reality (AR)-based graphic content 202a, such as a graphic element superimposed on a graphical user interface (GUI) 200 displayed on the screen 20 of the user device 10. Here, the GUI 200 presents real-time image data 122 representing an area of ​​interest, and the graphic content 202a superimposed on the GUI 200 may be overlaid on / around / proximity to one or more of the identified objects 50 to convey the insight identified by the digital assistant 105. In an example, the graphic content 202a superimposed on the GUI 200 may be overlaid on the object 50a to indicate the identified insight that the object 50a includes a bottle of shampoo. Here, the graphical content 202a overlaid on the object 50a may include text in the native language of the user 102 that conveys the name of the product (e.g., shampoo) associated with the object 50a. Although not shown in the example, the digital assistant 105 may also superimpose graphical content 202a on top of the other object 50b indicating that the other object 50b contains a hair conditioner to better inform the user 102 and distinguish the two bottles 50a, 50b of shampoo and conditioner from each other.

[0031] Additionally or alternatively, the content 202 generated by the digital assistant 105 for output from the user device 10 may include content 202b audibly output from the user device 10 to indicate the identified insight. For example, the content 202b may include a synthesized speech audibly output from the acoustic speaker 18 of the user device 10 (e.g., "The shampoo is in the bottle on the left"). In other examples, the audible content 202b may be any audio sound (e.g., an alarm when the user 102 asks if the product is safe to drink when the identified insight indicates that the product is toxic).

[0032] The system 100 illustrates the user device 10 as an assistant-enabled device running a digital assistant 105 with which a user 102 may interact by speech. The user device 10 may include smart glasses 10a, or a headset worn by the user 102, or another computing device 10b, such as a smartphone or tablet. The user device 10 may include other computing devices, such as, but not limited to, a smart display, a desktop / laptop, a smart watch, a smart appliance, a head-up display, or a smart vehicle windscreen. The user device 102 includes data processing hardware 12 and memory hardware 14, which stores instructions that, when executed on the data processing hardware 12, cause the data processing hardware 12 to perform operations. The user device 102 also includes (or communicates with) an array of one or more microphones 16 configured to capture acoustic sounds in streaming audio, such as speech, directed to the user device 10.

[0033] The user device 10 also includes an image capture device (i.e., an array of one or more cameras) 120 configured to capture image data 122 associated with a continuous sequence of image frames of an area of ​​interest of the user 102. The user device 10 also includes a display screen 20 configured to display a GUI 200 for presenting graphical content 202a indicative of identified insights associated with one or more objects 50 detected in the image data 122. If the user device 10 includes smart glasses 10a, the user 102 may view a natural scene through lenses that may also integrate with the display screen 20 to provide a GUI 200 for presenting graphical content 202 overlaid on the natural scene to indicate identified insights associated with one or more objects viewable in the natural scene.

[0034] In some examples, the user device 10 includes two or more user devices 10a, 10b paired with each other to provide shared functionality in a distributed manner. For example, the image capture device 120 and the display screen 20 may reside on smart glasses 10a worn by a user 102 and may capture image data 122 and present graphical content 202a, respectively, while the data processing hardware 12 and memory hardware 14 may reside on a smartphone 102b and generate the graphical content 202a.

[0035] With continued reference to FIG. 1, when a user 102 utters an utterance 106, "Okay computer, which bottle is shampoo?" in the vicinity of the user device 10, a microphone 16 of the user device 10 captures the utterance 106 in streaming audio, and audio data characterizing the utterance 106 is processed. Initial processing of the audio data may include filtering the audio data and converting the audio data from an analog signal to a digital signal. When the user device 10 processes the audio data, the user device 10 may store the audio data in a buffer in the memory hardware 14 for further processing. Once the audio data is in the buffer, the user device may detect whether the audio data includes a hot word using a hot word detector (not shown). The hot word detector is configured to identify hot words 110 included in the audio data without performing speech recognition on the audio data. In this example, the hot word detector may determine that the utterance 106, "Okay computer, which bottle is shampoo?" contains the hot word 110, "Okay computer," if the hot word detector detects an acoustic feature in the audio data that is characteristic of the hot word 110.

[0036] When the hot word detector determines that the audio data corresponding to the utterance 106 includes the hot word 110, the user device 10 may trigger a wake-up process to begin speech recognition on the audio data corresponding to the utterance 106. For example, an automatic speech recognition (ASR) model 130 running on the user device 10 may perform speech recognition on the audio data characterizing the utterance 106. The ASR model 130 may perform speech recognition on the portion of the audio data following the hot word 110. In this example, the ASR model 130 may recognize the phrase "Which bottle is shampoo" as a query 118 that refers to one or more objects 50 detected in the image data 122, and the query 118 requests the digital assistant 105 to identify an insight associated with the one or more objects 50 referenced by the query 118 (i.e., perform object classification to distinguish the objects from one another). Thus, the ASR model 130 may generate a transcription 132 of the query 118.

[0037] In some implementations, the ASR model 130 is located on the server 80 in addition to or instead of the user device 10. When the hot word detector triggers the user device 10 to wake up in response to detecting the hot word 110 in the utterance 106, the user device 10 may transmit audio data corresponding to the utterance 106 to the server 80 over the network 32. The server 80 may execute the ASR model 130 to perform speech recognition on the audio data and return a transcription 132 of the query 118 to the user device 10.

[0038] The user device 10 (or server 80) may perform speaker identification (SID) on the audio data to identify the speaker of the utterance 106. For example, the audio data may be processed to generate a speaker vector (e.g., a d-vector or an i-vector) that characterizes the voice of the user who spoke the utterance 106. The SID performed may include a text-dependent SID (TD-SID), where the SID is performed on portions of the audio data that correspond to a particular word or phrase, such as a predefined hot word 110. Additionally or alternatively, the SID performed may include a text-independent SID (TI-SID), where the SID is performed on the entire utterance 106, or on a portion of the utterance 106 that includes the query 118. In a TI-SID, the speaker vector is generated independently of the term or phrase spoken by the user 102, and may also be generated independently of the duration of the utterance 106.

[0039] The digital assistant 105 may have multiple registered users authorized to interact with the digital assistant 105. Here, the digital assistant 105 may run on one user device 10 that each registered user uses to interact with the digital assistant 105, and / or the digital assistant 105 may operate on multiple user devices 10, each associated with a different registered user (e.g., each registered user may have their own pair of smart glasses 10a). In the latter case, an SID may not be required since the digital assistant 105 may simply identify which registered user is speaking based on the user device 10 used to facilitate interaction with the digital assistant. During the registration process, each registered user speaks one or more phrases, and the digital assistant generates a respective reference speaker vector for that registered user and stores it in the registered user data store 194 as part of the user profile information. Later, when one of the registered users utters the utterance 106 captured by the user device 10, the SID may be performed to extract a speaker vector and compare it to a reference speaker vector stored in the registered user data store 194 to identify the particular registered user who uttered the utterance. The digital assistant 105 may use the speaker's identity to tailor or customize how it identifies insights, as well as tailor / customize the graphical and / or audible content 202 generated for output from the user device 10 to indicate the identified insights about one or more objects 50. For example, using the above example, assume that the user 102 who uttered the query 118 ("Which bottle is shampoo?") is identified as one of the registered users who is a child who cannot read. Here, the graphical content 202a overlaying on the object 50a identified as shampoo may include a green halo, a smiley face character, or other encouraging graphics that communicate to the child that the object 50a is a bottle of shampoo.Further, additional graphical content 202a may be overlaid on top of the other object 50b identified as a conditioner, and the additional graphical content may include a red "X", a skull and crossbones, or other deterrent graphic that communicates to the child that the object 50b is not a bottle of shampoo. These same techniques may be adapted to communicate which objects detected in the child user's scene are safe for use by the child and which are not. In such scenarios, the allergenic content of a particular food may be identified, and graphical content may be generated that communicates whether the food is safe for the child with specific allergy needs accessible to the child's respective user profile information stored in the registered user data store 194. Thus, a child known to have a dairy allergy may simply ask "Can I eat these?" in the query 118, and an encouraging / welcoming graphical element may be generated as graphical content overlaid on the food if the food does not contain dairy.

[0040] In particular, the user 102 may utter a query 118 to the digital assistant 105 in a natural manner, as if the user 102 were speaking to another human being in the same environment as the user. When expressed as a natural speech, the query 118 is likely to be ambiguous without additional context. For example, the query 118 "Which bottle is shampoo" simply refers to, but does not explicitly identify, a particular object 50 captured in the image data 122 associated with the insight that the query 118 is requesting the digital assistant 105 to identify. That is, without additional context, the digital assistant 105 has no way to ascertain from the transcription 132 of the query 118 alone whether the user 10 is referring to two bottles 50a, 50b on a shelf in front of the user 102, multiple bottles in a pantry (not shown) near the user 102, or a bottle containing other liquids, such as mouthwash or cleaning supplies, in a cabinet below the shelf.

[0041] Thus, to disambiguate a query 118 that is missing one or more parameters necessary for the digital assistant 105 to satisfy the query 118, embodiments herein are directed to a digital assistant 105 that processes the query 118 and image data 122 (i.e., processes a transcription 132 of the query 118) to uniquely identify one or more objects 50 referenced by the query 118 and determine insights associated with the identified one or more objects 50 for the digital assistant 105 to identify. The digital assistant 105 collectively employs a context extractor 140, a query interpreter 150, and a visual feature recognizer 160, running on the user device 10 or the server 80 to process the query 118 and image data 122 and disambiguate the query 118. Embodiments herein may further facilitate user interface (UI) interactions, such as a user tapping the user device 10 to invoke the ASR model 130 to perform speech recognition on subsequent utterances spoken by the user 102. For example, the user 102 may tap the side of the smart glasses 10a near the user's temple to turn on the microphone 16 and capture subsequent speech to be processed by the ASR model 130, without the user 102 needing to speak a specific hot word. Additionally or alternatively, the user 102 may perform a predefined gesture that, when captured by a camera or other sensor of the user device, triggers the ASR model 130 to perform speech recognition on the subsequent speech. Similarly, the user 102 may press a specific button on the user device 10 (e.g., the user presses and holds the power button on the user device 10) to trigger speech recognition. In some examples, the user device 10 may be a stationary device such as a kiosk, and the user may simply press a button and / or stand within the view of the camera to trigger speech recognition to be performed on the speech spoken by the user 102.

[0042] The context extractor 140 is configured to extract context data 142 from the image data 122 captured by the image device 120, and the extracted context data may be used by the digital assistant 105 to identify one or more objects 50 in the image data 122 referenced by the query 118. In some implementations, the context data 142 extracted from the image data may include a focus point of the image capture device 120, which may correspond to a focal point (FP) of the image data 122. As such, having the context data 142 identifying the focus point of the image capture device 120 conveys visually based localization data indicative of the object the user 102 was viewing, or more specifically, the object 50 on which the user was focusing the image capture device 120, when the user 102 uttered the utterance 106 that included the query 118.

[0043] The context extractor 140 may also obtain context data 142 in the form of user feedback provided by the user. For example, if the digital assistant cannot reliably identify the object 50 referenced by the query 118, the assistant 105 may provide a refinement prompt for the user to respond. For example, the digital assistant 105 may provide a graphical refinement prompt in the form of text (e.g., "Are you asking about the two bottles on the shelf or the bottles scattered on the counter below the shelf?") and / or a graphical indicator prompting the user to identify the referenced object. The user's response "on the shelf" serves as context data 142 informing the object 50 to be identified as the subject of the query 118. Similarly, rather than providing a refinement prompt that requires the user to speak an answer or select a graphic, the digital assistant 105 may instruct the user 102 to perform an action that assists the digital assistant 105 in better ascertaining information from the object 50 requested by the query. For example, the digital assistant 105 may provide a refinement prompt instructing the user to rotate an object 50, such as a bottle, so that the camera can capture the contents of the label, such that the user's 102 action of rotating the bottle 50 so that the label comes into view serves as contextual data 142 in the form of user feedback.

[0044] 2A and 2B, in some additional implementations, the context data 142 extracted from the image data 122 includes a hand 51 of the user 102 recognized in the image data 122. The recognized hand 51 may be identified by the visual feature recognizer 160 based on visual features 162 (FIG. 1) extracted from the image data 122. In particular, the context data 142 including the hand 51 may correspond to a reference object for ascertaining another object 50 to which the query refers. FIGS. 2A and 2B show an exemplary GUI 200 displayed on the screen 20 of the user device 10 depicting two objects 50a, 50b detected in the image data 122 of FIG. 1. The query 118 is more ambiguous than the example of FIG. 1, in that it only says "What is this?" without even being given a hint that the object to which the query 118 refers includes a bottle. FIG. 2A shows the user's hand 51 pointing toward the first object 50a on the left. Here, the recognized hand 51 is extracted as context data 142, and the digital assistant 105 is biased to identify the query 118 as referring to the left object 50a based on the proximity (and / or orientation) of the hand 51 being closer to the left object 50a than to the right object 50b. In contrast, the GUI 200 of FIG. 2B shows the hand 51 of the user 102 pointing instead toward a second object 50b on the right. Thus, the recognized hand 51 is extracted as context data 142, and the digital assistant 105 is biased to identify the right object 50b based on the proximity (and / or orientation) of the hand 51 being closer to the right object 50b than to the left object 50а.

[0045] Referring back to FIG. 1 , in some implementations, the query interpreter 150 is configured to perform query interpretation (i.e., semantic interpretation) on the transcription 132 of the query 118 to identify one or more terms that convey a descriptor 152 of one or more objects 50 referenced by the query. For example, the descriptor 152 may include an object category associated with one or more objects (e.g., bottle, box, medicine container, condiment, personal hygiene product, etc.), a physical trait associated with one or more objects 50 (e.g., color, size / dimension, shape, etc.), or a position of one or more objects relative to a reference object within the field of view of the image data (e.g., “What condiment is the penultimate condiment on the left?”). In some examples, the digital assistant 105 determines an association between the descriptor 152 and the contextual data 142 extracted from the image data 122. The query interpreter 150 may include a neural network trained using machine learning to identify terms that convey a descriptor of the object. In some examples, personal training is used to train neural networks using custom training samples provided by a user.

[0046] While the query interpreter 150 performs the query interpretation to identify the descriptors 152 of the objects 50, the visual feature recognizer 160 may extract visual features 162 from the received image data 122, and object recognition results 166 may be obtained, each characterizing a corresponding object 50 detected in the image data 122. Furthermore, the visual feature recognizer 160 may also extract textual or symbolic features 164 from the received image data for use in obtaining the object recognition results 166. For example, two bottles 50a, 50b may be of the same size and shape, but the bottles 50a, 50b may include a product label with a textual feature indicating that one bottle is a shampoo and the other is a conditioner. Thus, the visual feature recognizer 160 may combine the visual features 162 and the textual features 164 extracted from the image data to obtain a more accurate / robust object recognition result 166. The visual feature recognizer 160 may include a neural network trained using machine learning to identify objects based on visual features 162 and / or text / symbolic features 164 extracted from the image data. In some examples, personal training is used to train the neural network using custom training samples provided by a user.

[0047] To identify one or more objects referenced by the query 118, the digital assistant 105 may then determine an association between one or more of the object recognition results 166 and one or more object descriptors 152 conveyed by one or more terms identified in the query 118. In an example, performing query interpretation on the query 118 "Which bottle is shampoo?" identifies the term "bottle" as a descriptor 152 of the object category associated with the object 50. Thus, the digital assistant 105 may determine an association between the descriptor 152 of the term bottle and any visual features 162 extracted from the image data 122 that are characteristic of the bottle object class to identify the two objects 50a, 50b as one or more objects referenced by the query 118.

[0048] Additionally, the query interpreter 150 may be further configured to process the query 118 by performing query interpretation (i.e., semantic interpretation) on the transcription 132 of the query 118 to identify one or more types of insights for the digital assistant 105 to identify, and to determine insights associated with the one or more objects 50 identified for the digital assistant 105 to identify, where the query interpreter 150 may be trained with training queries, each of which includes a phrase paired with a corresponding label indicating a type of insight. For example, types of insights may include, without limitation, insights to uniquely identify a single object (e.g., in response to the query "What are the condiments needed for my Buffalo Wild Wings Rub?"), insights to identify multiple related objects (e.g., in response to the query "What is this for?" or in response to the query "What are the dosage information?"), insights to obtain additional information about the object (e.g., in response to the query "What is William's dose of this medication?"), insights to distinguish between two or more objects (e.g., in response to the query "Which bottle is shampoo?"), or insights to highlight available information associated with the object (e.g., in response to the query "What does the label say?").

[0049] After processing the query 118 and image data 122 to identify one or more objects and determine insights associated with the one or more objects 50 identified for the digital assistant 105 to identify, the digital assistant 105 may perform one or more operations to identify insights associated with the identified one or more objects. In the example of FIG. 1, based on identifying the two bottles 50a, 50b as the one or more objects 50 referenced by the query 118 and determining that the type of insights requested by the query 118 for the digital assistant 105 to identify include insights for distinguishing between two or more objects, the digital assistant 105 may perform operations including directing the visual feature recognizer 160 to extract text or symbol features 164 from the image data 122 and obtain object identifiers that uniquely identify the first object 50a as shampoo and the second object 50b as conditioner. For example, text features 164 extracted from image data 122 may be extracted from both product labels 56 disposed on bottle 50a that contain the Arabic word "shampoo" and product labels 56 disposed on bottle 50b that contain the Arabic word "conditioner." Using text (or symbol) features 164 extracted from image data 122, the digital assistant may uniquely identify each of objects 50a, 50b as shampoo and conditioner, respectively, even though visual features 162 indicate that the objects correspond to two identically sized and shaped bottles. Content generator 180 may translate the Arabic text indicated by text features 164 into the English word "shampoo" based on determining a profile in registered user data store 194 that the user speaks English as a first language. Content generator 180 may then generate a graphic element 202 for display in GUI 200, overlaid on top of object 50a, to indicate that object 50a is shampoo.

[0050] Similarly, in the example of FIG. 2B , the digital assistant 105 may identify the object 50b on the right side as the object referenced by the query 118, “What is this?”, and the digital assistant 105 may instruct the visual feature recognizer 160 to extract visual features 162 and / or textual features 164 from the image data 122 to obtain an object recognition result 166 that uniquely identifies the object 50b as a conditioner. The visual features 162 may be associated with the shape and size of a personal hygiene bottle (shampoo or conditioner). The textual features 164 may be associated with the product label 56 of the bottle, which includes the Arabic word “conditioner.” The content generator 180 may translate the Arabic text indicated by the textual features 164 into the English word “conditioner,” effectively identifying an insight to inform the user 102 that the object 50b next to the hand 51 is a conditioner. Thus, the content generator 180 may generate a graphic element 202 for display in the GUI 200, overlaid on top of the object 50b, to indicate that the object 50b is a conditioner.

[0051] Although the examples of Figures 1, 2A, and 2B above show the visual feature recognizer 160 extracting text or symbol features 164 on the product label 56 for use in identifying insights to communicate that the object includes shampoo and conditioner, the digital assistant 105 can uniquely identify a particular object without extracting the text features 164. This can be particularly useful when there are no text features 164 associated with the object, or when there are no text features to extract because the product label is peeling or worn off. In such scenarios, the extracted visual features 162 can indicate the color of the detected object and can distinguish between objects of similar shape / size. For example, in the example of Figure 1, the visual feature recognizer 160 can extract visual features 162 indicating that object 50a is blue and object 50b is white. The digital assistant 105 can then query the search engine 190 for search results including product information and determine that blue bottles are associated with shampoo and white bottles are associated with conditioner.

[0052] FIG. 2C illustrates an example GUI 200 displayed on the screen 20 of the user device 10 depicting an object 50 detected in the image data 122 corresponding to a bottle of acetaminophen referenced by the query 118, “What is my child’s medication information?” In this example, the type of insight that the query 118 requests the digital assistant 105 to identify includes an insight to obtain medication information for the identified object corresponding to the drug acetaminophen. In some examples, the digital assistant 105 queries a search engine or product data store to obtain search results that provide general medication information published by the drug’s manufacturer. Additionally or alternatively, the digital assistant 105 may extract text features 164 from a product label 56 located on the object 50 to identify general medication information for children listed on the product label 56.

[0053] The content generator 180 may generate a graphic element 202A for display within the GUI 200 adjacent to the bottle 50, where the identified insight is communicated, providing the user with general medication information for children listing medications for various weight ranges. In particular, the content generator 180 may provide the ability to highlight available information by extracting text from the product label and generating a graphic that conveys the text extracted from the label in a larger, more readable font. Alternatively, in some examples, merchants may have the ability to upload text data from their own product labels, allowing the content generator 180 to simply display the uploaded text within the GUI for graphical display, rather than requiring the content generator 180 to extract text from a captured image of the product label and then generate a graphic element 202A that highlights the information / text on the label.

[0054] In some additional examples, the digital assistant 105 identifies customized / personalized insights associated with the user 102. In the example of FIG. 2C, the digital assistant 105 may access the user's user profile from the registered user data store 194, which has personalized medication information for the user's 102 children, Bill and Jimbo. The user 102 (or Bill and Jimbo's healthcare provider) may have previously entered personalized medication information for Bill and Jimbo. Alternatively, the profile may simply indicate that Bill and Jimbo are the user's children and may include Bill and Jimbo's respective weights. The content generator 180 may now use the general medication information, as well as Bill and Jimbo's weights, to interpolate to identify personalized / customized medication information for Bill and Jimbo. The content generator 180 may generate a graphic element 202B for display in the GUI 200 adjacent to the bottle 50 to convey the identified insight that provides the user with personalized medication information for each of the user's 102 children, Bill and Jimbo, indicating the number of dosage units (mL) that Bill and Jimbo can each take every four (4) hours as needed.

[0055] In some examples, the digital assistant identifies additional customized / personalized insights that may be beneficial to the user 102 based on the context of the query 118 issued by the user 102. In the example of FIG. 2C, the digital assistant 105 may reference historical information, such as a medication log with time stamps indicating each time a dose of a particular medication was administered to one of the children, Bill and Jimbo. Here, the user 102 may issue the query "Give me my child's medication info" less than four (4) hours after administering Bill his last dose of acetaminophen. Because Bill's personalized medication information indicates that consecutive doses must be separated by four hours, the content generator 180 may generate a warning message stating, "Warning: Bill will not be able to receive his next dose for the next 30 minutes."

[0056] FIG. 2D illustrates an exemplary GUI 200 displayed on the screen 20 of the user device 10, which illustrates a plurality of objects 50 detected within the image data 122 corresponding to spices placed in the user's spice rack referenced by the query 118, "What are the ingredients in smoked wing rub?" In this example, the type of insight that the query 118 requests the digital assistant 105 to identify includes insights to identify the plurality of related objects 50, such as to identify all objects representing spices required for the recipe for smoked wing rub. In this example, the "smoked wing rub" referenced in the query 118 may refer to a custom recipe previously entered by the user that includes a list of spices to be added to the smoked chicken wings and the corresponding amount of each spice required to make a serving. Alternatively, the recipe may be a public recipe posted on a web page or software application that the digital assistant may retrieve.

[0057] Using any combination of the various techniques described above, the digital assistant may uniquely identify each object 50 detected in the image data 122 that represents one of the spices in the list of spices for the Smoked Wing Rub recipe. Moreover, the digital assistant may obtain additional information for each identified spice indicating the respective percentage / amount of the identified spice included in the Smoked Wing Rub recipe. For each object 50 identified as representing one of the spices, the content generator 180 may generate a pop-up graphic element 202P that indicates the percentage / amount of the spice specified in the recipe as well as a graphic element 202 that surrounds the object or highlights the object in some way to facilitate easy recognition by the user. In some examples, when the user grabs a spice bottle from the spice rack, the content generator may remove the graphic element surrounding the spice but nevertheless maintain the pop-up graphic element 202P because the user needs to add the required percentage / amount to the mixture of all spices specified in the recipe. In this manner, the content generator 180 may dynamically register the pop-up graphic elements 202P displayed in the GUI 200 to track movements by the corresponding objects 50.

[0058] 3A and 3B show an exemplary personalized training process 300 for training the visual feature recognizer 160 to recognize / identify specific objects and the content generator 180 to generate personalized graphical content 202 based on the identified results and, optionally, the specific user issuing the query 118. In the illustrated example, the user 102 is a parent of Kat who has a dairy allergy. The user 102 undergoes the personalized training process 300, which teaches the visual feature recognizer 160 not only to recognize the specific foods the user 102 is adding to the user's family pantry, but also to identify whether those foods are safe for Kat to eat. The user 102 may use the user device 10 (e.g., smart glasses worn by the user) to capture image data 122 of objects 50 representing the specific foods the user 102 is adding to the pantry. The user 102 may speak an utterance 106 directed to the digital assistant 105 that uniquely identifies the food represented by the object 50 in the captured image data 122 and indicates whether the food is safe for Kat to consume. The examples of Figures 3A and 3B show a user speaking an utterance 106 that may be recognized and converted to text (e.g., by the ASR model 130), and the user 102 may also optionally enter a text input via the user device 10 indicating the food and labeling the food as either safe or unsafe for Kat to consume.

[0059] In FIG. 3A, the image capture device 120 (FIG. 1) captures image data 122 of an object 50 representing packaging for gluten-free muffins that contain dairy products. The user 102 speaks an utterance 106 that states, "Add these gluten-free muffins to the pantry. They are not safe for Kat." The utterance 106 may also include a specific hot word or key phrase to trigger the digital assistant 105. The digital assistant 105 identifies that the object 50 captured in the image data 122 is associated with the food product "gluten-free muffin" communicated in the utterance 106 spoken by the user 102. The training process 300 trains the visual feature recognizer 160 with training samples that include image data 122 labeled with an item identifier 124 that uniquely identifies the object 50 as a "gluten-free muffin." Here, the visual feature recognizer 160 is trained to extract visual and / or textual features 162, 164 from the image data 122 and predict an output that recognizes that the object 50 in the image data 122 contains a “gluten free muffin.”

[0060] The training process 300 also trains the content generator 180 to generate personalized graphical content 202 for Kat when a query 118 is issued indicating whether the food is safe for Kat to eat. With continued reference to FIG. 3A, based on an utterance 106 including the phrase "They are not safe for Kat", the training process 300 trains the digital assistant 105 to learn that an object 50 identified as a "gluten-free muffin" in the image data 122 is not safe for Kat to consume. Thus, the training process 300 also trains the content generator 180 to generate deterrent graphical content 202D when the visual feature recognizer 160 identifies / classifies the object 50 as a "gluten-free muffin" in response to a query 118 indicating whether the food is safe for Kat to consume. For example, Kat may utter "Am I allergic to these muffins" or "Which muffin can I eat?" Alternatively, Kat's parent may utter "Which muffin is Kat's muffin?" Thus, the training process 300 provides a training sample as input to the content generator 180, the training sample including the deterrent graphic content 202D labeled with an item identifier 124 that uniquely identifies "gluten free muffin" and a user identifier 126 that identifies Kat. As used herein, providing the deterrent graphic content 202D may include the user 102 providing data including the deterrent graphic content 202D or may include the user 102 providing some identifier that identifies the deterrent graphic content 202D for the content generator 180 to generate.

[0061] In FIG. 3B, the image capture device 120 (FIG. 1) captures image data 122 of an object 50 representing packaging for vegan muffins. The user 102 utters an utterance 106 saying, "Add these vegan muffins to the pantry. Kat can eat them." The digital assistant 105 identifies that the object 50 captured in the image data 122 is associated with the food item "vegan muffin" communicated in the utterance 106 uttered by the user 102. The training process 300 trains the visual feature recognizer 160 with training samples that include image data 122 labeled with an item identifier 124 that uniquely identifies the object 50 as a "vegan muffin." Here, the visual feature recognizer 160 is trained to extract visual and / or textual features 162, 164 from the image data 122 and predict an output that recognizes that the object 50 in the image data 122 includes a "vegan muffin."

[0062] The training process 300 also trains the content generator 180 to generate personalized graphical content 202 for Kat when a query 118 is issued indicating whether the food is safe for Kat to eat. Still referring to FIG. 3B, based on the utterance 106 including the phrase "Kat can eat them", the training process 300 trains the digital assistant 105 to learn that the object 50 identified as a "vegan muffin" in the image data 122 is safe for Kat to consume. Thus, the training process 300 also trains the content generator 180 to generate welcome graphical content 202W when the visual feature recognizer 160 identifies / classifies the object 50 as a "vegan muffin" in response to the query 118 being issued indicating whether the food is safe for Kat to consume. For example, Kat may utter "Am I allergic to these muffins" or "Which muffins are safe for me". Alternatively, Kat's parent may utter "Which muffin is Kat's muffin". Thus, the training process 300 provides a training sample as input to the content generator 180, the training sample including welcome graphical content 202W labeled with an item identifier 124 that uniquely identifies "vegan muffin" and a user identifier 126 that identifies Kat. For example, the welcome graphical content 202W may include a smiley face. As used herein, providing the welcome graphical content 202W may include the user 102 providing data including the welcome graphical content 202W or may include the user 102 providing some identifier that identifies the welcome graphical content 202W for the content generator 180 to generate.

[0063] 4 is a flowchart of an example arrangement of operations of a method 400 for generating content 202 indicative of identified insights associated with one or more objects identified within image data 122 captured by image capture device 120. Method 400 may include a computer-implemented method executing on data processing hardware 510 (FIG. 5), which may reside on user device 10 or a remote server 80 in communication with user device 10. Data processing hardware 510 may execute instructions stored in memory hardware 520 (FIG. 5) to perform operations of method 400. Memory hardware 520 may reside on user device 10 or the remote server 80.

[0064] At operation 402, the method 400 includes receiving image data 122 of an area of ​​interest of the user 102 from an image capture device 120 in communication with data processing hardware 510. At operation 404, the method 400 includes receiving a query 118 from the user 102 that references one or more objects 50 detected in the image data 122, and requesting the digital assistant 105 to identify an insight associated with the one or more objects 50 referenced by the query 118.

[0065] At operation 406, the method 400 includes processing the query 118 and the image data 122 to identify one or more objects 50 referenced by the query 118 based on the context data 142 extracted from the image data 122, and determining insights associated with the identified one or more objects 50 for identification by the digital assistant 105. At operation 408, the method 400 also includes generating content 202 for output from a user device 10 associated with the user 102 indicating the identified insights associated with the identified one or more objects 50.

[0066] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0067] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0068] 5 is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described herein. Computing device 500 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0069] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 that connects to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and the storage device 530. Each component 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or may exist in other ways as needed. The processor 510 may include the data processing hardware 12 of the user device 10 or the data processing hardware of the server 80. The data processing hardware 510 may process instructions for execution within the computing device 500, including instructions stored in the memory 520 or the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to the high-speed interface 540. The memory 520 may include the memory hardware 14 of the user device 10 or memory hardware resident in the server 80. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as appropriate, and multiple computing devices 500 may be connected, each performing some of the required operations (e.g., as a bank of servers, a group of blade servers, or a multi-processor system).

[0070] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0071] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0072] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages low-bandwidth-intensive operations. Such an allocation of duties is merely an example. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and can be coupled, via a network adapter or the like, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0073] The computing device 500 can be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 500a, or multiple times within a cluster of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0074] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general purpose, coupled to receive data and instructions from the storage system, and to transmit data and instructions to the storage system.

[0075] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0076] The processes and logic flows described herein may be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data to generate output. The processes and logic flows may also be executed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operably coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0077] To provide for interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, spoken language, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0078] Although several implementations have been described, it is understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (400), when executed on data processing hardware (510), causing the data processing hardware (510) to: receiving image data (122) of a region of interest of a user (102) from an image capture device (120) in communication with said data processing hardware (510); receiving a query (118) from the user (102) referencing one or more objects (50) detected within the image data (122) and requesting a digital assistant (105) to identify insights associated with the one or more objects (50) referenced by the query (118); processing the query (118) and the image data (122); Identifying the one or more objects (50) referenced by the query (118) based on contextual data (142) extracted from the image data (122); Determining the insights associated with the identified one or more objects for identification by the digital assistant (105); and generating content (202) for output from a user device (10) associated with the user (102), the content (202) being indicative of the identified insights associated with the identified one or more objects (50); A computer-implemented method for causing a computer to perform operations including:

2. The computer-implemented method of claim 1 , wherein the query references, but does not explicitly identify, the one or more objects associated with the insight that the digital assistant is requested to identify.

3. the contextual data (142) extracted from the image data (122) includes a hand (51) of the user (102) recognized in the image data (122); 3. The computer-implemented method (400) of claim 1 or 2, wherein processing the query (118) and the image data (122) to identify the one or more objects (50) comprises processing the query (118) and the image data (122) to identify the one or more objects (50) based on a proximity of the hand (51) of the user (102) recognized in the image data (122) to at least one of the one or more objects (50) detected in the image data (122).

4. The contextual data (142) extracted from the image data (122) includes a focus point of the image capture device (120); The computer-implemented method (400) of any of claims 1 to 3, wherein processing the query (118) and the image data (122) to identify the one or more objects (50) referenced by the query (118) comprises processing the query (118) and the image data (122) to identify the one or more objects (50) based on a position of the one or more objects (50) detected in the image data (122) relative to the focus point of the image capture device (120).

5. Processing the query (118) and the image data (122) to identify the one or more objects (50) associated with the insight includes: performing a query (118) interpretation on the received query (118) to identify one or more terms that convey descriptors (152) of the one or more objects (50) referenced by the query (118); extracting visual features (162) from the received image data (122) to obtain object recognition results (166); determining an association between one or more of the object recognition results (166) and the descriptors (152) of the one or more objects (50); Identifying the one or more objects (50) referenced by the query (118) based on the association between one or more of the object recognition results (166) and the descriptors (152) of the one or more objects (50); The computer-implemented method (400) of any one of claims 1 to 4, comprising:

6. The operation is extracting text features (164) from the received image data (122); combining the text features (164) extracted from the received image data (122) with the visual features (162) extracted from the received image data (122) to obtain the object recognition results (166); The computer-implemented method (400) of any of claims 1 to 5, further comprising:

7. The descriptor (152) conveyed by the one or more terms identified by performing query (118) interpretation on the received query (118) comprises: an object category associated with said one or more objects (50); a physical trait associated with said one or more objects (50); or a position of the one or more objects (50) relative to a reference object within the field of view of the image data (122); The computer-implemented method (400) of any one of claims 1 to 6, comprising at least one of:

8. A computer-implemented method (400) according to any one of claims 1 to 7, wherein processing the query (118) and the image data (122) to determine the insights associated with the identified one or more objects (50) for identification by the digital assistant (105) includes performing a query (118) interpretation on the received query (118) to identify a type of insight for identification by the digital assistant (105).

9. The types of insights identified for the digital assistant (105) to identify include: Insight for uniquely identifying a single object (50); Insight for identifying a plurality of related objects (50); Insight for obtaining additional information about the object (50); Insights for providing personalized information about the object (50); Insight into distinguishing between two or more objects (50); or Insights to highlight available information; The computer-implemented method (400) of any of claims 1 to 8, comprising at least one of:

10. The operations include processing the query (118) and the image data (122) to identify the one or more objects (50) and determining the insights associated with the identified one or more objects for identification by the digital assistant (105), and then: performing one or more operations to identify the insight associated with the identified one or more objects (50); A computer-implemented method (400) according to any of claims 1 to 9, wherein generating content (202) indicative of the identified insight associated with the identified one or more objects (50) is based on the one or more operations performed to identify the insight.

11. Performing the one or more operations to identify the insight includes: extracting text features (164) from the image data (122) comprising detailed product information associated with at least one of the identified one or more objects (50); extracting text features (164) from the image data (122) including an object identifier that uniquely identifies at least one of the identified one or more objects (50); obtaining search results including product information associated with at least one of the identified one or more objects (50); obtaining text data including product information associated with at least one of the identified one or more objects (50), the text data being uploaded by a merchant; obtaining personal information associated with at least one of the identified one or more objects (50); or Obtaining custom information associated with at least one of the identified one or more objects (50); The computer-implemented method (400) of any of claims 1 to 10, comprising at least one of:

12. Generating content (202) indicative of the identified insight includes generating graphical content indicative of the identified insight; The computer-implemented method (400) of any of the preceding claims, wherein the graphical content is superimposed on a graphical user interface (200) displayed on a screen (20) of the user device (10).

13. Generating content indicative of the identified insight (202) includes generating audible content indicative of the identified insight; The computer-implemented method (400) of any of claims 1 to 12, wherein the audible content is output audibly from the user device (102).

14. The computer-implemented method (400) of any of claims 1 to 13, wherein the operations further include performing a personalized training process (300) to train a visual feature recognizer (160) with training samples to teach the visual feature recognizer (160) to learn to uniquely identify specific objects (50), each training sample including training image data representing one of the specific objects (50) and a corresponding ground truth label that uniquely identifies the specific object (50).

15. The computer-implemented method (400) of any of claims 1 to 14, wherein the operations further include performing a training process (300) to train a content generator (180) with training samples to teach and learn the content generator (180) to generate personalized graphical content (202) when a particular object (50) is uniquely identified in the image data (122), each training sample including data representative of the personalized graphical content (202) to be generated and a corresponding ground truth label that uniquely identifies a particular object (50).

16. A system (100) including data processing hardware (510) and memory hardware (520) in communication with the data processing hardware (510), the memory hardware stores instructions; The instructions, when executed on the data processing hardware (510), cause the data processing hardware (510) to: receiving image data (122) of a region of interest of a user (102) from an image capture device (120) in communication with said data processing hardware (510); receiving a query (118) from the user (102) referencing one or more objects (50) detected within the image data (122) and requesting a digital assistant (105) to identify insights associated with the one or more objects (50) referenced by the query (118); processing the query (118) and the image data (122); Identifying the one or more objects (50) referenced by the query (118) based on contextual data (142) extracted from the image data (122); Determining the insights associated with the identified one or more objects for identification by the digital assistant (105); and generating content (202) for output from a user device (10) associated with the user (102), the content (202) being indicative of the identified insights associated with the identified one or more objects (50); Causes a system to perform operations including

17. The system (100) of claim 16, wherein the query (118) references, but does not explicitly identify, the one or more objects (50) associated with the insight that the digital assistant (105) is requested to identify.

18. the contextual data (142) extracted from the image data (122) includes a hand (51) of the user (102) recognized in the image data (122); The system of claim 16 or 17, wherein processing the query (118) and the image data (122) to identify the one or more objects (50) comprises processing the query (118) and the image data (122) to identify the one or more objects (50) based on a proximity of the hand (51) of the user (102) recognized in the image data (122) to at least one of the one or more objects (50) detected in the image data (122).

19. The contextual data (142) extracted from the image data (122) includes a focus point of the image capture device (120); The system (100) of any of claims 16 to 18, wherein processing the query (118) and the image data (122) to identify the one or more objects (50) referenced by the query (118) comprises processing the query (118) and the image data (122) to identify the one or more objects (50) based on a position of the one or more objects (50) detected in the image data (122) relative to the focus point of the image capture device (120).

20. Processing the query (118) and the image data (122) to identify the one or more objects (50) associated with the insight includes: performing a query (118) interpretation on the received query (118) to identify one or more terms that convey descriptors (152) of the one or more objects (50) referenced by the query (118); extracting visual features (162) from the received image data (122) to obtain object recognition results (166); determining an association between one or more of the object recognition results (166) and the descriptors (152) of the one or more objects (50); Identifying the one or more objects (50) referenced by the query (118) based on the association between one or more of the object recognition results (166) and the descriptors (152) of the one or more objects (50); The system (100) according to any one of claims 16 to 19, comprising:

21. The operation is extracting text features (164) from the received image data (122); combining the text features (164) extracted from the received image data (122) with the visual features (162) extracted from the received image data (122) to obtain the object recognition results (166); The system (100) of any of claims 16 to 20, further comprising:

22. The descriptor (152) conveyed by the one or more terms identified by performing query (118) interpretation on the received query (118) comprises: an object category associated with said one or more objects (50); a physical trait associated with said one or more objects (50); or a position of the one or more objects (50) relative to a reference object within the field of view of the image data (122); The system (100) according to any one of claims 16 to 21, comprising at least one of:

23. The system (100) of any one of claims 16 to 22, wherein processing the query (118) and the image data (122) to determine the insights associated with the identified one or more objects (50) for identification by the digital assistant (105) includes performing a query (118) interpretation on the received query (118) to identify a type of insight for identification by the digital assistant (105).

24. The types of insights identified for the digital assistant (105) to identify include: Insight for uniquely identifying a single object (50); Insight for identifying a plurality of related objects (50); Insight for obtaining additional information about the object (50); Insights for providing personalized information about the object (50); Insight into distinguishing between two or more objects (50); or Insights to highlight available information; The system (100) according to any one of claims 16 to 23, comprising at least one of:

25. The operations include processing the query (118) and the image data (122) to identify the one or more objects (50) and determining the insights associated with the identified one or more objects for identification by the digital assistant (105), and then: performing one or more operations to identify the insight associated with the identified one or more objects (50); A system (100) as described in any of claims 16 to 24, wherein generating content (202) indicative of the identified insight associated with the identified one or more objects (50) is based on the one or more operations performed to identify the insight.

26. Performing the one or more operations to identify the insight includes: extracting text features (164) from the image data (122) comprising detailed product information associated with at least one of the identified one or more objects (50); extracting text features (164) from the image data (122) including an object identifier that uniquely identifies at least one of the identified one or more objects (50); obtaining search results including product information associated with at least one of the identified one or more objects (50); obtaining text data including product information associated with at least one of the identified one or more objects (50), the text data being uploaded by a merchant; obtaining personal information associated with at least one of the identified one or more objects (50); or Obtaining custom information associated with at least one of the identified one or more objects (50); The system (100) according to any one of claims 16 to 25, comprising at least one of:

27. Generating content (202) indicative of the identified insight includes generating graphical content indicative of the identified insight; The system (100) according to any of claims 16 to 26, wherein the graphical content is superimposed on a graphical user interface (200) displayed on a screen (20) of the user device (10).

28. Generating content indicative of the identified insight (202) includes generating audible content indicative of the identified insight; The system (100) of any of claims 16 to 27, wherein the audible content is output audibly from the user device (102).

29. The system (100) of any of claims 16 to 28, wherein the operations further include performing a personalized training process (300) to train the visual feature recognizer (160) with training samples to teach the visual feature recognizer (160) to learn to uniquely identify specific objects (50), each training sample including training image data representing one of the specific objects (50) and a corresponding ground truth label that uniquely identifies the specific object (50).

30. The system (100) of any of claims 16 to 29, wherein the operations further include performing a training process (300) to train the content generator (180) with training samples to teach and learn the content generator (180) to generate personalized graphical content (202) when a particular object (50) is uniquely identified in the image data (122), each training sample including data representing the personalized graphical content (202) to be generated and a corresponding ground truth label that uniquely identifies a particular object (50).

Images