Gaze-based command disambiguation

Abstract

Aspects of the subject technology provide improved techniques for human-machine interaction, including disambiguation of language commands of a human user. Improved techniques may include integrating language input from a user with additional input from sensors related to the user and the user's environment. In an aspect, an image of a user's environment may be first analyzed to identify an object in the environment. A second analysis of the identified subset of objects in the environment may be triggered with respect to the user's input, such as the user's gaze location. Results of these analyses may then be used to parse ambiguity in the language user input from the user.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 408,075, filed on September 19, 2022, entitled “GAZE-BASED COMMANDDISAMBIGUATION,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This specification relates generally to machine understanding of human communications, including natural language understanding. Background Art

[0004] Modern technology has enabled computers to understand aspects of human environments and human communications. For example, computers can analyze images from a camera to identify objects in the image, and other technologies can analyze audio recordings to estimate the words spoken by a user in an audio recording. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Certain features of the subject technology are set forth in the appended claims. However, for illustrative purposes, several specific implementations of the subject technology are set forth in the following figures.

[0006] Figure 1 An example system for sensing an environment of a human user is illustrated.

[0007] Figure 2 An example system for understanding user commands is illustrated.

[0008] Figure 3 An example method for understanding user commands is illustrated.

[0009] Figure 4 An example computing device is illustrated that can be used to implement aspects of the subject technology. DETAILED DESCRIPTION

[0010] The specific embodiments set forth below are intended to be descriptions of various configurations of the subject technology and are not intended to represent the only configuration by which the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, and one or more other specific implementations may be used to practice. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concepts of the subject technology.

[0011] Although artificial intelligence and machine learning have made great progress in enabling humans and computers to interact, various problems still hinder human-computer interaction from looking as natural as human-human interaction. Specifically, a computer's understanding of human language communication may require a more comprehensive understanding of the context of the human speaker, rather than just the understanding of the spoken words. For example, a human may ask a computer "What is this?", and in order for the computer to understand what "this" refers to, additional understanding of the speaker's context may be required. "This" may refer to an object held in the speaker's hand, an object the speaker is pointing to, or an object the user is looking at when asking the question. This article presents techniques for improved human-computer interaction.

[0012] Improved techniques for human-computer interaction may include integrating language input from a human user (such as words spoken, typed, or signed with gestures) with additional user input related to the user and the user's environment. Sensors may provide input about the user's environment, such as images or videos of areas close to the user. Sensors may also provide input about the user himself, providing, for example, indications of the user's gestures, postures, and / or gaze positions. In one aspect, an image of the user's environment may be first analyzed to identify objects in the environment. Input about the user himself (such as gaze position) may trigger a second (possibly deeper) analysis of a subset of objects identified in the environment. The results of these analyses may then be used to resolve ambiguities in the language user input from the user. In one aspect, a first analysis and / or a second analysis may be performed on an object in advance (e.g., before the user asks or talks about the object) to reduce the delay required to understand or respond to the received language user input. In another aspect, performing the analysis in advance before receiving a command may solve timing problems, where the command refers to an object that is no longer in the proximity area when the command is received.

[0013] Figure 1An example system 100 for sensing an environment of a human user is illustrated. A proximity zone 112 of a user 102 may include objects 110.1, 110.2, 110.3, 110.4. A device 120 may include a first sensor 104 and a second sensor 106. The first sensor 104 may scan a proximity zone 112 that includes objects 110.1, ..., 110.4, and the second sensor 106 may scan the user 102. A third sensor 105 may accept user command inputs and may be, for example, a microphone. The device 120 may be, for example, a handheld device (such as a camera or tablet), a head-mounted device, or a stand-alone device. A scan of the user (e.g., in conjunction with a scan of the proximity zone 112) may indicate a user's interest in one or more objects in the proximity zone 112. For example, a scan of the user by the second sensor 104 may indicate that the user's visual gaze direction 108 is directed to the location of a specific object 110.2 detected in the scan of the proximity zone 112. In other aspects not depicted, the scanning of the user by the second sensor 106 may indicate a gesture and / or other body posture of the user that indicates the user's interest in one or more of the objects 110.X in the proximity area 112. For example, the user may point a finger or a device such as a mobile phone at a particular object to indicate the particular object, hold the particular object in the hand, or lean over the particular object.

[0014] In one or more aspects, the first sensor 104 and the second sensor 106 can be the same type of sensor, such as a camera for capturing images of a scene including the proximity area 112 or the user 102, respectively. In other aspects, the first sensor 104 and the second sensor 106 can be different types of sensors, or operate in different modalities. For example, the first sensor can be a camera that produces visible light images, while the second sensor can be an eye tracker that produces an indication of the user's eye position or the user's gaze direction.

[0015] exist Figure 1 In other aspects not depicted in the drawings, a single sensor may be used to scan both the proximity zone 112 and the user 102. In some cases, a single scan of the single sensor may scan both the user 102 and the proximity zone 112 simultaneously. For example, a scene captured by a single camera may include both the user's proximity zone 112 and the user 102 in a single image. In other cases, the single sensor may scan the user 102 and the proximity zone 112 in separate scans. For example, a single camera may be oriented to capture a first scene including the proximity zone 112 in a first image, and then reoriented to capture a second scene including the user 102 in a second image.

[0016] exist Figure 1In other aspects not depicted in , the first sensor 104 and the second sensor 106 may be embodied in different devices. For example, a first handheld device (such as a tablet or camera) may include the first sensor 104, while the head mounted device includes the second sensor 106. In one aspect, these separate devices may be configured to work together as a system for command disambiguation. In other aspects, the system for disambiguation may include a third sensor or other user input feature capable of accepting user commands. For example, the third sensor may be a microphone included in the device 120, may be included in a separate device such as earbuds or other headphones, or may be included as a stand-alone separate device for audio interaction with the user 102.

[0017] The system 100 may include various sensors 104, 106, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near field communication sensors, etc. In addition, the system 100 may include hardware elements that can receive user input, such as hardware buttons or switches. The user input detected by such sensors and / or hardware elements corresponds to various input modalities for interacting with virtual content displayed in a given extended reality environment. For example, such input modalities may include but are not limited to facial tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breathing, temperature, electroencephalogram, smell), recognition of voice or audio (e.g., specific hot words), and activation of buttons or switches, etc.

[0018] Figure 2 An example system 200 for semantically understanding user commands is illustrated. The system 200 may be Figure 1 An example of a device 120 for detecting a user's input. A first analyzer 202 may analyze a scan of a user's proximity region to identify objects in the proximity region. A hit tester 204 may compare a first user input with the objects identified by the first analyzer to identify triggered objects, which may be a subset of all objects in the proximity region. A command processor 208 for processing user commands may include an ambiguity resolver for resolving ambiguities in user commands based on information about the triggered objects.

[0019] In optional aspects, the system 200 may include a second analyzer for analyzing the triggered object to generate information about the triggered object. In other aspects, the remote second analyzer may exist separately from the system 200 (e.g., in a computing cloud), and the system 200 may be connected to the remote second analyzer via a computer network. Such a remote second analyzer may provide information about the triggered object in response to a request from the system 200. In one aspect, a transmitter (not depicted) may transmit a request for a second analysis to the remote second analyzer, and a receiver (not depicted) may receive the requested information from the remote second analyzer. The command processor 208 may include an ambiguity identifier 212 for identifying ambiguities in user commands. In one aspect, the command processor 208 may also execute a user command after resolving the identified ambiguities in the user command. For example, when the user command includes a question, the command processor 208 may answer the user's question.

[0020] In one aspect, the first analyzer 202 and the second analyzer 206 may be implemented on a single device. In other aspects, the second analyzer may need to access information that is not available locally, or may require more computing resources than the first analyzer, for example because the second analysis requires substantially greater storage, processor power, or other computing resources. When the computing resource requirements between the first analysis and the second analysis are not similar, efficiency can be gained by implementing the first analysis and the second analysis on separate devices. In one aspect, the second analyzer may be implemented on a cloud computing device, and the second analyzer may be shared among many devices having a command disambiguation system.

[0021] In an optional aspect, the first analyzer may additionally identify a classification of one or more of the objects identified by the first analyzer. For example, the identified object may be classified as a houseplant or as containing text. When the classified object is selected as a triggered object (such as by the hit tester 204), the second analyzer may further base its analysis of the triggered object on the classification of the triggered object. In some cases, the first analysis may result in a rough classification, while the second analysis may result in a finer classification with more detail and / or classification within a larger number of categories. For example, the first classification may classify an object as only a "book", while the second analysis may identify the title of the book, the author of the book, the number of pages in the book, and web links to audiobook versions of the book and book reviews.

[0022] In other optional aspects, the system 200 may include one or more sensors. For example, the first sensor 220 may generate a scan of the proximity area; the second sensor 222 may detect the first user input; and the third sensor 224 may capture the user command. In various aspects, one or more of the first sensor, the second sensor, and the third sensor may be combined. For example, the first sensor and the second sensor may be a single camera sensor capable of scanning the user's proximity area and the user himself; alternatively, the second sensor and the third sensor may be combined into a single camera sensor capable of capturing the first user input including a gesture and capturing the user command as a gesture sign language.

[0023] In one or more aspects, Figure 2 Elements (such as 202, 204, 206 and 208) may be implemented as computer programs executed on one or more processors. Figure 2 Each of the elements of may also be implemented using an artificial intelligence processor, such as a neural network using weights trained using training data. In various aspects, Figure 2 The elements may be implemented as one or more convolutional neural network (“CNN”)-based and / or generative adversarial network (“GAN”)-based methods or processors, as application specific integrated circuits, or generally as any integrated circuit or processing circuit system.

[0024] In one or more aspects, the hit tester 204 may select one or more triggered objects from the set of objects identified by the first analyzer 202. The triggered object may be selected based on a comparison of the first user input with information about the object identified by the first analyzer. The first user input may be based on the user's focus on the object, the direction or position of gaze, a gesture such as a pointing finger, or a body posture. In an example of a hit test, the first user input may indicate a location within the scanned proximity area, and one or more objects at or near the indicated location may be selected as triggered objects. In one aspect, the first analyzer may identify objects, including the location or location range of the object, such as a bounding box of a detected object. In one aspect, only the closest object may be selected; in another aspect, all objects without a threshold distance may be selected as triggered objects. In another aspect, selecting an object as a triggered object may trigger a request for the triggered object to be analyzed by a second analyzer.

[0025] In one or more aspects, the second analyzer may generate a variety of information about the triggered object, and may be provided as a feature vector enumerating a variety of different features about the analyzed object. The triggered object information generated by the second analyzer may be derived from visual information (such as an image of a book containing the title of the book) or other information (such as the direction from the user to the object from different types of sensors (such as directional radio sensors)). In the example of a book object, the second analysis of the book object may include the title, author, number of pages, a link to an audio book version of the book, and a link for purchasing the book, as well as a review rating for the book. This information about the book may be organized together into a feature vector. Additional examples of objects identified by the first analyzer 202 and their corresponding information generated by the second analyzer 206 may include: an indoor plant may have corresponding information identifying the species and care information of the plant; a person may have corresponding information identifying the name and contact information of the person; a box of cereal may have corresponding information identifying the nutritional information of the cereal; and so on.

[0026] In one aspect, the scan of the proximity area includes one or more images of the proximity area, and the first analysis may include analyzing the images at a first resolution, and the second analysis may include analyzing the images (or portions of the images corresponding to the triggered objects) at a second resolution higher than the first resolution.

[0027] Figure 3 An example method 300 for semantically understanding a user command is illustrated. The method 300 includes identifying objects in a user's proximity area through a first analysis of a scan of the proximity area (box 304). A first user input may be used to identify a subset of the identified objects as triggered objects (box 306). A request for first information about the triggered objects (box 308) may be transmitted, for example, to a remote device, which performs a second analysis to determine the requested first information. The requested information about the triggered objects may be received, for example, from the remote device or other second analysis device (box 310). For example, in response to determining that the user command includes ambiguity, the ambiguity in the user command may be resolved based on the received first information about the triggered objects (box 312).

[0028] In an optional aspect of method 300, a proximity region may be scanned to generate a scan of the proximity region, for example, by converting raw sensor data into a format for use by a first analysis operation (block 302). Ambiguities in user commands may be identified, for example, by a natural language command processor based on a language user command received from a user (block 316). The user command may be executed based on the resolved ambiguities in the user command (block 314). In one aspect, execution of the user command may be performed locally along with other operations of method 300, or in another aspect, execution of the user command may be performed remotely, such as in response to a request from a local device to execute a command on a remote device.

[0029] In an optional aspect, second information may be determined from the first information (block 318), and a command may be executed based on the second information. For example, a book object may be triggered (block 306), thereby generating first information including the title of the book. In response to a command including a user question about the author of the book, second information indicating the author's name may be determined from the title (block 318). The command may then be executed by answering the user's question about the author (block 314).

[0030] In one or more aspects, the first user input (for identifying the triggered object) and the user command may be received via two different user input modalities. For example, the first user input may be obtained via a passive modality (such as a camera for monitoring and / or scanning the user's gaze or hand movements), which may allow the user to be passively monitored without the user intentionally engaging the user command system, while the user command may be received via an active modality (such as via a microphone that records the user's intentional verbal commands).

[0031] In one or more aspects, a user command may include ambiguity due to an implicit reference to the subject of the command. For example, a user command for "turn up the volume" may have a loudspeaker as an implicit subject, or a user question for "who wrote that book" may implicitly reference a book that the user is viewing or near the user without explicitly identifying the title of the book. In a first example, a first analysis of a scanned scene may identify a black box as an object in the scene. A first user input including a viewer's gaze on the black box may trigger a request for a second analysis of the black box image, and the second analysis may return information about the black box, indicating that the black box is a loudspeaker, and the information may also include an indication of a control mechanism for controlling the speaker. Subsequently, when a user command for "turn up the volume" is received, the command processor may identify an implicit type of ambiguity in the command because the subject of the command is not identified in the command itself. The command processor may resolve an implicit type command by examining the most recently triggered object to select which most recently triggered object may best match the missing subject of the current ambiguous command. In one aspect, the selection of the most recently triggered object may be based on a classification from the first analysis, and / or may be based on information obtained from the second analysis. The command processor may use the information instructing the control mechanism of the loudspeaker to execute a command to increase the volume.

[0032] In one or more aspects, a user command may include ambiguity due to an indirect reference to the subject of the command. For example, a user command answers the question "what is this" or "who wrote that book". In this case, the explicit subject of the command "this" is an indirect reference to some other ambiguous subject. In this indirect type of ambiguity example, a first analysis of a scanned scene may identify a book as an object in the scene. A first user input including a viewer's gaze on the book triggers a request for a second analysis of the image of the book, and the second analysis may return information about the book indicating the title and author of the book. Later, when a user command for "what is this" or "who wrote that book" is received, perhaps even after the book has left the scene, a command processor may identify the indirect type ambiguity in the command, which may resolve the ambiguity by examining the most recently triggered object to select the best match for the indirect subject of the current ambiguous command. The command may then be executed by answering the user's question using the information from the second analysis indicating the title and author of the book.

[0033] In one or more aspects, a user command may be received via a language input from a user, such as a spoken or typed word. A command processor may, for example, use a natural language processor to analyze the command. The natural language processor may identify one or more ambiguities in the command, and may attempt to resolve the ambiguity based on one or more triggered objects that have been triggered by the first user input, such as when a first user input is received within a threshold time difference from the time when the ambiguous user command is received. In one aspect, a question from a user may be interpreted as a command to answer the question. In one aspect, the ambiguity in the user command may be resolved using the triggered object information generated by the second analysis. In another aspect, after resolving the ambiguity in the user command, the triggered object information may be used to execute the user command.

[0034] In one or more aspects, the time difference between receiving the first user input and receiving the user command may allow the triggered object to be pre-fetched by triggering a request for information about the triggered object before receiving the user command. Such pre-fetching of one or more triggered objects before receiving the command can reduce the delay in processing and / or executing the user command when the user command is received. In one aspect, the triggered objects may constitute only a subset of the identified objects. For example, hundreds of objects may be identified in the user's proximity area (box 304), and one object is triggered by the first user input. By requesting only additional information for the object triggered by the first user input (box 308), the delay for receiving the information can be reduced because fewer objects need to be analyzed in response to the request for additional information. This delay reduction may be significant when the processing requirements of the second analysis are large compared to the processing requirements of the first analysis.

[0035] In one or more aspects, the scan of the proximity area may include an image of text, and the text may be identified as an object by the first analyzer. In one aspect, the first analysis of the text object may classify the object as text. In another aspect, the second analysis of the text object may include performing text recognition on the image containing the text to produce a computer-understandable version of the text, which may be received as information about the text image object in block 310.

[0036] Figure 4 An example computing device 400 that can be used to implement various aspects of the subject technology according to one or more specific implementations is illustrated. The computing device 400 can be any computing device or server and / or a portion thereof for generating the features and processes described above, including but not limited to a laptop computer, a smart phone, a tablet device, a wearable device such as goggles or glasses, earplugs or other audio devices, a housing for an audio device, etc. The computing device 400 may include various types of computer-readable media, as well as interfaces for various other types of computer-readable media. The computing device 400 includes a permanent storage device 402, a system memory 404 (and / or a buffer), an input device interface 406, an output device interface 408, a bus 410, a ROM 412, one or more processing units 414, one or more network interfaces 416, and / or subsets and variations thereof.

[0037] The bus 410 generally represents all system, peripheral, and chipset buses that communicatively connect the many internal devices of the computing device 400. In one or more specific implementations, the bus 410 communicatively connects the one or more processing units 414 with the ROM 412, the system memory 404, and the permanent storage device 402. The one or more processing units 414 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in the present subject matter. In different specific implementations, the one or more processing units 414 can be a single processor or a multi-core processor.

[0038] ROM 412 stores static data and instructions required by one or more processing units 414 and other modules of computing device 400. On the other hand, permanent storage device 402 can be a read-write memory device. Permanent storage device 402 can be a non-volatile memory unit that stores instructions and data even when computing device 400 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) can be used as permanent storage device 402.

[0039] In one or more specific implementations, a removable storage device (such as a floppy disk, a flash drive and its corresponding disk drive) can be used as the permanent storage device 402. Like the permanent storage device 402, the system memory 404 can be a read-write memory device. However, unlike the permanent storage device 402, the system memory 404 can be a volatile read-write memory, such as a random access memory. The system memory 404 can store any of the instructions and data that one or more processing units 414 may need at runtime. In one or more specific implementations, the processes disclosed in the subject matter are stored in the system memory 404, the permanent storage device 402, and / or the ROM 412. The one or more processing units 414 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more specific implementations.

[0040] Bus 410 is also connected to input device interface 406 and output device interface 408. Input device interface 406 enables a user to transmit information and select commands to computing device 400. Input devices that can be used with input device interface 406 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). Output device interface 408 may, for example, enable the display of images generated by computing device 400. Output devices that can be used with output device interface 408 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information.

[0041] One or more implementations may include a device that functions as both an input device and an output device, such as a touch screen. In these implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form, including acoustic, voice, or tactile input.

[0042] Finally, if Figure 4 As shown, bus 410 also couples computing device 400 to one or more networks and / or to one or more network nodes via one or more network interfaces 416. In this manner, computing device 400 may be part of a computer network such as a LAN, a wide area network ("WAN"), or an intranet, or may be part of one of a plurality of networks such as the Internet. Any or all components of computing device 400 may be used with the subject disclosure.

[0043] Implementations within the scope of the present disclosure may be implemented in part or in whole using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions. Tangible computer-readable storage media may also be non-transitory in nature.

[0044] Computer-readable storage media can be any storage media that can be read, written, or otherwise accessed by a general or special computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0045] In addition, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more specific implementations, the tangible computer-readable storage medium may be directly coupled to the computing device, while in other specific implementations, the tangible computer-readable storage medium may be indirectly coupled to the computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0046] Instructions can be directly executable, or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or can be implemented as high-level language instructions that can be compiled to produce executable or non-executable machine code. In addition, instructions can also be implemented as data, or can include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those skilled in the art, the details including but not limited to the number, structure, sequence and organization of instructions can be significantly different without changing the underlying logic, function, processing and output.

[0047] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0048] It is understood that the use of personally identifiable information should be subject to privacy policies and practices that are generally recognized to meet or exceed industry or government requirements for maintaining user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of the authorized use should be clearly stated to users.

[0049] Those skilled in the art will recognize that the various illustrative frames, modules, elements, components, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, components, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application. Various components and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways) without departing from the scope of the subject technology.

[0050] It should be understood that the specific order or the hierarchical structure of the frame in the process disclosed by the present invention are the illustration of the example method. Based on the design preferred requirements, it should be understood that the specific order or the hierarchical structure of the frame in the process can be rearranged or all the illustrated frames are executed. Any frame in these frames can be executed simultaneously. In one or more specific implementations, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the specific implementation described above should not be understood as requiring such division in all specific implementations, and it should be understood that the program component (e.g., computer program product) and the system can be generally integrated together in a single software product or encapsulated in multiple software products.

[0051] As used in this specification and any claims of this patent application, the terms "base station", "receiver", "computer", "server", "processor" and "memory" all refer to electronic devices or other technical devices. These terms exclude people or groups of people. For the purpose of this specification, the term "display" or "displaying" means displaying on an electronic device.

[0052] As used herein, the phrase "at least one of" following a list of items, any of which is separated by the terms "and" or "or", modifies the list as a whole, rather than each member of the list (i.e., each item). The phrase "at least one of" does not require selection of at least one of each item listed; rather, the phrase allows for a meaning that includes at least one of any one item and / or at least one of any combination of items and / or at least one of each item. For example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" each refers to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0053] The predicate words "configured to," "operable to," and "programmed to" do not imply any specific tangible or intangible modification of a subject matter but are intended to be used interchangeably. In one or more specific implementations, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Similarly, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0054] Phrases such as aspect, this aspect, on the other hand, some aspects, one or more aspects, a specific implementation, this specific implementation, another specific implementation, some specific implementations, one or more specific implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, configuration, this configuration, other configurations, some configurations, one or more configurations, subject technology, disclosure, the present disclosure, other variations thereof, etc. are for convenience and do not mean that the disclosure involving such one or more phrases is essential to the subject technology, nor does it mean that such disclosure applies to all configurations of the subject technology. Disclosures involving such one or more phrases may apply to all configurations or one or more configurations. Disclosures involving such one or more phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to the other aforementioned phrases.

[0055] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. In addition, to the extent that the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive in a manner similar to how the term "comprising" is interpreted when used as a transitional word in a claim.

[0056] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be made available to the public regardless of whether such disclosure is explicitly stated in the claims. No claim element shall be interpreted under the provisions of 35 U.S.C. §112(f) unless the element is explicitly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “step for…”

[0057] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects are apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein the reference to the singular element is not intended to mean "only one", but to refer to "one or more", unless specifically noted. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used only for convenience and do not limit the subject disclosure.

Claims

1. A method, comprising: scanning the proximity area with at least one sensor; first analyzing the scan of the proximity region to identify objects in the proximity region; detecting a trigger corresponding to at least one of the identified objects based on a first user input regarding the at least one of the identified objects; In response to said detecting of said trigger, sending a request for first information about the at least one of the identified objects, and receiving, in response to the request, information about the at least one of the identified objects; receiving a command from a user, wherein the command includes an ambiguity; resolving the ambiguity in the command based on the information about the at least one of the identified objects; The command is executed based at least in part on the resolved ambiguity. 2 . The method of claim 1 , wherein the first user input is received via a first user input modality and the command is received via a second user input modality different from the first user input modality. The method of claim 2 , wherein the first user input modality is a passive user input modality. 4 . The method of claim 1 , wherein the ambiguity in the command comprises an implicit reference to one or more of the at least one of the identified objects. 5 . The method of claim 1 , wherein the ambiguity in the command comprises an indirect reference to one or more of the at least one of the identified objects.

6. The method according to claim 1, wherein: The first analysis also identifies a classification of the identified object; and The sending the request for information includes the classification.

7. A method according to claim 1, wherein the scan of at least one of the identified objects includes a text image, the first analysis classifies at least one of the identified objects as including a text image, and the received information includes recognized text corresponding to the text image. 8 . The method of claim 1 , wherein the first user input regarding the at least one of the identified objects comprises a gaze of the user on the at least one of the identified objects for a predetermined amount of time. 9 . The method of claim 1 , wherein the first user input regarding the at least one of the identified objects comprises a gesture related to the at least one of the identified objects.

10. A method according to claim 1, wherein the first analysis is based on a first image of the proximity area, and the sending of the request includes sending a second image of the at least one of the identified objects, wherein the first image includes the at least one of the identified objects at a first resolution, and the second image includes the at least one of the identified objects at a second resolution higher than the first resolution.

11. The method of claim 1, wherein the command is received verbally from the user.

12. The method of claim 1, wherein the command is a question and executing the command comprises answering the question.

13. The method of claim 1, wherein the processing of the command includes determining second information based on the first information, and the processing of the command is based on the second information.

14. A device, comprising: Memory; and at least one processor, the at least one processor being configured to: scanning the proximity area with at least one sensor; first analyzing the scan of the proximity region to identify objects in the proximity region; detecting a trigger corresponding to at least one of the identified objects based on a first user input regarding the at least one of the identified objects; In response to said detecting of said trigger, sending a request for information about the at least one of the identified objects, and receiving, in response to the request, information about the at least one of the identified objects; receiving a command from a user, wherein the command includes an ambiguity; resolving the ambiguity in the command based on the information about the at least one of the identified objects; The command is executed based at least in part on the resolved ambiguity.

15. The apparatus of claim 14, wherein: The first analysis also identifies a classification of the identified object; and The sending the request for information includes the classification.

16. A device according to claim 14, wherein the scan of at least one of the identified objects includes a text image, the first analysis classifies the at least one of the identified objects as including a text image, and the received information includes recognized text corresponding to the text image. 17 . The apparatus of claim 14 , wherein the first user input regarding the at least one of the identified objects comprises a gaze of the user on the at least one of the identified objects for a predetermined amount of time.

18. A non-transitory computer readable memory comprising instructions that, when executed by a processor, cause the processor to: scanning the proximity area with at least one sensor; first analyzing the scan of the proximity region to identify objects in the proximity region; detecting a trigger corresponding to at least one of the identified objects based on a first user input regarding the at least one of the identified objects; In response to said detecting of said trigger, sending a request for information about the at least one of the identified objects, and receiving, in response to the request, information about the at least one of the identified objects; receiving a command from a user, wherein the command includes an ambiguity; resolving the ambiguity in the command based on the information about the at least one of the identified objects; The command is executed based at least in part on the resolved ambiguity.

19. The non-transitory computer readable memory of claim 18, wherein: The first analysis also identifies a classification of the identified object; and The sending the request for information includes the classification.

20. A non-transitory computer-readable memory according to claim 18, wherein the scan of at least one of the identified objects includes a text image, the first analysis classifies at least one of the identified objects as including a text image, and the received information includes recognized text corresponding to the text image.

21. The non-transitory computer readable memory of claim 18, wherein the first user input regarding the at least one of the identified objects comprises a gaze of the user on the at least one of the identified objects for a predetermined amount of time.

Images