Thumbnail for disambiguation in visual search
By combining sensor data and camera images, electronic devices select and present subset thumbnails of the physical environment during visual search, solving the problem that users have difficulty determining the target object and improving the accuracy and user experience of visual search.
Patent Information
- Application Number
- CN202410626351.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-04-19
- Filing Date
- 2024-05-20
- Publication Date
- 2024-11-22
AI Technical Summary
It is difficult for existing electronic devices to accurately distinguish the target physical objects of the user during visual search, resulting in the user being unable to determine the focus of the visual search if there is insufficient clues provided.
By combining sensor data such as gaze detection information, head posture information and depth information, select a subset of the physical environment used for visual search, and use the camera to capture images, crop and present thumbnails to disambiguate the ambiguity, present on the display with Visual search results associated with physical objects.
Effectively eliminates the ambiguity when users select targets among multiple physical objects, improves the accuracy and user experience of visual search, and ensures that users can quickly identify the objects of interest.
Smart Images

Figure CN119007276A_ABST
Abstract
Description
[0001] This application claims priority to U.S. Patent Application No. 18 / 640,553 filed on April 19, 2024, which claims the benefit of U.S. Provisional Patent Application No. 63 / 503,616 filed on May 22, 2023, which applications are hereby incorporated by reference in their entirety. Background Art
[0002] The present disclosure relates generally to electronic devices, and more particularly to electronic devices having cameras.
[0003] Some electronic devices (such as head-mounted devices) use cameras to perform visual searches for nearby physical objects. However, it may be difficult to distinguish which physical object is the target of the visual search. Summary of the invention
[0004] An electronic device may include one or more sensors, one or more displays, one or more processors, and a memory storing instructions configured to be executed by the one or more processors for selecting a subset of a physical environment for visual search, obtaining an image of the subset of the physical environment via a first subset of the one or more sensors, and presenting thumbnails of the images including the subset of the physical environment using the one or more displays. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] Figure 1 is a schematic diagram of an illustrative electronic device according to some embodiments.
[0006] Figure 2A is a view of a physical object through a display of an illustrative electronic device according to some embodiments.
[0007] Figure 2B is a view of a physical object and visual search results associated with the physical object presented on a display of an illustrative electronic device according to some embodiments.
[0008] Figure 3 is a flowchart showing an illustrative method for operating an electronic device for performing a visual search and presenting content associated with the visual search according to some embodiments. DETAILED DESCRIPTION
[0009] Head-mounted devices can display different types of extended reality content for users. Head-mounted devices can display virtual objects perceived at an apparent depth within the user's physical environment. Virtual objects can sometimes be displayed at a fixed location relative to the user's physical environment. For example, consider an example in which the user's physical environment includes a table. A virtual object can be displayed to the user so that the virtual object appears to be resting on the table. As the user moves their head and otherwise interacts with the XR environment, the virtual object remains in the same fixed location on the table (e.g., as if the virtual object were another physical object in the XR environment). This type of content can be referred to as world-locked content (because the position of the virtual object is fixed relative to the user's physical environment).
[0010] Other virtual objects may be displayed at locations defined relative to the head-mounted device or the user of the head-mounted device. First, consider an example of a virtual object displayed at a location defined relative to the head-mounted device. When the head-mounted device moves (e.g., as the user's head rotates), the virtual object remains in a fixed position relative to the head-mounted device. For example, a virtual object may be displayed at a specific distance in front and in the center of the head-mounted device (e.g., in the center of the device field of view or the user's field of view). When the user moves their head to the left and right, their field of view of their physical environment changes accordingly. However, when the user moves their head, the virtual object may remain fixed at a specific distance in the center of the device field of view or the user's field of view (assuming that the gaze direction remains constant). This type of content may be referred to as head-locked content. The head-locked content is fixed in a given position relative to the head-mounted device (and therefore relative to the user's head supporting the head-mounted device). The head-locked content may not be adjusted based on the user's gaze direction. In other words, if the user's head position remains constant and their gaze is directed away from the head-locked content, the head-locked content will remain in the same apparent position.
[0011] Secondly, consider an example of a virtual object displayed at a position defined relative to a portion of the user of the head-mounted device (e.g., relative to the user's torso). This type of content may be referred to as body-locked content. For example, a virtual object may be displayed in front and to the left of the user's body (e.g., at a position defined by a distance and an angular offset from the forward direction of the user's torso), regardless of which direction the user's head is facing. If the user's body is facing a first direction, the virtual object will be displayed in front and to the left of the user's body. When facing the first direction, regardless of whether the user rotates their head left and right (to view toward and away from the virtual object), the virtual object may remain at the same fixed position relative to the user's body in the XR environment. However, the virtual object may move within the device field of view or the user's field of view in response to the user rotating their head. If the user turns around and their body faces a second direction opposite to the first direction, the virtual object will be repositioned within the XR environment so that the virtual object is still displayed in front and to the left of the user's body. When facing the second direction, regardless of whether the user rotates their head left and right (to view toward and away from the virtual object), the virtual object may remain at the same fixed position relative to the user's body in the XR environment.
[0012] In the aforementioned examples, body-locked content is displayed at a fixed position / orientation relative to the user's body, even when the user's body rotates. For example, a virtual object may be displayed at a fixed distance in front of the user's body. If the user is facing north, the virtual object is a fixed distance in front of the user's body (to the north). If the user turns around and faces south, the virtual object is a fixed distance in front of the user's body (to the south).
[0013] Alternatively, the distance offset between the body-locked content and the user can be fixed relative to the user, while the orientation of the body-locked content can remain fixed relative to the physical environment. For example, when the user faces north, the virtual object can be displayed in front of the user's body at a fixed distance from the user. If the user turns around and faces south, the virtual object remains north of the user's body at a fixed distance from the user's body.
[0014] The body locked content is also configured to always maintain gravity or horizontal alignment so that head and / or body changes in roll orientation will not cause the body locked content to move within the XR environment. Translational movement can cause the body locked content to be repositioned within the XR environment to maintain a fixed distance from the user. The subsequent description of body locked content may include both types of body locked content described above.
[0015] Figure 1 A schematic diagram of an illustrative electronic device is shown. Figure 1As shown, the electronic device 10 (sometimes referred to as a head-mounted device 10, a system 10, a head-mounted display 10, etc.) may have a control circuit 14. In addition to being a head-mounted device, the electronic device 10 may also be other types of electronic devices, such as a cellular phone, a laptop computer, a speaker, a computer monitor, an electronic watch, a tablet computer, etc.
[0016] The control circuit 14 can be configured to perform operations in the head mounted device 10 using hardware (e.g., dedicated hardware or circuitry), firmware, and / or software. Software code and other data for performing operations in the head mounted device 10 are stored on a non-transitory computer readable storage medium (e.g., a tangible computer readable storage medium) in the control circuit 14. Software code may sometimes be referred to as software, data, program instructions, instructions, or code. Non-transitory computer readable storage media (sometimes commonly referred to as memory) may include non-volatile memory such as non-volatile random access memory (NVRAM), one or more hard disk drives (e.g., disk drives or solid-state drives), one or more removable flash drives, or other removable media, etc. The software stored on the non-transitory computer readable storage medium may be executed on a processing circuit of the control circuit 14. The processing circuit may include a dedicated integrated circuit having a processing circuit, one or more microprocessors, a digital signal processor, a graphics processing unit, a central processing unit, or other processing circuits.
[0017] The head mounted device 10 may include input-output circuitry 16. The input-output circuitry 16 may be used to allow a user to provide user input to the head mounted device 10. The input-output circuitry 16 may also be used to gather information about the environment in which the head mounted device 10 is operating. Output components in the circuitry 16 may allow the head mounted device 10 to provide output to the user.
[0018] like Figure 1As shown, the input-output circuit 16 may include a display, such as a display 18. The display 18 may be used to display images for a user of the head mounted device 10. The display 18 may be a transparent or translucent display so that the user can observe physical objects through the display, while the computer generated content is superimposed on top of the physical object by presenting the computer generated image on the display. The transparent or translucent display may be formed by a transparent or translucent pixel array (e.g., a transparent organic light emitting diode display panel), or may be formed by a display device that provides an image to the user through a transparent structure such as a beam splitter, a holographic coupler, or other optical coupler (e.g., a display device such as a liquid crystal on a silicon display). Alternatively, the display 18 may be an opaque display that blocks light from the physical object when the user operates the head mounted device 10. In this type of arrangement, a perspective camera may be used to display the physical object to the user. The perspective camera may capture an image of the physical environment, and the physical environment image may be displayed on the display for viewing by the user. Additional computer-generated content (e.g., text, game content, other visual content, etc.) can optionally be superimposed on top of the image of the physical environment to provide the user with an extended reality environment. When the display 18 is opaque, the display can also optionally display the entire computer-generated content (e.g., without displaying an image of the physical environment).
[0019] The display 18 may include one or more optical systems (e.g., lenses) (sometimes referred to as optical components) that allow an observer to view images on the display 18. A single display 18 may produce images for both eyes, or a pair of displays 18 may be used to display images. In a configuration with multiple displays (e.g., a left-eye display and a right-eye display), the focal length and position of the lenses may be selected so that any gaps that exist between the displays will not be visible to the user (e.g., so that the images of the left and right displays overlap or merge seamlessly). A display module (sometimes referred to as a display component) that generates different images for the left and right eyes of a user may be referred to as a stereoscopic display. A stereoscopic display may be able to present two-dimensional content (e.g., a user notification with text) and three-dimensional content (e.g., a simulation of a physical object such as a cube).
[0020] Display 18 may include an organic light emitting diode display or other display based on an array of light emitting diodes, a liquid crystal display, a liquid crystal on silicon display, a projector or a display based on projecting a light beam directly or indirectly onto a surface through specialized optical devices (e.g., a digital micromirror device), an electrophoretic display, a plasma display, an electrowetting display, or any other desired display.
[0021] Input-output circuitry 16 may include various other input-output devices. For example, input-output circuitry 16 may include one or more speakers 20 configured to play audio, and one or more microphones 30 configured to capture audio data from a user and / or from the physical environment surrounding the user.
[0022] Input-output circuitry 16 may include one or more cameras 22. Cameras 22 may include one or more outward-facing cameras (as an example, when the electronic device is mounted on the user's head, the camera faces the physical environment around the user). Cameras 22 may capture visible light images, infrared images, or any other desired type of image. If desired, the camera may be a stereo camera. Outward-facing cameras may capture perspective video for device 10. Cameras 22 may also include inward-facing cameras (e.g., for gaze detection).
[0023] like Figure 1 As shown, the input-output circuit 16 may include a position and motion sensor 24 (e.g., a compass, a gyroscope, an accelerometer, and / or other device for monitoring the location, orientation, and motion of the electronic device 10, a satellite navigation system circuit such as a global positioning system circuit for monitoring the user's location, etc.). For example, the control circuit 14 may use the sensor 24 to monitor the current direction of the user's head relative to the surrounding environment (e.g., the user's head pose). The camera in the camera 22 may also be considered to be part of the position and motion sensor 24. The camera may be used for facial tracking (e.g., by capturing images of the user's chin, mouth, etc. when the device is worn on the user's head), body tracking (e.g., by capturing images of the user's torso, arms, hands, legs, etc. when the device is worn on the user's head) and / or for positioning (e.g., using visual odometry, visual inertial odometry, or other simultaneous localization and mapping (SLAM) techniques).
[0024] The input-output circuit 16 may include a gaze tracking sensor 26 (sometimes referred to as a gaze tracker 26, a gaze tracking system 26, a gaze detection sensor 26, etc.). The gaze tracking sensor 26 may include a camera and / or other gaze tracking sensor components (e.g., a light source that emits a light beam so that reflections of the light beam from the user's eye can be detected) to monitor the user's eye. The gaze tracker 26 may face the user's eye and may track the user's gaze. The camera in the gaze tracking system may determine the position of the user's eye (e.g., the center of the user's pupil), may determine the orientation direction of the user's eye (the direction in which the user is looking), may determine the user's pupil size (e.g., such that light modulation and / or other optical parameters, and / or sequential amounts used to spatially adjust one or more of these parameters, and / or areas in which one or more of these optical parameters are adjusted, these areas are adjusted based on pupil size), may be used to monitor the current focus of the lens in the user's eye (e.g., whether the user is focusing on the near field or the far field, which may be used to assess whether the user is daydreaming or thinking strategically or tactically) and / or other gaze information. The camera in the gaze tracking system may sometimes be referred to as an inward-facing camera, a gaze detection camera, an eye tracking camera, a gaze tracking camera, or an eye monitoring camera. Other types of image sensors (e.g., infrared and / or visible light emitting diodes and photodetectors, etc.) may also be used in monitoring the user's gaze if desired. The use of a gaze detection camera in the gaze tracker 26 is merely illustrative.
[0025] The input-output circuitry 16 may include one or more depth sensors 28. Each depth sensor may be a pixelated depth sensor (e.g., configured to measure multiple depths across a physical environment) or a point sensor (configured to measure a single depth in a physical environment). Each depth sensor (whether a pixelated depth sensor or a point sensor) may measure depth using phase detection (e.g., phase detection autofocus pixels) or light detection and ranging (LIDAR). Any combination of depth sensors may be used to determine the depth of a physical object in a physical environment.
[0026] The input-output circuit 16 may include a button 32. The button may include a mechanical switch that detects a user press during operation of the head mounted device. Alternatively, the button 32 may be a virtual button that uses touch sensing to detect a user press.
[0027] If desired, the input-output circuit 16 may also include other sensors and input-output components (e.g., ambient light sensors, force sensors, temperature sensors, touch sensors, capacitive proximity sensors, light-based proximity sensors, other proximity sensors, strain gauges, gas sensors, pressure sensors, humidity sensors, magnetic sensors, audio components, tactile output devices such as actuators and / or vibration motors, light-emitting diodes, other light sources, etc.).
[0028] The head mounted device 10 may also include communication circuitry 56 to allow the head mounted device to communicate with external equipment (e.g., a tethered computer, a portable device, one or more external servers, or other electrical equipment). The communication circuitry 56 may be used for both wired and wireless communication with external equipment.
[0029] The communication circuit 56 may include a radio frequency transceiver circuit formed by one or more integrated circuits, power amplifier circuits, low noise input amplifiers, passive radio frequency (RF) components, one or more antennas, transmission lines, and other circuits for processing RF wireless signals. Wireless signals may also be sent using light (e.g., using infrared communications).
[0030] For example, the radio frequency transceiver circuitry in the wireless communication circuitry 56 may process wireless local area network (WLAN) communication frequency bands such as 2.4 GHz and (IEEE802.11) frequency band; Wireless Personal Area Network (WPAN) communication frequency band, such as 2.4 GHz Communication bands; cellular telephone communication bands, such as cellular low band (LB) (e.g., 600 MHz to 960 MHz), cellular low-mid band (LMB) (e.g., 1400 MHz to 1550 MHz), cellular mid band (MB) (e.g., 1700 MHz to 2200 MHz), cellular high band (HB) (e.g., 2300 MHz to 2700 MHz), cellular ultra-high band (UHB) (e.g., 3300 MHz to 5000 MHz), or other cellular communication bands between about 600 MHz and about 5000 MHz (e.g., 3G bands, 4G LTE bands, 5G new radio frequency range 1 (FR1) bands below 10 GHz, etc.); near field communication (NFC) bands (e.g., 13.56 MHz); satellite navigation bands (e.g., L1 global positioning system (GPS) band of 1575 MHz, L5 GPS band, Global Navigation Satellite System (GLONASS) band, BeiDou Navigation Satellite System (BDS) band, etc.); ultra-wideband (UWB) communication band supported by IEEE802.15.4 protocol and / or other UWB communication protocols (for example, a first UWB communication band of 6.5 GHz and / or a second UWB communication band of 8.0 GHz); and / or any other required communication band.
[0031] The radio frequency transceiver circuitry may include millimeter wave / centimeter wave transceiver circuitry that supports communications at frequencies between about 10 GHz and 300 GHz. For example, the millimeter wave / centimeter wave transceiver circuitry may support communications in an extremely high frequency (EHF) or millimeter wave communications band between about 30 GHz and 300 GHz and / or in a centimeter wave communications band between about 10 GHz and 30 GHz (sometimes referred to as a super high frequency (SHF) band). For example, the millimeter wave / centimeter wave transceiver circuitry may support communications in the following communications bands: the IEEE K communications band between about 18 GHz and 27 GHz, the K communications band between about 26.5 GHz and 40 ... a Communications band, K between approximately 12 GHz and 18 GHz u The mmWave / cmWave transceiver circuit may support IEEE 802.11ad communications at 60 GHz (e.g., WiGig or 60 GHz Wi-Fi bands of approximately 57 GHz to 61 GHz) and / or 5th generation mobile networks or 5th generation wireless systems (5G) New Radio (NR) frequency range 2 (FR2) communications bands between approximately 24 GHz and 90 GHz, if desired.
[0032] The antenna in the wireless communication circuit 56 may include an antenna having a resonant element formed by a loop antenna structure, a patch antenna structure, an inverted F-shaped antenna structure, a slot antenna structure, a planar inverted F-shaped antenna structure, a spiral antenna structure, a dipole antenna structure, a monopole antenna structure, a combination of these designs, etc. Different types of antennas can be used for different frequency bands and frequency band combinations. For example, one type of antenna may be used in forming a local wireless link, and another type of antenna may be used in forming a remote wireless link antenna.
[0033] During operation, the head mounted device 10 can use the communication circuit 56 to communicate with one or more external servers 60 via a network 58. Examples of the communication network 58 include a local area network (LAN) and a wide area network (WAN) (e.g., the Internet). The one or more communication networks can be implemented using any known network protocol, including various wired or wireless protocols, such as, for example, Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.
[0034] The external server 60 may be implemented on one or more stand-alone data processing devices or a distributed network of computers. The external server 60 may provide information (such as visual search information) to the head mounted device 10 (via the network 58) in response to information from the head mounted device 10.
[0035] The head-mounted device 10 can communicate with an external server 60 to obtain visual search information. For example, the head-mounted device can send a visual search request to the external server 60 in response to user input. The visual search request can include various information for identifying physical objects near the head-mounted device (e.g., within the user's field of view). The information transmitted by the head-mounted device for identifying the physical object can include images from one or more cameras on the head-mounted device, feature point information, depth information, and / or other desired information. The external server can compare the received information with a database and identify whether there is a match with a physical object in the database. When there is a match, the external server can send information about the physical object (sometimes referred to as visual search information) to the head-mounted device. The head-mounted device can then present content to the user based on the visual search information.
[0036] The example of communicating with the external server 60 to perform a visual search is merely illustrative. If desired, the head mounted device 10 can perform a visual search using information stored on the head mounted device 10 (eg, stored in a memory of the control circuit 14).
[0037] The head mounted device 10 may be used to perform a visual search for physical objects in the physical environment surrounding the head mounted device. Figure 2A 6 is a view of physical objects 62-1, 62-2, and 62-3 as seen through display 18 in head mounted device 10. A user may request a visual search (e.g., by providing user input to an input device such as a microphone and / or buttons) while viewing physical objects 62-1, 62-2, and 62-3 through display 18. In response to the request for a visual search, head mounted device 10 may perform a visual search.
[0038] Sensor data from one or more sensors in the head mounted device 10 can be used to determine which physical object is the target of the user's visual search request. For example, the head mounted device 10 can use gaze detection information (e.g., the user may tend to search areas that overlap with their gaze point), head pose information (e.g., the user may tend to search areas centered on their field of view), depth information (e.g., by identifying physical objects in a normal depth range for visual search), user input, images from one or more cameras, etc. to select a subset of the physical environment for visual search.
[0039] Sensor data, results of a visual search for one or more physical objects, and / or user input by a user may be used to determine a hierarchy of most likely targets for the visual search.
[0040] Consider a case where an observer provides a voice command specifying "perform a visual search for this object" or "what is this object?" In this case, the user has not provided a clue as to which of physical objects 62-1, 62-2, and 62-3 is the target of the visual search. Continuing with this example, the user may be looking at physical object 62-1 while issuing the voice command. Therefore, physical object 62-1 may be determined to be the most likely target of the visual search, and physical object 62-2 may be determined to be the second most likely target of the visual search.
[0041] For another example, the user may point to physical object 62-1 (e.g., may perform a gesture to identify physical object 62-1) while issuing a voice command (or shortly before / after issuing a voice command). As a result, physical object 62-1 may be determined to be the most likely target of the visual search, and physical object 62-2 may be determined to be the second most likely target of the visual search.
[0042] For another example, the user's head posture may be centered on physical object 62-2 when issuing a voice command. Therefore, physical object 62-2 may be determined as the most likely target of the visual search.
[0043] In some cases, the user may provide additional information that disambiguates between candidates for the visual search. Consider a case where the observer provides a voice command specifying "perform a visual search for a red object" or "what is that red object?" If physical object 62-1 is red, physical object 62-2 is green, and physical object 62-3 is blue, then physical object 62-1 may be determined to be the most likely target for the visual search.
[0044] Image processing filters can be applied to images from camera 22 to identify candidate physical objects for visual search. Image processing filters can be applied to changing parts of images from camera 22. As an example, a first image processing filter can be applied to the entire image from camera 22. A second image processing filter can be applied to half of the image from camera 22 at a time. A third image processing filter can be applied to a small portion (e.g., less than half) of the image from camera 22 at a time. The first image processing filter can detect large objects within the user's field of view, the second image processing filter can detect medium-sized objects within the user's field of view, and the third image processing filter can detect small objects within the user's field of view. In one possible scenario, image processing filters can be used to identify candidate physical objects for visual search. Other input and / or sensor information (e.g., gaze detection information, head posture information, depth information, etc.) can be used to determine which of the candidate objects is most likely to be the user's desired target for visual search.
[0045] To clearly show the user which object is the target of the visual search, thumbnail images of candidate physical objects may be cropped from the image from camera 22 and presented on display 18. By displaying the thumbnail images captured by camera 22, the thumbnail images will directly match physical objects in the user's physical environment, thereby eliminating ambiguity as to which physical object is the subject of the visual search.
[0046] Figure 2B A view of visual search results including thumbnail images associated with physical objects presented on an illustrative display of an electronic device. Figure 2B As shown, Figure 2A Similarly, physical objects 62-1, 62-2, and 62-3 are visible through display 18. When a user requests a visual search, these three physical objects may be candidates for the visual search.
[0047] exist Figure 2BIn the example of , physical object 62-1 is determined to be the most likely candidate subject for visual search, followed by physical object 62-2. Therefore, a thumbnail image 64-1 of physical object 62-1 is presented on display 18. A description 66-1 associated with physical object 62-1 may be presented adjacent to thumbnail 64-1.
[0048] The thumbnail 64-1 may be of a fixed size. The portion of the image captured by the camera 22 that includes the physical object 62-1 may be cropped and / or scaled (enlarged or reduced) to match the fixed size of the thumbnail. The thumbnail may have a rectangular shape with rounded corners or any other desired shape.
[0049] Description 66 - 1 may include a description of the identified physical object, web content associated with the identified physical object, notifications associated with the identified physical object, application-specific content associated with the identified physical object, and the like.
[0050] As previously mentioned, the second most likely subject of the visual search may be determined to be the physical object 62-2. Accordingly, a thumbnail image 64-2 of the physical object 62-2 is also presented on the display 18. A description 66-2 associated with the physical object 62-2 may be presented adjacent to the thumbnail 64-2.
[0051] The thumbnail 64-2 may be of a fixed size. The portion of the image captured by the camera 22 that includes the physical object 62-2 may be cropped and / or scaled (enlarged or reduced) to match the fixed size of the thumbnail. The thumbnail may have a rectangular shape with rounded corners or any other desired shape.
[0052] Description 66 - 2 may include a description of the identified physical object, web content associated with the identified physical object, notifications associated with the identified physical object, application-specific content associated with the identified physical object, and the like.
[0053] Figure 2BThe example of displaying two thumbnails in is merely illustrative. In general, any desired number of thumbnails 64 having visual search results (such as descriptions 66) may be displayed on the display 18. In addition, not all available thumbnails 64 need to be displayed simultaneously. For example, one or more thumbnails 64 associated with an object determined to be the most likely subject of the visual search may be displayed. In response to user input (e.g., voice command, gaze input, gesture, selection of affordance, etc.), the head-mounted device 10 may optionally present additional thumbnails 64 associated with other detected objects in descending order of likelihood. If the head-mounted device 10 only identifies one possible candidate physical object for the visual search, only one thumbnail may be displayed on the display 18.
[0054] In some examples, head mounted device 10 may additionally or alternatively present affordances for actions that may be performed on the object associated with thumbnail 64. For example, in addition to or in lieu of providing description 66 of the object associated with thumbnail 64, head mounted device 10 may provide an option to share thumbnail 64 and / or description 66 with another user, an option to purchase the physical object associated with thumbnail 64, etc.
[0055] When multiple thumbnails (each thumbnail corresponding to a different candidate physical object) are displayed on the display 18, each thumbnail can be the same size. Alternatively, the thumbnail for the candidate physical object that is considered the most likely target of the visual search can be larger than the remaining thumbnails. In one possible solution, the remaining thumbnails can all be the same size. In another possible solution, the thumbnails can gradually become smaller as the likelihood that the associated physical object is the target physical object decreases.
[0056] As an example, the thumbnail displayed on the display 18 may be head-locked content. Alternatively, the thumbnail may be world-locked content or body-locked content.
[0057] Here, a thumbnail may refer to a small or cropped version of a larger image of a user's physical environment, and / or a small or cropped image of a subset of the user's physical environment. The small version of a larger image of a user's physical environment may be a reduced version of a larger image of the user's physical environment and / or a cropped version of a larger image of the user's physical environment. As an example, each thumbnail may be a cropped version of a larger image of the user's physical environment, such that each thumbnail image includes a candidate physical object. As another example, each thumbnail may be a small image of a candidate physical object.
[0058] As specific examples, each thumbnail may occupy less than 50% of the total display area when presented using display 18, less than 30% of the total display area when presented using display 18, less than 20% of the total display area when presented using display 18, less than 10% of the total display area when presented using display 18, less than 5% of the total display area when presented using display 18, less than 3% of the total display area when presented using display 18, and so on.
[0059] When presented by display 18, the apparent size of each thumbnail may have an apparent length and an apparent width of less than 60 centimeters, less than 50 centimeters, less than 40 centimeters, less than 30 centimeters, less than 20 centimeters, less than 10 centimeters, less than 5 centimeters, less than 3 centimeters, between 3 centimeters and 60 centimeters, between 5 centimeters and 30 centimeters, etc.
[0060] In some cases, visual search results (such as thumbnails 64 and / or descriptions 66) may be presented on display 18 even when the user's target is a physical object that is not overlapped by the active area of display 18 (from the user's perspective). Consider an example where the active area of display 18 does not span the entire surface of display 18, such that head mounted device 10 may not overlay virtual content on all portions of the physical environment visible to the user through display 18. In addition, head mounted device 10 may include an outward-facing camera (e.g., camera 22) having a field of view greater than the field of view of the active area of display 18 and a gaze tracking sensor 26 that is capable of tracking the user's gaze in directions beyond the boundaries of the active area of display 18. In these examples, the user may gaze at a physical object and trigger a visual search for the physical object. Head mounted device 10 may determine that the user wishes to perform a visual search for the physical object based on the gaze point overlapping the physical object. However, the gaze point and / or the physical object may not overlap the active area of display 18. Therefore, display 18 may not overlay content directly over the physical object. However, the display may still present visual search results identifying the physical object. Presenting a thumbnail that includes the physical object is a useful way to confirm to the user that the correct physical object is being searched for without overlaying virtual content directly on the physical object.
[0061] Figure 3 1 is a flow chart showing an illustrative method for operating an electronic device that performs a visual search and presents content associated with the visual search. At block 102, the head mounted device 10 may obtain a user request for a visual search. The head mounted device 10 may receive a user input that signals an intent to perform a visual search. The user input may include audio input (e.g., a voice command), a button press, a gaze-based input, etc.
[0062] At block 104, the head mounted device 10 may select a subset of the physical environment for visual search. The subset of the physical environment may be selected for visual search based on gaze detection information (e.g., a user may tend to search areas that overlap with their gaze point), head pose information (e.g., a user may tend to search areas centered on their field of view), depth information (e.g., by identifying physical objects in a normal depth range for visual search), images from one or more cameras, and the like.
[0063] At block 106, the head mounted device 10 may obtain an image of the subset of the physical environment. The camera 22 may capture an image of the subset of the physical environment. The camera 22 may capture an image of only the subset of the physical environment, or the camera 22 may capture a larger image that includes the subset of the physical environment and other portions of the physical environment. The larger image may then be cropped if desired.
[0064] At block 108, a characteristic of the image of the subset of the physical environment may be analyzed. The characteristic may be blurriness, contrast, a type of content contained within the image (e.g., text, non-text, etc.), or another desired characteristic. At block 108, if it is determined that the characteristic is satisfactory (e.g., if the characteristic meets a criterion) (e.g., if the blurriness is below a threshold, if the contrast is above a threshold, if the type of content is non-text, etc.), the method may proceed to block 110. At block 110, if it is determined that the characteristic is unsatisfactory (e.g., if the characteristic fails to meet the criterion) (e.g., if the blurriness is above a threshold, if the contrast is below a threshold, if the type of content is text, etc.), the head mounted device may forgo presenting a thumbnail of the image that includes the subset of the physical environment.
[0065] At block 110, the head mounted device 10 may present a thumbnail of an image including a subset of the physical environment on the display 18. The image from the camera 22 may be cropped and / or scaled (zoomed in or out) to obtain the thumbnail. The thumbnail may be presented as head locked content.
[0066] At box 112, the head mounted device 10 can perform a visual search and can present visual search result information adjacent to the thumbnail using the display 18. The head mounted device 10 can perform a visual search using an image captured by the camera 22. The image can be analyzed and compared to a database stored on the head mounted device 10 and / or a database on one or more external servers 60. When text is identified in the subset of the physical environment, optical character recognition (OCR) can be performed. Thus, the visual search can identify one or more physical objects in the subset of the physical environment, can identify and recognize text in the subset of the physical environment, etc.
[0067] The visual search result information displayed at block 112 may include information identifying the physical object, a description of the identified physical object, web content associated with the identified physical object, notifications associated with the identified physical object, application-specific content associated with the identified physical object, etc. The visual search result information may be presented as head-locked content, body-locked content, and / or world-locked content.
[0068] If it is determined at block 108 that the characteristics of the image of the subset of the physical environment are unsatisfactory, then photographs associated with the physical objects identified in the visual search may be presented on the display 18. The photographs may be stored in the head mounted device 10 and / or obtained from an external database 60. The photographs may be presented as thumbnails or using any other desired format.
[0069] As a specific example, at box 102, a user wearing the head mounted device 10 may view an indoor plant and provide an audio command that triggers a visual search. At box 104, due to one or more image processing filters identifying the indoor plant as a candidate physical object for visual search, the user's gaze point overlapping with the indoor plant, a gesture (e.g., pointing) from the user identifying the indoor plant, and / or the indoor plant being aligned with the center of the user's field of view determined using the user's head posture, the head mounted device 10 selects a subset of the physical environment including the indoor plant for visual search. At box 106, the camera 22 may capture an image of the subset of the physical environment including the indoor plant. The image of the subject of the physical environment may include or be derived from the same image or a different image as the image used to select the subject of the physical environment at box 104. At box 108, the head mounted device 10 analyzes the characteristics (e.g., blurriness) of the image of the subset of the physical environment. In response to determining that the characteristics are satisfactory (e.g., the blurriness of the image of the subset of the physical environment is sufficiently low), at box 110, the head mounted device 10 presents a thumbnail of the image including the subset of the physical environment on the display 18. The image of the subset of the physical environment may be cropped and / or scaled (enlarged or reduced) to obtain the thumbnail. At box 112, visual search result information associated with indoor plants may be presented adjacent to the thumbnail. The visual search result information may include the name of the type of indoor plant, a description of the indoor plant, and a link to a website with instructions on how to care for the indoor plant.
[0070] In response to determining at block 108 that the characteristic is unsatisfactory (e.g., the blurriness of the image of the subset of the physical environment is too high), at block 110, the head mounted device 10 may forgo presenting a thumbnail of the image including the subset of the physical environment on the display 18. Instead, in response to determining at block 108 that the characteristic is unsatisfactory, the head mounted device 10 may present a photo of an indoor plant identified in the subset of the physical environment. The photo of the indoor plant identified in the subset of the physical environment may be obtained using the Internet. Other visual search result information associated with the indoor plant may be presented adjacent to the photo in the scene.
[0071] As another example, at box 102, a user wearing the head mounted device 10 may view a menu and provide an audio command that triggers a visual search. At box 104, due to one or more image processing filters identifying the dish name as a candidate physical object for visual search, the user's gaze point overlapping the dish name, a gesture (e.g., pointing) from the user identifying the dish name, and / or the dish name being aligned with the center of the user's field of view determined using the user's head posture, the head mounted device 10 selects a subset of the physical environment (e.g., a portion of the menu corresponding to the dish name) for visual search. At box 106, the camera 22 may capture an image of a subset of the physical environment including the dish name. The image of the main body of the physical environment may include or be derived from the same image or a different image as the image used to select the main body of the physical environment at box 104. At box 108, the head mounted device 10 analyzes the characteristics of the image of the subset of the physical environment (e.g., the type of content). In response to determining that the characteristic is unsatisfactory (e.g., the image of the subset of the physical environment contains text), at block 110, the head mounted device 10 may forgo presenting a thumbnail of the image including the subset of the physical environment on the display 18. In this example, the head mounted device 10 may instead perform OCR on the text contained in the image, obtain an image corresponding to the text (e.g., an image of a dish from a local database or a remote database), and present the obtained image as a thumbnail on the display 18. At block 112, visual search result information associated with the name of the dish may be presented adjacent to the thumbnail. The size of the visual search result information may be larger than the size of the thumbnail.
[0072] Figure 3 An example is described in which a single thumbnail is displayed on the display 18. The subset of the physical environment selected for visual search at block 104 may be the most likely candidate for visual search. However, if desired, additional candidates may also be searched and presented for visual search. In general, the search may be repeated (or performed in parallel) for any desired number of subsets of the physical environment (with corresponding physical objects for visual search). Figure 3A method for presenting a physical environment in a manner that allows a user to select a desired target and a candidate for the user to select the desired target. A thumbnail may be presented for each subset of the physical environment, each thumbnail including an image of the corresponding subset of the physical environment. The thumbnails may all be the same size, or the thumbnails may be different sizes (e.g., decreasing in size as the likelihood that the candidate is the user's desired target decreases). The thumbnails may be presented as head-locked content.
[0073] As described above, one aspect of the present invention technology is to collect and use information such as sensor information. The present disclosure contemplates that, in some cases, data including personal information data may be collected that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, user name, password, biometric information, or any other identification or personal information.
[0074] The present disclosure recognizes that the use of such personal information in the present technology can be used to benefit users. For example, personal information data can be used to deliver targeted content that users are more interested in. Therefore, the use of such personal information data enables users to have control over the delivered content. In addition, the present disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into the user's overall health status, or can be used as positive feedback to individuals who use technology to pursue health goals.
[0075] The disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information data will comply with sound privacy policies and / or privacy measures. In particular, such entities should implement and adhere to the use of privacy policies and practices that are generally considered to meet or exceed the industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easy for users to access and should be updated as changes in the collection and / or use of data occur. Personal information from users should be collected for legal and reasonable entity purposes and should not be shared or sold outside these legal purposes. In addition, such collection / sharing should be carried out after receiving the informed consent of the user. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data, and ensure that other entities with access to personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and measures. In addition, policies and practices should be adapted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including considerations of special jurisdictions. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.
[0076] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates hardware elements and / or software elements to prevent or block access to such personal information data. For example, the technology of the present invention can be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for a service. For another example, a user can choose not to provide certain types of user data. For another example, a user can choose to limit the length of time that user-specific data is retained. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user can be notified that their personal information data will be accessed when downloading an application ("app"), and then the user is reminded again before the personal information data is accessed by the app.
[0077] Furthermore, it is the intent of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting the collection of data and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. Where appropriate, de-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or characteristics of stored data (e.g., collecting location data at a city level rather than an address level), controlling how the data is stored (e.g., aggregating data across users), and / or other methods.
[0078] Thus, while the present disclosure broadly encompasses the use of information that may include personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without requiring access to personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data.
[0079] According to an embodiment, an electronic device is provided, which includes one or more sensors, one or more displays, one or more processors, and a memory storing instructions configured to be executed by the one or more processors, the instructions being used to: select a subset of a physical environment for visual search; obtain an image of the subset of the physical environment via a first subset of the one or more sensors; and present a thumbnail of the image including the subset of the physical environment using the one or more displays.
[0080] According to another embodiment, the instructions include instructions for presenting the visual search result information adjacent to the thumbnails using one or more displays.
[0081] According to another embodiment, the instructions include instructions for: selecting an additional subset of the physical environment for additional visual search; obtaining additional images of the additional subset of the physical environment via a first subset of one or more sensors; presenting additional thumbnails including the additional images of the additional subset of the physical environment using one or more displays and while presenting the thumbnails; and presenting additional visual search result information adjacent to the thumbnails using one or more displays.
[0082] According to another embodiment, the instructions include instructions for obtaining a user request for a visual search via a second subset of the one or more sensors.
[0083] According to another embodiment, the instructions include instructions for: obtaining sensor data via a third subset of one or more sensors, and selecting the subset of the physical environment for visual search includes: selecting the subset of the physical environment for visual search based on the sensor data.
[0084] According to another embodiment, a third subset of one or more sensors includes an outward-facing camera configured to detect gestures, a gaze detection sensor configured to determine a gaze point, or an accelerometer configured to obtain head posture information.
[0085] According to another embodiment, one or more displays have an associated field of view, and a subset of the physical environment is outside the field of view.
[0086] According to another embodiment, the instructions include instructions for: analyzing a characteristic of an image of a subset of the physical environment; in response to determining that the characteristic fails to meet a criterion, forgoing presenting a thumbnail; and in response to determining that the characteristic fails to meet the criterion, presenting a photograph associated with a visual search of the subset of the physical environment that is different from the image of the subset of the physical environment obtained.
[0087] According to another embodiment, the instructions include instructions for analyzing a characteristic of the image of the subset of the physical environment, presenting the thumbnail includes presenting the thumbnail based on a determination that the characteristic satisfies a criterion.
[0088] According to an embodiment, a method of operating an electronic device, the electronic device comprising one or more sensors and one or more displays, the method comprising: selecting a subset of a physical environment for visual search; obtaining an image of the subset of the physical environment via a first subset of the one or more sensors; and presenting a thumbnail of the image comprising the subset of the physical environment using the one or more displays.
[0089] According to another embodiment, the method includes presenting visual search result information adjacent to the thumbnail using one or more displays.
[0090] According to another embodiment, the method includes: selecting an additional subset of the physical environment for additional visual search; obtaining additional images of the additional subset of the physical environment via a first subset of one or more sensors; presenting additional thumbnails including the additional images of the additional subset of the physical environment using one or more displays and at the same time as presenting the thumbnails; and presenting additional visual search result information adjacent to the additional thumbnails using one or more displays.
[0091] According to another embodiment, the method includes obtaining, via a second subset of the one or more sensors, a user request for a visual search.
[0092] According to another embodiment, the method includes obtaining sensor data via a third subset of the one or more sensors, and selecting the subset of the physical environment for visual search includes selecting the subset of the physical environment for visual search based on the sensor data.
[0093] According to another embodiment, a third subset of one or more sensors includes an outward-facing camera configured to detect gestures, a gaze detection sensor configured to determine a gaze point, or an accelerometer configured to obtain head posture information.
[0094] According to another embodiment, one or more displays have an associated field of view, and a subset of the physical environment is outside the field of view.
[0095] According to another embodiment, the method includes analyzing a characteristic of an image of a subset of the physical environment; in response to determining that the characteristic fails to meet a criterion, forgoing presenting a thumbnail; and in response to determining that the characteristic fails to meet the criterion, presenting a photo associated with a visual search of the subset of the physical environment that is different from the image of the subset of the physical environment obtained.
[0096] According to another embodiment, the method includes analyzing a characteristic of the image of the subset of the physical environment, and presenting the thumbnail includes presenting the thumbnail based on a determination that the characteristic satisfies a criterion.
[0097] According to an embodiment, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device including one or more sensors and one or more displays, the one or more programs including instructions for: selecting a subset of a physical environment for visual search; obtaining an image of the subset of the physical environment via a first subset of one or more sensors; and presenting thumbnails of the images of the subset of the physical environment using one or more displays.
[0098] According to another embodiment, the instructions include instructions for presenting the visual search result information adjacent to the thumbnails using one or more displays.
[0099] According to another embodiment, the instructions include instructions for: selecting an additional subset of the physical environment for additional visual search; obtaining additional images of the additional subset of the physical environment via a first subset of one or more sensors; presenting additional thumbnails including the additional images of the additional subset of the physical environment using one or more displays and while presenting the thumbnails; and presenting additional visual search result information adjacent to the thumbnails using one or more displays.
[0100] According to another embodiment, the instructions include instructions for obtaining a user request for a visual search via a second subset of the one or more sensors.
[0101] According to another embodiment, the instructions include instructions for: obtaining sensor data via a third subset of one or more sensors, and selecting the subset of the physical environment for visual search includes: selecting the subset of the physical environment for visual search based on the sensor data.
[0102] According to another embodiment, a third subset of one or more sensors includes an outward-facing camera configured to detect gestures, a gaze detection sensor configured to determine a gaze point, or an accelerometer configured to obtain head posture information.
[0103] According to another embodiment, one or more displays have an associated field of view, and a subset of the physical environment is outside the field of view.
[0104] According to another embodiment, the instructions include instructions for: analyzing a characteristic of an image of a subset of the physical environment; in response to determining that the characteristic fails to meet a criterion, forgoing presenting a thumbnail; and in response to determining that the characteristic fails to meet the criterion, presenting a photograph associated with a visual search of the subset of the physical environment that is different from the image of the subset of the physical environment obtained.
[0105] According to another embodiment, the instructions include instructions for analyzing a characteristic of the image of the subset of the physical environment, presenting the thumbnail includes presenting the thumbnail based on a determination that the characteristic satisfies a criterion.
[0106] The foregoing is merely illustrative, and various modifications may be made to the described embodiments. The foregoing embodiments may be implemented independently or in any combination.
Claims
1. An electronic device, the electronic device include: one or more sensors; one or more displays; one or more processors; and a memory storing instructions configured to be executed by the one or more processors, the instructions being used to: Select a subset of the physical environment for visual search; obtaining, via a first subset of the one or more sensors, an image of a subset of the physical environment; as well as Thumbnails of images comprising a subset of the physical environment are presented using the one or more displays.
2. The electronic device according to claim 1, wherein the instructions further comprise instructions for: Visual search result information is presented adjacent to the thumbnail using the one or more displays.
3. The electronic device according to claim 2, wherein the instructions further comprise instructions for: selecting an additional subset of the physical environment for additional visual search; obtaining, via the first subset of the one or more sensors, additional images of additional subsets of the physical environment; using the one or more displays and concurrently with presenting the thumbnails, presenting additional thumbnails comprising additional images of additional subsets of the physical environment; as well as Additional visual search result information is presented adjacent to the additional thumbnails using the one or more displays.
4. The electronic device according to claim 1, wherein the instructions further comprise instructions for: A user request for a visual search is obtained via a second subset of the one or more sensors.
5. The electronic device according to claim 1, wherein the instructions further comprise instructions for: obtaining sensor data via a third subset of the one or more sensors, wherein a subset of the physical environment is selected for the visual search include: A subset of the physical environment is selected for the visual search based on the sensor data.
6. An electronic device according to claim 5, wherein a third subset of the one or more sensors includes an outward-facing camera configured to detect gestures, a gaze detection sensor configured to determine a gaze point, or an accelerometer configured to obtain head posture information.
7. The electronic device of claim 1, wherein the one or more displays have an associated field of view, and wherein the subset of the physical environment is outside of the field of view.
8. The electronic device of claim 1, wherein the instructions further comprise instructions for: analyzing characteristics of the image of a subset of the physical environment; In response to determining that the characteristic fails to meet the criteria, abandoning presentation of the thumbnail; and In response to determining that the characteristic fails to satisfy a criterion, presenting a photograph associated with the visual search of the subset of the physical environment, the photograph being different from the obtained image of the subset of the physical environment.
9. The electronic device of claim 1, wherein the instructions further comprise instructions for: The image of the subset of the physical environment is analyzed for characteristics, wherein presenting the thumbnail includes presenting the thumbnail based on a determination that the characteristic satisfies a criterion.
10. A method of operating an electronic device, the electronic device comprising one or more sensors and one or more displays, the method include: Select a subset of the physical environment for visual search; obtaining, via a first subset of the one or more sensors, an image of a subset of the physical environment; as well as Thumbnails of images comprising a subset of the physical environment are presented using the one or more displays.
11. The method according to claim 10, further comprising: include: Visual search result information is presented adjacent to the thumbnail using the one or more displays.
12. The method according to claim 11, further comprising: include: selecting an additional subset of the physical environment for additional visual search; obtaining, via the first subset of the one or more sensors, additional images of additional subsets of the physical environment; using the one or more displays and concurrently with presenting the thumbnails, presenting additional thumbnails comprising additional images of additional subsets of the physical environment; as well as Additional visual search result information is presented adjacent to the additional thumbnails using the one or more displays.
13. The method according to claim 10, further comprising: include: A user request for a visual search is obtained via a second subset of the one or more sensors.
14. The method according to claim 10, further comprising: include: Sensor data is obtained via a third subset of the one or more sensors, wherein selecting a subset of the physical environment for the visual search comprises selecting the subset of the physical environment for the visual search based on the sensor data.
15. The method of claim 14, wherein a third subset of the one or more sensors comprises an outward-facing camera configured to detect gestures, a gaze detection sensor configured to determine a gaze point, or an accelerometer configured to obtain head posture information.
16. The method of claim 10, wherein the one or more displays have an associated field of view, and wherein the subset of the physical environment is outside of the field of view.
17. The method according to claim 10, further comprising: include: analyzing characteristics of the image of a subset of the physical environment; In response to determining that the characteristic fails to meet the criteria, abandoning presentation of the thumbnail; as well as In response to determining that the characteristic fails to satisfy a criterion, presenting a photograph associated with the visual search of the subset of the physical environment, the photograph being different from the obtained image of the subset of the physical environment.
18. The method according to claim 10, further comprising: include: The image of the subset of the physical environment is analyzed for characteristics, wherein presenting the thumbnail includes presenting the thumbnail based on a determination that the characteristic satisfies a criterion.
19. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device including one or more sensors and one or more displays, the one or more programs comprising instructions for: Select a subset of the physical environment for visual search; obtaining an image of a subset of the physical environment via a first subset of the one or more sensors; and Thumbnails of images comprising a subset of the physical environment are presented using the one or more displays.
20. The non-transitory computer-readable storage medium of claim 19, wherein the instructions further comprise instructions for: presenting visual search result information adjacent to the thumbnail using the one or more displays; selecting an additional subset of the physical environment for additional visual search; obtaining, via the first subset of the one or more sensors, additional images of additional subsets of the physical environment; using the one or more displays and concurrently with presenting the thumbnails, presenting additional thumbnails comprising additional images of additional subsets of the physical environment; as well as Additional visual search result information is presented adjacent to the additional thumbnails using the one or more displays.
