Selecting Real-World Objects Using a Wearable Device

Abstract

The method includes receiving an image from a sensor of the wearable device, rendering the image on a display of the wearable device, identifying a set of targets in the image, tracking a gaze direction associated with a user of the wearable device, rendering a gaze on the displayed image based on the tracked gaze direction, identifying a subset of targets based on the set of targets in a region of the image based on the gaze, triggering an action, and in response to the trigger, estimating candidate targets based on the subset of targets.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. Nonprovisional Patent Application No. 17 / 651,209, entitled "SELECTION OF REAL-WORLD OBJECTS USING A WEARABLE DEVICE," filed February 15, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] Embodiments relate to the selection of objects and / or object attributes (eg, text) using a wearable device. [Background technology]

[0003] Augmented reality (AR) devices (e.g., wearable devices) can be used to perform many actions to enhance user experience. One of these actions can be, for example, converting text. To perform some of these actions, the AR device selects virtual objects and / or real-world objects and / or object attributes (e.g., text) through user interaction as input for the action. Inability to accurately select virtual objects and / or real-world objects and / or object attributes can negatively impact the user experience. Summary of the Invention

[0004] In a general aspect, a device, system, non-transitory computer readable medium (having stored thereon computer executable program code that can be executed on a computer system), and / or method may perform a process by which the method includes receiving an image from a sensor of a wearable device, rendering the image on a display of the wearable device, identifying a set of targets in the image, tracking a gaze direction associated with a user of the wearable device, rendering a gaze on the displayed image based on the tracked gaze direction, identifying a subset of targets based on the set of targets in a region of the image based on the gaze, triggering an action, and in response to the trigger, inferring candidate targets based on the subset of targets.

[0005] In another general aspect, a wearable device includes an image sensor, a display, at least one processor, and at least one memory including computer program code configured to cause, using the at least one processor, the wearable device to: receive an image from the image sensor; render the image on the display; identify a set of targets within the image; track a gaze direction associated with a user of the wearable device; render a gaze on the displayed image based on the tracked gaze direction; identify a subset of targets based on the set of targets within a region of the image based on the gaze; trigger an action; and, in response to the trigger, estimate candidate targets based on the subset of targets.

[0006] Implementations may include one or more of the following features. For example, the method (and / or computer program code) may further include identifying a subset of targets based on a region encompassing the line of sight and estimating a depth associated with each target of the set of targets, where the estimation of candidate targets is based on an intersection of the line of sight at a depth encompassed by the region. The method (and / or computer program code) may further include detecting a change in the line of sight direction, determining that the change is less than a threshold, and re-rendering the image on a display of the wearable device. The method (and / or computer program code) may further include detecting a change in the line of sight direction, determining that the change is less than a threshold, and re-rendering the line of sight. The method (and / or computer program code) may further include detecting a change in the line of sight direction, determining that the change is in the rendered image and closer to the subset of targets, and re-rendering the line of sight with a change in color. The method (and / or computer program code) may further include detecting a change in the line of sight direction, determining that the change is greater than a threshold, and receiving another image from the sensor. The method (and / or computer program code) may further include rendering a reticle on the displayed image based on a position of the candidate target. The method may further include repositioning the reticle to a different position on the displayed image, and the candidate target is estimated based on the repositioned reticle. The method (and / or computer program code) may further include calibrating the wearable device based on a position of a sensor of the wearable device and a center of a display of the wearable device.

[0007] Exemplary embodiments will become more fully understood from the following detailed description herein and the accompanying drawings, in which like elements are indicated with like reference numerals and which are provided by way of example only and not by way of limitation of the exemplary embodiments. [Brief description of the drawings]

[0008] [Figure 1A] 1 illustrates a side perspective view of a user gazing at multiple objects in a real-world scene, according to an exemplary implementation. [Figure 1B] 1 illustrates a front perspective view of a user gazing at multiple objects in a real-world scene, according to an exemplary implementation. [Diagram 2] 1 illustrates a three-dimensional rendering of an image space, according to an exemplary embodiment. [Diagram 3] 1 illustrates a three-dimensional rendering of an image space, according to an exemplary embodiment. [Figure 4] FIG. 2 illustrates a block diagram of calibration data flow, according to an exemplary embodiment. [Diagram 5] FIG. 1 illustrates a block diagram of gaze tracking with target identification data flow, according to an exemplary embodiment. [Figure 6] 1 illustrates a block diagram of a method for identifying a target, according to an exemplary embodiment. [Figure 7] 1 illustrates a block diagram of a system according to an exemplary embodiment. [Figure 8] 1 illustrates an example of a computing device and a mobile computing device in accordance with at least one exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] It should be noted that these figures are intended to illustrate general features of methods, structures, and / or materials utilized in certain exemplary embodiments and to supplement the following description. However, these figures are not to scale, may not precisely reflect the exact structural or performance characteristics of any given embodiment, and should not be construed as defining or limiting the range of values ​​or properties encompassed by the exemplary embodiments. For example, the relative thicknesses and arrangements of molecules, layers, regions, and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various figures is intended to indicate the presence of similar or identical elements or features.

[0010] A user wearing a wearable device (e.g., smart glasses) may want to perform an action (e.g., transform certain text) based on a target (e.g., a road sign) or a portion of a target (e.g., a line of a multi-line road sign) in the user's environment. However, there may be multiple candidate targets (multiple road signs) or portions of candidate targets (e.g., multiple road signs) on which to perform an action (e.g., transformation).

[0011] The disclosed solutions allow a user to communicate to a wearable device (e.g., smart glasses) which targets or portions of targets the user intends to perform an action on. For example, the solutions may provide functionality that a user of a wearable device can use to communicate a selection of real-world targets, objects, and / or areas of interest in the real world to the wearable device.

[0012] The technical solution may include identifying a set of targets in an image captured using an image sensor of a wearable device. A line of sight of a user of the wearable device may then be used to determine a subset of targets based on a line of sight direction. A candidate target(s) may then be estimated and / or selected from the subset of targets when an action is triggered. Various techniques may be used to reduce the set of targets to the subset of targets and / or estimate the candidate target(s). For example, line of sight, a reticle, and / or some other visual tool may be used to focus or help focus the user's line of sight to limit the targets on which the user may intend to perform an action.

[0013] An advantage of this solution is that it improves the user experience, for example by minimizing the frustration of receiving erroneous information. A technical advantage may be that it reduces the usage of limited resources (processing, power, etc.) in the wearable device as a result of performing an action again due to getting an inaccurate or erroneous response from the action.

[0014] A user wearing a wearable device can see many targets (or objects) in a real-world scene. It may be difficult to determine which target and what of the targets the user is interested in through the wearable device. Figures 1A and 1B can be used to explain (and / or refer to) an exemplary embodiment for determining which of many targets are candidate targets. Figures 1A and 1B show how the embodiments described herein can address the difficulty of identifying which of many targets are candidate targets (or targets of interest).

[0015] FIG. 1A illustrates a side perspective view of a user gazing at multiple targets in a real-world scene according to an exemplary embodiment. FIG. 1A illustrates a user 105 wearing a wearable device 110 (e.g., an AR / VR device) and viewing a scene (e.g., a real-world scene including multiple targets 115, 120, 125, 130, 135). The multiple targets 115, 120, 125, 130, 135 are at depths D1, D2, D3, D4 (e.g., distances away from the wearable device 110). FIG. 1B illustrates a front perspective view of a user gazing at multiple targets in a real-world scene according to an exemplary embodiment. As shown in FIG. 1B, a real-world scene 140 includes multiple targets 115, 120, 125, 130, 135. Multiple targets 115, 120, 125, 130, 135 have associated texts: Text 1, Text 2, Text 3, Text 4, Text 5, Text 6, Text 7. Figures 1A and 1B show gaze directions GD1, GD2, GD3 as directions of vision (up, down, left, right) of a user 105 viewing a real-world scene 140.

[0016] With reference to Figure 1A, targets 115 and 120 are at depth D1, target 135 is at depth D2, target 125 is at depth D3, and target 130 is at depth D4. With reference to Figure 1B, targets 115 and 125 are on the left, targets 120 and 130 are on the right, and target 135 is between and overlaps targets 115 and 120 (note that target 135 is behind targets 115 and 120, as indicated by the dashed line).

[0017] With reference to FIG. 1A, GD1 is generally oriented upwards toward target 120 and target 115. With reference to FIG. 1B, GD1 is generally oriented leftward toward target 115 and target 125. Based on the orientation of GD1, the most likely region of the image representing the real-world scene 140 may be (or include) target 115. However, target 115 includes text 1, text 2, and text 3. Thus, target 115 may be within a region of the image representing the real-world scene 140, and a subset of targets may be identified as text 1, text 2, and text 3. In an exemplary implementation, a candidate target may be estimated based on the subset of targets, or may be estimated as one of text 1, text 2, and text 3. Using techniques described below, the subset of targets may be reduced and / or the candidate targets may be estimated. In other words, using the techniques described below, one of Text 1, Text 2, and Text 3 can be selected or estimated as a candidate target (e.g., a target of interest to the user 105 of the wearable device 110).

[0018] With reference to FIG. 1A, GD2 is typically slightly downward from the straight-on orientation toward targets 115, 120, 130, and target 135. With reference to FIG. 1B, GD2 is typically slightly right from the straight-on orientation toward targets 120 and 135. Based on the orientation of GD2, the most likely region of the image representing the real-world scene 140 may be (or include) targets 120 and 135, where target 120 includes text 4 and text 5. Thus, targets 120 and 135 may be within the region of the image representing the real-world scene 140, and a subset of targets may be identified as text 4, text 5, and text 8 (text 8 being within target 135). In an exemplary implementation, a candidate target may be estimated based on the subset of targets, or may be estimated as one of text 4, text 5, and text 8. Using techniques described below, the subset of targets may be reduced and / or the candidate targets may be estimated. In other words, using the techniques described below, one of text 4, text 5, and text 8 can be selected or inferred as a candidate target (e.g., a target of interest to user 105 of wearable device 110). Because target 120 is at depth D1 and target 135 is at depth D2, one technique for inferring a candidate target can be based on depth.

[0019] With reference to FIG. 1A, GD3 is generally downwards toward targets 130 and 135. With reference to FIG. 1B, GD3 is generally leftward toward targets 115 and 125. Based on the orientation of GD3, the most likely region of the image representing the real-world scene 140 may be (or include) target 125. Thus, target 125 may be within a region of the image representing the real-world scene 140, and a subset of targets may be identified as text 6. In an exemplary implementation, a candidate target may be estimated based on the subset of targets or may be estimated as text 6. Using techniques described below, the subset of targets may be reduced and / or candidate targets may be estimated. Because the subset of targets is a subset of 1, the most likely result may be estimating or selecting text 6 as a candidate target (e.g., a target of interest to user 105 of wearable device 110).

[0020] In the techniques described above, details about the image space can be used to determine useful information about line of sight, camera center and offset, object depth, and user input (e.g., via a reticle, head movement, etc.) Figures 2-5 show various image space details that are used to determine the useful information.

[0021] FIG. 2 illustrates a three-dimensional rendering of an image space according to an exemplary embodiment. The three-dimensional (3D) rendering can be based on the view frustum of the camera and the view frustum of the lens or screen. The view frustum is a frustum that determines what is in the field of view. Only objects within the range of the view frustum can be displayed on the screen and / or in the image. The camera (or eye) is at the tip of the pyramid. The pyramid extends in a direction away from the camera (or eye). The view frustum starts at the near face and ends at the far face. These faces are parallel and their normals are along the line of sight or camera direction (e.g., the direction the eye or camera is looking). The length of the view frustum is determined by the distance from the camera (or eye) to the near face and the distance from the camera to the far face. In an exemplary embodiment, the division into far field / near field can arise from the observation that the epipolar line of the user's line of sight as seen by the camera (e.g., the line from the eye to the image plane) is concentrated in a small region of the image space. In an exemplary wearable device use case, a far field object may be an object having a distance of more than 1 meter.

[0022] 2, there may be a camera (e.g., camera 250) view frustum 210, a lens (of the wearable device) view frustum 245-1, 245-2, and a screen (displayed on the lens of the wearable device) view frustum 220-1, 220-3. The camera view frustum 210 may have an associated epipolar line 230, and the screen view frustum 220-1, 220-3 may have an associated line of sight 235 (e.g., epipolar line).

[0023] An image 220-2 of a real-world scene 205 can be located in an image plane associated with a view frustum 220-1, 220-3 of a screen. The image 220-2 can include objects 225-1, 225-2, 225-3.

[0024] In an exemplary implementation, the lines of sight 235-1, 235-2 can be used to identify the line of sight directions (e.g., GD1, GD2, GD3). In FIG. 2, the lines of sight 235-1, 235-2 point to the object 225-1. Furthermore, there is no indication that the lines of sight 235-1, 235-2 point to any other objects (e.g., objects 225-2 and 225-3). Thus, in the example of FIG. 2, the object 225-1 may be or include a candidate target (e.g., a target of interest to the user 105 of the wearable device 110). If the object 225-1 includes several targets (e.g., text), an exemplary implementation can include the use of lines of sight, a reticle, and / or several other visual tools to help the user indicate which of the several targets are candidate targets. FIG. 3 can be used to explain the use of lines of sight and a reticle.

[0025] FIG. 3 illustrates a three-dimensional rendering of an image space, according to an exemplary implementation. As illustrated in FIG. 3, a screen 305 (e.g., as part of a lenticular display or lens of a wearable device) illustrates a line of sight 235 (representing either line of sight 235-1, 235-2). Line of sight 235 may be of fixed size dimensions as displayed on screen 305. Line of sight 235 may be located on (within, with, etc.) the rendered image with a first end at an edge of the rendered image and a second end near the center of the rendered image. Line of sight 235 may have a somewhat triangular or conical shape. Line of sight 235 may have a first end at or near the outer edge (e.g., the right-most edge) of the rendered image. The first end of line of sight 235 may be relatively longer than the second end. The second end of line of sight 235 may lead to a point that tapers from the first end of line of sight 235.

[0026] The line of sight 235 can be used to indicate the line of sight direction and can be used as a pointer to an object that may be a candidate target. The line of sight 235 can be displayed with multiple portions 325, 330, 335 of different colors. The portions 325, 330, 335 can be arranged along a longitudinal axis of the line of sight 235. The portions 325, 330, 335 can each have a length less than the total length of the line of sight 235. The portions 325, 330, 335 can taper along the longitudinal axis of the line of sight 235. The first portion 325 can be arranged at a second edge of the line of sight 235 and can include a point of the line of sight 235. The third portion 335 can be arranged at a second end of the line of sight 235 at an edge of the rendered image. The third portion 330 can be arranged along the longitudinal axis of the line of sight 235 between the first portion 325 and the third portion 335. The portions 325, 330, 335 of different colors can be used to indicate proximity to an object. For example, there may be three colors: Color 1 (associated with portion 325) may indicate that the user 105 is close to the object (e.g., within 1 meter); Color 2 (associated with portion 330) may indicate that the user 105 is in a medium range to the object (e.g., between 1-3 meters); and Color 3 (associated with portion 335) may indicate that the user 105 is in a far range to the object (e.g., more than 3 meters).

[0027] The screen 305 can be a fixed size such that objects that are far away (e.g., far field) can be relatively small in size (e.g., compared to the same object that is closer). Additionally, objects that are closer (e.g., near field) can be relatively large in size (e.g., compared to the same object that is closer). Additionally, as the user 105 approaches the object (or the object approaches the user 105), the object may adapt and appear larger. In an exemplary implementation, the dominant color of the line of sight 235 can be color 3 (associated with portion 335) because the user 105 is relatively far from the object. As the user 105 approaches the object, color 3 can become less dominant, color 2 becomes the dominant color, color 1 becomes less dominant, and color 1 (associated with portion 325) and color 2 (associated with portion 330) become more prevalent until color 3 is not shown (or may be a small streak). Then, as the user 105 approaches the object, color 2 can progress to being less dominant until it is no longer visible (or may be a small streak), and color 1 can become more dominant until it is substantially the only color in the line of sight 235.

[0028] The line of sight 235 can be used to identify a subset of targets based on a region encompassing the line of sight. A depth associated with each target of the set of targets can be estimated. The estimation of candidate targets can be based on intersections of the line of sight at depths encompassed in the region and / or targets of the subset of targets. A change in line of sight direction can be detected. For example, head movement and / or eye movement can be detected. If the change in line of sight direction is below a threshold, the image can be re-rendered on the display of the wearable device (e.g., to account for minimal change in line of sight direction). Also, the line of sight can be redrawn. If the change in line of sight direction is equal to or greater than a threshold, another image can be received from the sensor and rendered on the display of the wearable device. The line of sight may or may not be redrawn.

[0029] As shown in FIG. 3, the screen 305 can include reticles 320-1, 320-2. The reticles 320-1, 320-2 can be used to identify targets (e.g., with minimal ambiguity). The reticles 320-1, 320-2 are illustrated as rectangular. However, the reticles 320-1, 320-2 can be any shape (e.g., square, circular, oval, and / or the like). The user 105 can move the reticles 320-1, 320-2 on the screen 305 (e.g., by head and / or eye movement) to select (or help select) a candidate target with more precision (e.g., less chance of a false target). Alternatively (or additionally), the user 105 can move the image with the reticles 320-1, 320-2 maintained in position on the screen 305. For example, reticle 320-1 is at a first position and reticle 320-2 is at a second position. In an exemplary embodiment, the second position may be a preferred position for selecting a target and / or a candidate target. Thus, user 105 may move (e.g., by head and / or eye movement) the position of reticle 320-1 to the position of reticle 320-2 on screen 305.

[0030] Estimating the gaze may include processing the gaze of the user 105 at a fixed position relative to the wearable device 110. For example, the gaze direction may be collinear with the head of the user 105 in the case of a head-mounted wearable device 110. However, the gaze direction may also be based on the viewing direction and the head direction. In an exemplary implementation, the reticle 320 may force the viewing direction to be fixed (e.g., the eyes focus on the position of the reticle 320). For example, the screen 305 may be a pass-through display such that the reticle 320 and the line of sight 235 drawn on the screen 305 may intersect, and the gaze direction may be collinear with the head gaze direction of the user 105 when selecting an object as a candidate target.

[0031] Calibration can be used to align the screen 305 and the camera 250 (e.g., center the screen 305 and the camera 250). For example, the circle 310 can represent the center vector of the camera 250, and the line of sight 235 point can represent the center vector of the screen 305. The offset line 315 can represent the distance and direction of the offset between the center vector of the camera 250 and the center vector of the screen 305. Thus, calibration can shift the image received from the camera 250 and displayed on the screen 305 based on the offset (represented by the offset line 315). Alternatively, calibration can shift the line of sight 235 and / or the reticle 320 displayed on the screen 305 based on the offset (represented by the offset line 315). Calibration can include, for example, computer-aided design (CAD) calibration, factory calibration, in-field user calibration, and / or in-field automatic calibration. FIG. 4 illustrates a block diagram of calibration data flow, according to an exemplary implementation.

[0032] CAD calibration 405 can be determined during CAD design of wearable device 110. For example, CAD calibration can be based on the orientation and positioning between camera 250 and a given user's line of sight (e.g., an average user's line of sight) when wearable device 110 is designed, for example, using a CAD software tool. Factory calibration 410 can be a process or operation that can result in an adjustment of the calibration (e.g., CAD calibration) after a particular wearable device 110 is manufactured. Factory calibration can take into account the actual positioning between camera 250 and a given user's line of sight (e.g., an average user's line of sight), as determined for a particular wearable device 110.

[0033] In-field user calibration 415 can be a process or operation whereby the user 105 can further adjust the calibration by performing a series of predetermined steps. In-field user calibration can be performed before the first use by the user 105. For example, the user 105 can use an object that is easily recognizable by a computer vision algorithm as, for example, a calibration marker. For example, the object can be printed on a product box, or the product box can be used as the object. The user 105 can place the object in the user 105's environment in the far field (e.g., greater than 1 meter). The user 105 can start the calibration software while gazing at the object. In-field user calibration 420 can be repeated multiple times for greater accuracy. In-field auto-calibration can be a process or operation whereby the calibration is further adjusted during use. For example, discrepancies between the center of the detected object and the gaze estimate can be used as corrective feedback to the calibration process or operation. After calibration, target identification can be performed by gaze tracking (in-field auto-calibration while performing target identification by gaze tracking).

[0034] 5 shows a block diagram of gaze tracking with target identification data flow according to an exemplary embodiment. As shown in FIG. 5, the data flow includes a near field 505 block, a far field 510 block, a reticle 515 block, a gaze adjustment 520 block, a gaze tracking 525 block, and an identification target 530 block.

[0035] Near field 505 can be configured to estimate the user's line of sight and / or line of sight direction in the near field (e.g., within 1 meter of the wearable device). In the near field, the user's line of sight can cover a significant portion of the image. Thus, many objects can be in the user's field of view. Some objects can be farther away than what is considered to be in the near field. In other words, the user can see the object of the displayed image even if the object is far away (e.g., more than 1 meter from the wearable device). For example, referring to FIGS. 1A and 1B, D1 can be 0.5 meters from the wearable device 110 and D3 can be 5 meters from the wearable device 110. Thus, the target 115 and the target 120 can be in the near field and the target 125 can be in the far field. However, the user's line of sight can cover a significant portion of the image (including the object as a target) displayed on the screen 305. Thus, the target 115, the target 120, and the target 125 can appear to be in the user's line of sight and possibly in the near field.

[0036] In an exemplary implementation, the wearable device 110 may include a depth sensor (or may include another method of determining depth). Thus, an image displayed on the screen 305 may have an associated depth map (or other depth information). The depth map may be used to estimate the user's line of sight and / or line of sight direction, as well as a subset of targets (e.g., objects) that are in the near field. For example, the near field may be predetermined (with respect to the wearable device 110) as one meter or less. Thus, the depth map may be used to exclude target 125 from the subset of targets.

[0037] The far field 510 can be configured to estimate the user's gaze direction in the far field (e.g., more than one meter from the wearable device). In the far field, the user's gaze can span a variable portion of the image. For example, as the user's gaze moves away from the wearable device, the portion of the image spanned by the user's gaze can become smaller and smaller. Thus, the further away the user is looking, the fewer objects the user may see. Thus, estimating the user's gaze and / or gaze direction can be more accurate (compared to estimating in the near field). As described above, the wearable device 110 can include a depth sensor, and the image displayed on the screen 305 can have an associated depth map (or other depth information). Thus, the line of sight 235 depicted on the screen 305 can use the depth map to increase the accuracy of identifying targets. Furthermore, the depth along the line of sight (e.g., metric depth) can be determined using one of the calibration techniques described above. Thus, identifying targets intersecting the line of sight 235 may also include determining and / or estimating the depth of the identified targets.

[0038] The reticle 515 can be configured to force a fixation of the line of sight (e.g., the eyes focus on the position of the reticle). For example, the screen 305 can be a pass-through display such that the reticle 320 and the line of sight 235 drawn on the screen 305 can intersect, and the line of sight direction can be collinear with the head line of sight direction of the user 105 when selecting an object as a candidate target. Estimating the line of sight direction can include determining that the user's line of sight is fixed relative to the wearable. For example, the line of sight direction can be collinear with the user's head in the case of a head-mounted wearable device. However, humans tend to gaze with their heads as well as their eyes. Thus, the reticle 516 can be configured to force a fixation of the line of sight.

[0039] Gaze adjustment 520 can be configured to fix the gaze direction based on correlation with the wearable device's position, device trajectory, and the user's gaze. For example, when the user looks at a sign (e.g., facing upwards), the user's eyes can be monitored to estimate the gaze adjustment. A model that takes as input the position of the wearable in space up to a known range (3dof or 6dof) and past trajectory and generates a correction to the gaze estimate. The model can be a machine learning model (e.g., a trained neural network) or an algorithm used to calculate an offset (based on the current gaze direction). Reticle 515 and gaze adjustment 520 can be used together and / or separately.

[0040] Eye tracking 525 can be configured to track the rotational gaze of the user (e.g., using head movement). For example, the beginning of rotational tracking (e.g., 3dof movement) can include the use of sensors associated with the wearable device. The sensors can include movement sensors (e.g., inertial measurement units (IMUs)) that provide linear acceleration and rotational velocity. Changes in the orientation of the wearable can be translated into changes in gaze position in the image space, allowing the user to continuously select among the detected objects. Eye tracking 525 can be performed without capturing or rendering new images. Rotational tracking can also be used to re-trigger capture and detection if the user's current gaze falls outside the currently displayed image. In an exemplary implementation, it may not be sensitive to small translational movements, where small is relative to the distance to the object of interest. However, in the case of large translational movements (e.g., movements above a threshold), the current set of detected targets (e.g., objects) can be discarded and the capture and rendering of a new image can be triggered along with the identification of the target. Depth information can also be used to reproject objects as the wearable device moves without capturing new images.

[0041] The identification target 530 can be configured to identify possible targets. The line of sight direction and / or line of sight can be used to identify a subset of targets based on a region encompassing the line of sight. A depth associated with each target of the set of targets can be estimated. The estimation of candidate targets can be based on intersections of the lines of sight at depths encompassed in the region and / or targets of the subset of targets. In an exemplary implementation, an action can be triggered (e.g., convert text, find best price, read a map, get directions, identify an image, read a product label, identify a storefront, identify a restaurant, read a menu, identify a building, identify a product, identify natural objects (e.g., plants, flowers, trees, and / or the like), etc.) and / or, in response to a trigger, a candidate target can be estimated (or selected) based on the subset of targets. For example, if the action is to convert text, the text can be associated with a candidate target.

[0042] 6 shows a block diagram of a method for identifying a target according to an exemplary embodiment. As shown in FIG. 6, in step S605, an image is received from a sensor of a wearable device. For example, the camera 250 can capture an image (or multiple images) representing a real-world scene. The image (or one of the multiple images) can be rendered on a screen 305.

[0043] In step S610, a set of targets for the image is identified. For example, the image may include multiple objects. The objects, or a subset of the objects, may be selected as the set of targets. In step S615, a gaze direction associated with a user of the wearable device is tracked. For example, a gaze (e.g., gaze 235) may be drawn on the screen. The gaze may be used to track the user's gaze.

[0044] In step S620, a subset of targets from the set of targets within a region of the image is identified based on the line of sight direction, for example, the subset of targets can be identified based on the region encompassing the line of sight.

[0045] In step S625, an instruction to trigger an action is received. For example, an action can be triggered (e.g., converting text, finding best price, getting directions, and / or the like). For example, if the action is to convert text, the text can be associated with a candidate target. The instruction can be a voice command, a gesture, a contact with a wearable device, and / or the like.

[0046] In step S630, in response to an instruction that triggers an action, a candidate target is identified (determined, estimated, and / or the like) from the subset of targets. For example, in response to a trigger, a candidate target can be estimated (or selected) based on the subset of targets. A depth associated with each target in the set of targets can be estimated. The estimation of the candidate target can be based on an intersection of lines of sight at a depth included in the region. The candidate target can be one of the subset of targets selected based on the intersection of lines of sight at a depth included in the region.

[0047] FIG. 7 illustrates a block diagram of a system according to an exemplary implementation. In the example of FIG. 7, it should be understood that the system (e.g., a wearable device) virtually represents any computing device that can include or be associated with a computing system or at least one computing device (e.g., a mobile computing device, a mobile phone, a laptop computer, a tablet, etc.) and is configured to perform the techniques described herein. Thus, it can be understood that the system includes various components that can be utilized to implement the techniques described herein, or different or future versions thereof. By way of example, the system can include a processor 705 and a memory 710 (e.g., a non-transitory computer-readable memory). The processor 705 and the memory 710 can be connected (e.g., communicatively connected) by a bus 715.

[0048] The processor 705 may be utilized to execute instructions stored in the at least one memory 710. As such, the processor 705 may implement various features and functions described herein or additional or alternative features and functions. The processor 705 and the at least one memory 710 may be utilized for a variety of other purposes. For example, the at least one memory 710 may represent examples of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

[0049] At least one memory 710 may be configured to store data and / or information associated with the device. At least one memory 710 may be a shared resource. Thus, at least one memory 710 may be configured to store data and / or information associated with other elements (e.g., image / video processing or wired / wireless communication) in a larger system. The processor 705 and at least one memory 710 may be utilized together to perform the techniques described herein. Thus, the techniques described herein may be implemented as code segments (e.g., software) stored in the memory 710 and executed by the processor 705. Thus, the memory 710 may include a calibration 400 block, a near field 505 block, a far field 510 block, a reticle 515 block, a gaze adjustment 520 block, a gaze tracking 525 block, and an identification target 530 block. In one or more exemplary implementations, a subset of the components shown as included in the memory 710 may be used. For example, the memory 710 may include a calibration 400 block without other components.

[0050] FIG. 8 illustrates an example of a computing device 800 and a mobile computing device 850 that may be used with the techniques described herein (e.g., to implement a wearable device). The computing device 800 includes a processor 802, a memory 804, a storage device 806, a high-speed interface 808 that connects to the memory 804 and a high-speed expansion port 810, and a low-speed interface 812 that connects to a low-speed bus 814 and the storage device 806. Each of the components 802, 804, 806, 808, 810, and 812 are interconnected using various buses and may be mounted on a common motherboard or in other manners as desired. The processor 802 can process instructions for execution within the computing device 800, including instructions stored in the memory 804 or the storage device 806, to display graphical information for a GUI on an external input / output device, such as a display 816 connected to the high-speed interface 808. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and types of memories. Also, multiple computing devices 800 may be connected, each performing some of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0051] The memory 804 stores information within the computing device 800. In one implementation, the memory 804 is a volatile memory unit(s), while in another implementation, the memory 804 is a non-volatile memory unit(s). The memory 804 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0052] The storage device 806 can provide mass storage for the computing device 800. In one embodiment, the storage device 806 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a number of devices including a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. The computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 804, the storage device 806, or the memory of the processor 802.

[0053] A high-speed controller 808 manages the more bandwidth-intensive operations of the computing device 800, and a low-speed controller 812 manages the less bandwidth-intensive operations. This allocation of functionality is merely exemplary. In one embodiment, the high-speed controller 808 is connected to the memory 804, a display 816 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 810 that can accept various expansion cards (not shown). In an embodiment, the low-speed controller 812 is connected to the storage device 806 and a low-speed expansion port 814. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may connect, for example, via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0054] The computing device 800 may be implemented in many different forms, as shown. For example, it may be implemented as a standard server 820, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 824. Additionally, it may be implemented in a personal computer, such as a laptop computer 822. Alternatively, the components of the computing device 800 may be combined with other components of a mobile device (not shown), such as device 850. Each such device may include one or more of the computing devices 800, 850, and the entire system may consist of multiple computing devices 800, 850 in communication with each other.

[0055] Computing device 850 includes, among other components, a processor 852, memory 864, input / output devices such as a display 854, a communication interface 866, and a transceiver 868. Device 850 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 850, 852, 864, 854, 866, and 868 are interconnected using various buses, and some components may be mounted on a common motherboard or in other manners as desired.

[0056] The processor 852 can execute instructions within the computing device 850, including instructions stored in the memory 864. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide coordination with other components of the device 850, such as, for example, control of a user interface, applications run by the device 850, and wireless communications by the device 850.

[0057] The processor 852 may communicate with a user via a control interface 858 and a display interface 856 connected to a display 854. The display 854 may be, for example, a thin film transistor liquid crystal display (TFT LCD), a light emitting diode (LED) or organic light emitting diode (OLED) display, or other suitable display technology. The display interface 856 may include suitable circuitry for driving the display 854 to present graphical and other information to the user. The control interface 858 may receive commands from the user and convert them for submission to the processor 852. Additionally, an external interface 862 may be provided in communication with the processor 852 to enable short-range communication between the device 850 and other devices. The external interface 862 may, for example, provide for wired communication in some implementations or wireless communication in other implementations, and multiple interfaces may be used.

[0058] The memory 864 stores information within the computing device 850. The memory 864 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 874 may also be provided and connected to the device 850 via an expansion interface 872. The expansion interface 872 may include, for example, a single in-line memory module (SIMM) card interface. Such expansion memory 874 may provide extra storage space for the device 850 or may store applications or other information for the device 850. In particular, the expansion memory 874 may include instructions for performing or supplementing the above processes, and may also include secure information. Thus, for example, the expansion memory 874 may be provided as a security module for the device 850 and may be programmed with instructions that enable secure use of the device 850. Furthermore, secure applications may be provided via the SIMM card, with additional information, such as identification information, being placed on the SIMM card in an unhackable manner.

[0059] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 864, expansion memory 874, or memory on processor 852, and may be received, for example, via transceiver 868 or external interface 862.

[0060] The device 850 may communicate wirelessly via a communication interface 866, which may include digital signal processing circuitry if necessary. The communication interface 866 may provide for communication in a variety of modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, via a radio frequency transceiver 868. Additionally, short-range communication may occur, such as using Bluetooth, WiFi, or other such transceivers (not shown). Additionally, a global positioning system (GPS) receiver module 870 may provide additional navigation- and location-related wireless data to the device 850, which may be used as needed by applications executing on the device 850.

[0061] Device 850 may also perform voice communications using audio codec 860, which may receive voice information from a user and convert it into usable digital information. Audio codec 860 may also generate sounds that are heard by a user, such as through a speaker (e.g., in a handset of device 850). Such sounds may include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on device 850.

[0062] Computing device 850 may be implemented in many different forms, as shown, including as a mobile phone 880, or as part of a smartphone 882, personal digital assistant, or other similar mobile device.

[0063] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0064] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0065] To provide interaction with a user, the systems and techniques described herein can be implemented in a computer having a display device (such as an LED (light emitting diode), OLED (organic organic light emitting diode), or LCD (liquid crystal display) monitor / screen) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and input from the user can be received in any form, such as acoustic, spoken language, or tactile input.

[0066] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or in a combination of such back-ware, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0067] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0068] In some implementations, the computing device shown in the figure may include sensors that interface with the AR headset / HMD device 890 and generate an augmented environment for viewing the inserted content in the physical space. For example, one or more sensors included in the computing device 850 shown in the figure or other computing devices may provide input to the AR headset 890 or generally provide input to the AR space. The sensors may include, but are not limited to, a touch screen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 850 may use the sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, which may then be used as input to the AR space. For example, the computing device 850 may be embedded in the AR space as a virtual object, such as a controller, a laser pointer, a keyboard, a weapon, etc. When embedded in the AR space, the positioning of the computing device / virtual object by the user allows the user to position the computing device to view the virtual object in a particular way in the AR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were an actual laser pointer. A user can move a computing device left and right, up and down, in a circular direction, etc., to use the device in a manner similar to using a laser pointer. In some implementations, a user can use a virtual laser pointer to point to a target location.

[0069] In some implementations, one or more input devices included in or connected to the computing device 850 can be used as input to the AR space. The input devices can include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or buds with input capabilities, a game controller, or other connectable input devices. When the computing device is integrated into the AR space, a user can interact with the input devices included in the computing device 850 to cause certain actions to occur in the AR space.

[0070] In some implementations, the touch screen of the computing device 850 can be rendered as a touchpad in the AR space. A user can interact with the touch screen of the computing device 850. The interaction is rendered, for example in the AR headset 890, as movement on the rendered touchpad in the AR space. The rendered movement can control virtual objects in the AR space.

[0071] In some implementations, one or more output devices included in the computing device 850 can provide output and / or feedback to a user of the AR headset 890 within the AR space. The output and feedback can be visual, tactical, or audio. The output and / or feedback can include, but are not limited to, vibration, turning on and off one or more lights or strobes, or blinking and / or flashing, sounding an alarm, playing a chime, playing a song, and playing an audio file. The output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

[0072] In some implementations, the computing device 850 may be displayed as another object in the computer-generated 3D environment. A user's interaction with the computing device 850 (e.g., rotating, shaking, touching the touch screen, swiping a finger on the touch screen) may be interpreted as an interaction with an object in the AR space. In the example of a laser pointer in the AR space, the computing device 850 appears as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates the computing device 850, the user in the AR space sees the movement of the laser pointer. The user receives feedback from the interaction with the computing device 850 in the AR environment on the computing device 850 or the AR headset 890. The user's interaction with the computing device may be translated into an interaction with a user interface generated in the AR environment for the controllable device.

[0073] In some implementations, the computing device 850 may include a touch screen. For example, a user may interact with the touch screen to interact with a user interface of the controllable device. For example, the touch screen may include user interface elements, such as sliders, that can control characteristics of the controllable device.

[0074] Computing device 800 is intended to represent various types of digital computers and devices, which may include, but are not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 850 is intended to represent various types of mobile devices, such as personal digital assistants, mobile phones, smart phones, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0075] A number of embodiments have been described. Of course, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure.

[0076] Moreover, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Moreover, other steps may be provided in or removed from the described flows, and other components may be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0077] In addition to the above, the system, program, or functionality described herein may provide controls that allow the user to select whether and when user information (e.g., information about the user's social network, social actions, or activities, occupation, user preferences, or the user's current location) may be collected, as well as whether the user is sent content or communications from the server. Additionally, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that the user's personally identifiable information cannot be identified, or if location information is obtained (such as to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific location cannot be identified. Thus, the user may control what information is collected about the user, how that information is used, and what information is provided to the user.

[0078] As described herein, while certain features of the described embodiments have been illustrated, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. It is to be understood that these are presented by way of example only, and not by way of limitation, and that various changes in form and details may be made. Any part of the apparatus and / or method described herein may be combined in any combination, except in mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

[0079] Exemplary embodiments may include various modifications and alternative forms, which are shown by way of example in the drawings and described in detail herein. It is to be understood, however, that the exemplary embodiments are not limited to the particular forms disclosed, but rather cover all modifications, equivalents, and alternatives falling within the scope of the claims. Like numerals refer to like components throughout the description of the figures.

[0080] Some of the above exemplary embodiments are described as a process or method that is depicted as a flowchart. Although the flowcharts describe operations as sequential processes, many of the operations may occur in parallel, concurrently, or simultaneously. The order of operations may also be rearranged. A process may be terminated when its operations are completed, or may have additional steps not included in the figures. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0081] The above methods, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine- or computer-readable medium, such as a storage medium. A processor(s) may perform the necessary tasks.

[0082] Specific structural and functional details disclosed herein are presented merely for purposes of describing example embodiments, however, example embodiments may be embodied in many alternative forms and should not be construed as being limited to only the embodiments set forth herein.

[0083] Terms such as first, second, etc. may be used herein to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term and / or includes any and all combinations of one or more of the associated listed items.

[0084] When an element is referred to as being connected or coupled to another element, it will be understood that the element may be directly connected or coupled to the other element, or there may be intervening elements. In contrast, when an element is referred to as being directly connected to or directly coupled to another element, there are no intervening elements present. Other words used to indicate the relationship between elements should be interpreted similarly (e.g., between and directly between, adjacent and directly adjacent, etc.).

[0085] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of example embodiments. As used herein, the singular forms a, an, and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is further understood that the terms "comprises," "comprising," "includes," and / or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0086] It should also be noted that in some alternative implementations, the functions / acts shown may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed concurrently or may sometimes be executed in the reverse order, depending on the functions / acts involved.

[0087] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs. Furthermore, terms, for example those defined in commonly used dictionaries, should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0088] Portions of the above exemplary embodiments and corresponding detailed description are presented in terms of software, or algorithms, and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the way those skilled in the art effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used herein, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. These steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0089] In the above exemplary embodiments, references to symbolic representations of acts and operations that may be implemented as program modules or functional processes (e.g., in the form of a flowchart) include routines, programs, objects, components, data structures, etc. that perform particular types of tasks or particular types of abstract data and may be described and / or implemented using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate array (FPGA) computers, etc.

[0090] It should be recognized, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, or as will be apparent from the description, terms such as processing, operating, calculating, determining, displaying, and the like, refer to the actions and processes of a computer system or similar computing device that manipulate and convert data represented as physical, electronic quantities in the computer system's registers and memory into other data that are similarly represented as physical quantities in the computer system's memory or registers, or other such information storage, transmission, or display device.

[0091] It should also be noted that the software implementation aspects of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented via some type of transmission medium. The program storage medium may be magnetic (e.g., floppy disk or hard drive) or optical (e.g., compact disk read only memory, or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or other suitable transmission medium known in the art. The exemplary embodiments are not limited by these aspects of any given implementation.

[0092] Finally, it should also be noted that, although the appended claims recite particular combinations of the features described in this specification, the scope of the disclosure is not limited to the specific combinations claimed below, but instead is extended to encompass any combination of features or embodiments disclosed herein, whether or not that particular combination is currently specifically recited in the appended claims.

Claims

1. Receiving an image from a sensor of a wearable device; Rendering the image on a display of the wearable device; and identifying a set of targets within the image; Tracking a gaze direction associated with a user of the wearable device; and rendering a gaze direction on the displayed image based on the tracked gaze direction; identifying a subset of targets based on the set of targets within a region of the image based on the line of sight; Triggering an action; and in response to the trigger, estimating candidate targets based on the subset of targets.

2. identifying a subset of the targets based on a region encompassing the line of sight; estimating a depth associated with each target of the set of targets; Further comprising: The method of claim 1 , wherein the estimation of the candidate targets is based on an intersection of the lines of sight at a depth that is included in the region.

3. Detecting a change in gaze direction; determining that the change is less than a threshold; re-rendering the image on a display of the wearable device; The method of claim 1 further comprising:

4. Detecting a change in gaze direction; determining that the change is less than a threshold; re-rendering the line of sight; and The method of claim 1 further comprising:

5. Detecting a change in gaze direction; determining that the change is in the rendered image and is closer to the subset of targets; re-rendering said line of sight with a change in color; The method of claim 1 further comprising:

6. Detecting a change in gaze direction; determining that the change is greater than a threshold; receiving another image from the sensor; and The method of claim 1 further comprising:

7. The method of claim 1 , further comprising rendering a reticle on the displayed image based on the positions of the candidate targets.

8. The method of claim 7 , further comprising repositioning the reticle to a different location on the displayed image, and the candidate targets are estimated based on the repositioned reticle.

9. The method of claim 1 , further comprising calibrating the wearable device based on a position of the sensor of the wearable device and a center of a display of the wearable device.

10. An image sensor; A display and At least one processor; at least one memory containing computer program code; The at least one memory and the computer program code are configured to cause the wearable device, using the at least one processor, to: receiving an image from the image sensor; Rendering the image on the display; and identifying a set of targets within the image; Tracking a gaze direction associated with a user of the wearable device; and rendering a gaze direction on the displayed image based on the tracked gaze direction; identifying a subset of targets based on the set of targets within a region of the image based on the line of sight; Triggering an action; and responsive to the trigger, estimating candidate targets based on the subset of targets; A wearable device configured to:

11. The computer program code further comprises: identifying a subset of the targets based on a region encompassing the line of sight; estimating a depth associated with each target of the set of targets; Let them do so, The wearable device of claim 10 , wherein the estimation of the candidate targets is based on an intersection of the lines of sight at a depth included in the region.

12. The computer program code further comprises: Detecting a change in gaze direction; determining that the change is less than a threshold; re-rendering the image on a display of the wearable device; The wearable device of claim 10 .

13. The computer program code further comprises: Detecting a change in gaze direction; determining that the change is less than a threshold; re-rendering the line of sight; and The wearable device of claim 10 .

14. The computer program code further comprises: Detecting a change in gaze direction; determining that the change is in the rendered image and is closer to the subset of targets; re-rendering said line of sight with a change in color; The wearable device of claim 10 .

15. The computer program code further comprises: Detecting a change in gaze direction; determining that the change is greater than a threshold; receiving another image from the sensor; and The wearable device of claim 10 .

16. The computer program code further comprises: The wearable device of claim 10 , further configured to render a reticle on the displayed image based on the position of the candidate target.

17. The computer program code further comprises: The wearable device of claim 16 , further comprising: repositioning the reticle to a different position on the displayed image; and the candidate targets are estimated based on the repositioned reticle.

18. The computer program code further comprises: The wearable device of claim 10 , further comprising: a sensor for detecting a position of the wearable device and a center of a display of the wearable device.

19. A non-transitory computer-readable medium storing executable instructions that, when executed by at least one processor, cause the at least one processor to: Receiving an image from a sensor of a wearable device; Rendering the image on a display of the wearable device; and identifying a set of targets within the image; Tracking a gaze direction associated with a user of the wearable device; and rendering a gaze direction on the displayed image based on the tracked gaze direction; identifying a subset of targets based on the set of targets within a region of the image based on the line of sight; Triggering an action; and responsive to the trigger, estimating candidate targets based on the subset of targets; The non-transitory computer-readable medium.

20. The executable instructions further include causing the processor to: identifying a subset of the targets based on a region encompassing the line of sight; estimating a depth associated with each target of the set of targets; Let them do so, 20. The non-transitory computer-readable medium of claim 19, wherein the estimation of the candidate targets is based on an intersection of the lines of sight at a depth that is included in the region.

Images