Object tracker using gaze estimation

Abstract

A mobile device, such as a smartphone, comprises: a first camera having a first field of view FOV1 directed towards a scene to be captured, the first camera being configured to capture first image data; a second camera having a second field of view FOV2 directed to a scene including the eyes of the user, the second camera configured to capture second image data; and a processor including an object tracker and an eye movement tracker, the processor configured to perform object tracking using the first image data and perform gaze estimation using the second image data, the object tracker is configured to perform an action selected from a group consisting of selection of an object in the scene to be tracked by the object tracker, verification of the tracked object in the scene, and re-identification of the tracked object in the scene using the gaze estimation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to and claims priority to U.S. Provisional Patent Application No. 63 / 394,441, filed on August 2, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The subject matter disclosed herein relates generally to camera algorithms for mobile devices, and more particularly, to camera algorithms for smartphones. Background Art

[0004] Modern mobile electronic devices (or simply "mobile devices"), such as smartphones, tablets, laptops, smartwatches, or headsets (also called "glasses") for advanced reality (AR) or virtual reality (VR), include various camera-related technologies. In terms of hardware, they usually include a screen (or display) and multiple cameras with different fields of view (FOV), which can be located on different surfaces of the mobile device. Typically, a mobile device includes at least one front-facing camera (also called "user-facing camera" or "selfie camera"), which is located on a first (or "front") surface of the mobile device that includes the screen, and at least one rear-facing camera (or "world-facing camera"), which is located on a second (or "back") surface of the mobile device, the second surface pointing in a direction opposite to the first surface (see Figure 6 ). Thus, the mobile device can capture different fragments of a scene simultaneously. For example, the mobile device can capture a scene using its rear camera while simultaneously capturing a user controlling the mobile device, such as by touching a touch screen, using its front camera. In terms of software, the mobile device includes advanced image processing methods for processing image data generated (or captured) by multiple cameras. Examples of such image processing methods include methods for object detection, saliency detection, and object tracking.

[0005] These image processing methods allow a user of a mobile device to capture a photo or video of a scene according to the user's intention. "Intent" with respect to a user (i.e., "user intent") is herein referred to as a user's intended camera control situation (e.g., an intended focus situation), or a user's intended mobile device control situation (e.g., an intended image processing situation). As an example of an intended focus situation, a user may intend to focus a camera on a particular object in a scene. As an example of an intended image processing situation, a user may intend to achieve a particular brightness and / or contrast for a particular object in a scene.

[0006] Figure 1 The steps of a known method for object tracking are shown, referenced 100 .

[0007] In step 102 , a user points the mobile device at a scene (or “aim the scene”) using one or more cameras included in the mobile device.

[0008] In step 104, one or more cameras capture image data. Typically, the image data is captured continuously, i.e., a stream of images (or video streams) is captured. In some examples, the mobile device may capture additional data. The additional data may or may not be image data. For example, the additional data may be audio data, directional audio data, location or orientation data of the mobile device, or data of other mobile devices located near the mobile device.

[0009] In step 106, a processor included in the mobile device runs an algorithm (or program) for analyzing the scene. For scene analysis, the image data captured in step 104 is typically analyzed. An example of scene analysis is detecting objects in the scene and calculating a saliency map of the scene. In the saliency map, a saliency score is assigned to each segment in the scene. In some examples, the processor may generate or use additional image-based data. The additional image-based data may include information about the type of object (e.g., whether it is a face, a person or an animal, etc.), the location of the object detected in the scene, etc. Step 106 may be optional. In some scene analysis examples, additional non-image-based data and additional image-based data are analyzed. Examples of results from such scene analysis are the location and type of detected objects, the location of scene segments with particularly high (or low) saliency scores, etc. In some examples, the processor may generate a list including the results from the scene analysis, hereinafter referred to as a "scene analysis list".

[0010] In step 108, an object to be tracked ("target object") is selected.

[0011] In an example of autonomous target object selection, the processor uses the scene analysis list to select a specific object or scene segment from the scene analysis list for tracking. "Autonomous target object selection" herein means that the target object selection is performed by the mobile device without any user intervention.

[0012] In the example of user target object selection, a specific object or scene segment can be selected based on a user command. "User target object selection" in this article means performing target object selection based on user intent. For example, additional non-image-based data may include a user command issued by a user to a mobile device indicating the user's intent. As an example of a user command, a user can indicate the user's intent to select an object in a scene by touching a location on a touch screen displaying an object. Another example of a user command is that the user sends a voice command. It should be noted that in order to indicate the user's intent through a user command, physical interaction between the user and the mobile device is necessary (e.g., touching the mobile device, or speaking to the mobile device).

[0013] The data used and generated in step 108 is hereinafter referred to as "initialization information" because it is used to "initialize" (or "define" or "set up") the object tracker module (or simply "object tracker") within the processor.

[0014] In step 112, the object tracker within the processor tracks the object selected in step 108 in a continuous manner, that is, in each (or every second, every third, even every eighth or tenth, etc.) image of the captured image stream, the position of the tracked object in the scene is calculated. Typically, the verification step is performed simultaneously. In some examples, a "confidence score" is calculated for verification. The confidence score represents the probability that the calculated position of the tracked object is correct, that is, it verifies the validity of the results of the object tracker. The higher the confidence score, the higher the probability that the calculated position of the tracked object is correct. Therefore, the confidence score is a measure of the "reliability" of the object tracker. "Reliability" in this article refers to the ability of the object tracker to correctly track the target object, that is, to correctly calculate the position of a specific target object. The position of the tracked object can be used to control one or more cameras included in the mobile device (for example, for focusing the camera) or for further image processing (for example, for optimizing the brightness of the captured image). In some cases, for example, starting from a specific image in the captured image stream, the processor cannot track the object, that is, the processor cannot successfully calculate the position of the tracked object in the scene, resulting in an undesirable phenomenon referred to as "target loss" in this article.

[0015] If the target is lost, then in step 114, the processor uses the initialization information to re-detect and identify the target object and its location in the scene, a process called "re-identification". Typically, re-identification includes additional sub-steps for object detection. If re-identification is successful, the process returns to step 112 and the processor continues to track the target object. If re-identification fails, it returns to step 106 or step 108, which is not desirable. The frequency of re-identification failures is related to the quality of the object tracker. Object trackers that only have a low frequency of re-identification failures are preferred and are called "robust" object trackers, and the quality of the object tracker is called "robustness". It should be noted that in step 114, typically only initialization information is available, but no real-time information about user intent is available.

[0016] Using additional information about the user's intent would be very beneficial because (1) the target object can be automatically selected without physical interaction between the user and the mobile device, and / or (2) the reliability of the object tracker can be improved, and / or (3) the robustness of the object tracker can be improved. These features are lacking in the prior art. Summary of the invention

[0017] In various examples, a mobile device is provided, comprising: a first camera configured to capture first image data and having a first field of view (FOV1) directed toward a first scene; a second camera configured to capture second image data and having a second field of view (FOV2) directed toward a second scene including eyes of a user; and a processor comprising an object tracker and an eye tracker, wherein the object tracker is configured to perform object tracking using the first image data, and the eye tracker is configured to perform gaze estimation using the second image data.

[0018] In some examples, the object tracker is configured to use the line of sight estimate to select an object in the first scene for tracking by the object tracker.

[0019] In some examples, the object tracker is configured to use line of sight estimation to verify the tracked object in the first scene.

[0020] In some examples, the object tracker is configured to use line of sight estimation to re-identify the tracked object in the first scene.

[0021] In some examples, gaze estimation is performed directly by where the user is looking in the scene.

[0022] In some examples, the mobile device has a front side with a front surface and a back side with a back surface, the front surface includes a screen, a first camera is located on the back side, and a second camera is located on the front side. In some examples, the gaze estimation is performed indirectly by a user looking at a location on the screen.

[0023] In some examples, object tracking provides additional scene information, and the additional scene information is used to control the second camera. In some examples, the additional scene information is used to control the mobile device.

[0024] In some examples, control of the second camera includes control selected from the group consisting of: control for focusing the second camera, control for personalized object selection, control for improving image quality of the area of ​​interest within FOV2, control for zooming the second camera, control for selecting image resolution, and any combination thereof.

[0025] In some examples, the control of the mobile device includes control to prioritize scene segments in FOV1 or FOV2.

[0026] In some examples, the second camera is the active camera.

[0027] In some examples, the mobile device is a smartphone.

[0028] In some examples, the mobile device is a tablet computer.

[0029] In some examples, the mobile device is an augmented reality (AR) or virtual reality (VR) head-mounted device.

[0030] In some examples, the mobile device is a smart watch.

[0031] In various examples, a system is provided, including: a first mobile device, including a first camera, the first camera is configured to capture first image data and has a first field of view (FOV1), the first camera is pointed at a first scene including a user's eyes; a second mobile device, including a second camera, the second camera is configured to capture second image data and has a second field of view (FOV2), the second camera is pointed at a second scene; and one or more processors, including an object tracker and an eye tracker, wherein the eye tracker is configured to perform gaze estimation using the first image data, and the object tracker is configured to select an object in the second scene for object tracking using the gaze estimation and perform object tracking using the second image data.

[0032] In some examples, the one or more processors are included in the second mobile device.

[0033] In some examples, the one or more processors include a first processor included in a first mobile device and a second processor included in a second mobile device.

[0034] In some examples, the first mobile device is configured to transmit the first image data to the second mobile device.

[0035] In some examples, the first processor performs gaze estimation, the first mobile device is configured to transmit gaze estimation information to the second mobile device, and the second processor performs object tracking.

[0036] In some examples, gaze estimation is performed directly by where the user is looking in the scene.

[0037] In some examples, the first mobile device includes a screen, and the gaze estimation is performed indirectly by the user gazing at a location on the screen.

[0038] In some examples, object tracking provides additional scene information, and the additional scene information is used to control the second camera.

[0039] In some examples, control of the second camera includes control selected from the group consisting of: control for focusing the second camera, control for personalized object selection, control for improving image quality of the area of ​​interest within FOV2, control for zooming the second camera, control for selecting image resolution, and any combination thereof.

[0040] In some examples, the first camera is the active camera.

[0041] In some examples, the first mobile device and the second mobile device are smartphones.

[0042] In some examples, the first mobile device is a head mounted device and the second mobile device is a smartphone.

[0043] In some examples, the first mobile device is a smart watch and the second mobile device is a smart phone. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Non-limiting examples are described below with reference to the drawings listed after this paragraph. The same structure, element, or component that appears in multiple figures is generally labeled with the same reference numeral in all figures in which it appears. The drawings and descriptions are intended to explain and illustrate examples of the subject matter disclosed herein and should not be considered limiting in any way. In the drawings:

[0045] Figure 1 An example of a known object tracking method is shown;

[0046] Figure 2 An exemplary method disclosed herein for automatically tracking target object selection using line of sight estimation is shown;

[0047] Figure 3 An exemplary method for reliable object tracking using line of sight estimation disclosed herein is shown;

[0048] Figure 4Another exemplary method for robust object tracking using line of sight estimation disclosed herein is shown;

[0049] Figure 5 An example of a mobile device for object tracking disclosed herein is schematically shown;

[0050] Figure 6 An example of a mobile device for object tracking disclosed herein is shown. DETAILED DESCRIPTION

[0051] Figure 2 The steps of an exemplary method disclosed herein for object selection and tracking are shown and are labeled 200. The method 200 allows for automatic selection of a target object without requiring physical interaction between a user and a mobile device.

[0052] Method 200 includes all the steps included in method 100, and in addition, method 200 includes a seventh step 207. Reference numerals 202, 204, 206, 208, 212, and 214 denote steps corresponding to Figure 1 The same steps as 102, 104, 106, 108, 112, and 114. Figure 3 Marks 302, 304, 306, 308, 312, 314 and Figure 4 The marks 402, 404, 406, 408, 412, and 414 in also represent the same steps respectively. In step 207, the user's eyes are tracked to estimate the position where the user is looking in the scene, that is, "line of sight estimation" is performed. In order to perform line of sight estimation, image data is analyzed. For example, a first camera captures first image data of a first scene, the first scene includes a target object, wherein the first image data is used for object tracking. A second camera captures second image data of a second scene, the second scene includes the user's eyes, wherein the second image data is used to perform line of sight estimation. The first camera can be a rear camera, such as a rear camera 520 for shooting or capturing a video of the first scene, and the second camera can be a front camera, such as a front camera 510. Generally, the first scene can be different from the second scene. Here and in the following, it is assumed that the line of sight estimation is performed continuously and in real time, for example, by analyzing a video image stream. The line of sight estimation can represent additional data, which includes (or can be interpreted as) a user's command to the mobile device, which indicates the user's intention. That is, by using line of sight estimation, the user command can be transmitted to the mobile device. It is important to note that when gaze estimation is used to transmit user commands, no physical interaction is required between the user and the mobile device (e.g., no need to touch the mobile device or speak to the mobile device). This facilitates fast, convenient, and natural interaction between the user and the mobile device.

[0053] For example, the user's intention is to select a target object to be tracked. When the user looks at a specific location in the scene, the specific location information can be interpreted as the user's intention to select a specific object located in the scene segment and track it. In short, the specific object is selected because it is located at or close to the specific location.

[0054] In these examples, the line of sight estimate and the derived user command of step 207 are used as initialization data in step 208 .

[0055] As a first example of using gaze estimation to transmit user commands (the “focus tracking example”), a user may wish to continuously focus the camera on a particular selected object for capturing a video image stream or a single continuous still image so that the particular selected object is always in focus.

[0056] As a second example of using line of sight estimation to transmit user commands (“Focus Position Example”), the user may wish to continuously focus the camera on a particular FOV segment for capturing a video image stream or a single continuous still image so that the particular selected FOV segment is always in focus.

[0057] As a third example of using gaze estimation to transmit user commands ("Region of Interest (ROI) Optimization Example"), a user may wish to optimize the quality of an image segment at a specific location where the user is looking. This image segment is referred to as the ROI hereinafter. This means that the user may be willing to accept a lower quality of image segments other than the ROI if the quality of the ROI is improved. The first example of ROI optimization refers to a pre-capture situation, where the sensor exposure can be controlled to achieve a beneficial output for the ROI. The beneficial output can be a larger signal-to-noise ratio (SNR) or a larger dynamic range in the ROI. The second example of ROI optimization refers to a post-capture situation, where automatic white balance or tone mapping can be controlled to achieve a beneficial output for the ROI.

[0058] As a fourth example of using gaze estimation to transmit user commands (the "zoom example"), the user may wish to capture the ROI at a higher resolution than the resolution used to capture image segments other than the ROI. For an image sensor with adaptive pixel resolution, the resulting action may be that the spatial (or pixel) resolution of the image sensor switches to a higher resolution than the resolution of the other image segments in the ROI. For multi-camera, the resulting action may be that the multi-camera switches to a different camera than the currently used one (e.g., a telephoto camera) so that the ROI can be captured at a higher resolution.

[0059] Line of sight estimation is used to select the target object, i.e. to initialize the object tracker, e.g. (see Figure 5) object tracker 538, which tracks a specific object after initialization and continuously transmits position information to an application processor (AP), such as AP 530. A camera controller (e.g., 540) included in AP 530 calculates auto focus information and transmits it to, for example, a rear camera 520, so as to focus the rear camera 520 so that the specific object is in focus.

[0060] Figure 3 The steps of an exemplary method disclosed herein for reliable object tracking are shown and labeled 300. The method 300 allows for improving the reliability of an object tracker.

[0061] Method 300 includes all the steps included in method 100, and in addition, method 300 includes step 309. In step 309, the mobile device performs line of sight estimation. Here, the line of sight estimation is used to continuously transmit user commands. That is, in contrast to method 200, the line of sight estimation and derived user commands of step 309 are not used as initialization data, but are used to continuously monitor whether the user's intention is satisfied. In the first case, when the user looks at a specific location in a scene that includes a tracked target object, the location information can be interpreted as an indication that the object tracker is indeed tracking the target object. Therefore, for example, a higher confidence score can be assigned. "Higher confidence score" here means that the confidence score assigned using line of sight estimation is higher than the confidence score assigned in the absence of line of sight estimation information or in the case where the user is looking at a location in the scene that does not include a tracked target object.

[0062] In the second case, when the user gazes at a specific location in the scene that does not include the target object, the location information can be interpreted as an indication that the object tracker is not tracking the target object and a lower confidence score can be assigned. In other words, the line of sight estimation is used to verify the results of the object tracker. With line of sight estimation, in addition to the initialization information, real-time information can also be obtained. The real-time information is interpreted as user intention, which increases the reliability of the object tracker.

[0063] Figure 4 The steps of an exemplary method of robust object tracking disclosed herein are shown, labeled 400. Method 400 allows for increasing the robustness of an object tracker. Note that the steps of methods 200, 300, and 400 may be mixed to obtain a method for increasing any two or all of the combination of automatic selection of objects to be tracked, reliability and / or robustness of object tracking.

[0064] Method 400 includes all the steps in method 100, and in addition, method 400 includes a seventh step 413. In step 413, the mobile device performs line of sight estimation. The line of sight estimation of step 413 may not be used to derive user commands, but may be used, for example, to prioritize scene segments or objects in the scene.

[0065] For example, after the target is lost, when the user looks at a specific position in the scene, the position information can be interpreted as indicating that the specific position includes the (lost) target object. This can significantly promote the re-identification of the target object after the target is lost. The following example relates to the additional object detection sub-step contained in the re-identification step. For example, the processor can prioritize specific scene fragments based on the position information, which is conducive to fast calculation time and low computing power consumption. In some examples, the processor can perform object detection only at a scene fragment smaller than the FOV including the specific position, rather than performing object detection at the entire FOV. Therefore, compared with the case without line of sight estimation, object detection is accelerated and / or consumes less power. In other examples, the processor can first perform object detection at a first scene fragment smaller than the FOV, the first scene fragment including the specific position, and only later can perform object detection at a further scene fragment of the FOV. In other examples, in order to prioritize the scene fragment including the specific position, the processor can intentionally reduce the image resolution of the image used for object detection, so that the resolution of the scene fragment decreases as the distance from the specific position increases. That is, the first scene fragment near the specific position can have a higher image resolution than the second scene fragment far from the specific position.

[0066] In general, gaze estimation is used to improve the robustness of object trackers. Through gaze estimation, real-time information is obtained in addition to initialization information. The real-time information is interpreted as user intention, which increases the robustness of object trackers.

[0067] Line of sight estimation can be performed using image data from a suitable camera included in the mobile device that is not used for the object tracking task. "Suitable camera" in this article means that the camera must cover a scene segment that includes the user's eyes. For example, image data from a rear camera (or "world-facing camera") (e.g., rear camera 520) can be used for object tracking, and image data from a front camera (or "user-facing camera" or "selfie camera") (e.g., front camera 510) can be used for line of sight estimation. Line of sight estimation can be performed according to two different approaches.

[0068] The first approach refers to the case where the screen (or display) of the mobile device displays the scene fragment captured by the camera included in the mobile device in real time. In the first approach, the user is required to look at the mobile device screen. The specific position where the user is looking at in the scene is indirectly estimated by estimating the position where the user is looking at on the screen. That is, during the indirect line of sight estimation, the user looks at the screen.

[0069] The second approach refers to the case where the user is looking at the scene itself. In the second approach, the specific location in the scene that the user is looking at is directly estimated. Therefore, "direct line of sight estimation" refers to the user looking at the scene (rather than the user looking at the screen showing a specific location in the scene).

[0070] Figure 5 An embodiment of a mobile device (e.g., a smartphone) configured to perform the methods disclosed herein is schematically shown, labeled 500. The mobile device 500 includes a front camera 510 and a rear camera 520. Each camera has a FOV. The mobile device may include more than two cameras, each with its own FOV, as is known. The mobile device 500 may be operated to simultaneously capture front image data using the front camera 510 and capture rear image data using the rear camera 520.

[0071] The mobile device 500 further includes an application processor (AP) 530. The AP 530 includes a user controller 532 (e.g., configured to receive user input transmitted through a touch screen (e.g., screen 550)), an eye tracker module (or simply "eye tracker") 534 (e.g., configured to receive image data for continuously estimating user gaze), an object selector module (or simply "object selector") 536 (e.g., configured to receive information from the user controller 532 and the eye tracker 534 and run an object detection algorithm and select a target pair), an object tracker module (or simply "object tracker") 538 (configured to continuously track a target object), and a camera control module 540 (configured to calculate camera control signals, such as autofocus control signals for the front camera 510 and the rear camera 520). It should be noted that the modules 534, 536, 538, and 540 can be implemented in software (SW) or a combination of SW and hardware (HW). In some examples, the front camera 510 and / or the rear camera 520 may be a multi-aperture camera (or simply a multi-camera). In some examples, the front camera 510 may include a device for illuminating the scene captured by the front camera 510. Such a device for illuminating may be, for example, a light emitting diode ("LED"), a vertical cavity surface emitting laser ("VCSEL"), an edge emitting laser ("EEL"), etc. In general, the device for illuminating the scene may be located near the image sensor included in the front camera. A camera including a device for illuminating is referred to herein as an "active camera".

[0072] The mobile device 500 also includes a screen 550 for displaying information. The screen 550 may be a touch screen configured to receive user commands. The mobile device 500 also includes a memory 560, for example, for storing calibration data between the front camera 510 and the rear camera 520. In other examples, the memory 560 may include a personal "gallery" including various images captured and / or stored in the past by a particular user of the mobile device 500. In other examples, the personal gallery of a particular user may not be stored on the mobile device 500, but may be stored in a cloud server accessible from the mobile device 500, for example. The images contained in the personal gallery may be used to extract additional information about the user's intent (in addition to the eye tracking information), for example, by performing statistical analysis on the images contained in the personal gallery. For example, a particular object (e.g., a particular person who has been captured relatively frequently in the past (and therefore appears relatively frequently in the personal gallery)) may be preferred over an object that has not yet (or not as frequently as the particular object) appeared in the personal gallery. In the case where the user is looking at a specific location in a scene (or FOV) that includes a specific object and one or more other objects, the specific object (e.g., the target object in step 208) can be selected based on the statistics of the personal gallery. This selection process is called "personalized object selection". The mobile device 500 can also include several additional sensors for capturing additional non-image-based data used by the object selector 536 and / or the object tracker 538, such as a microphone or even a directional microphone, a position sensor (e.g., GPS), or an inertial measurement unit (IMU).

[0073] In some examples, line of sight estimation may be performed partially or entirely not by the mobile device 500, but by an additional mobile device (e.g., another smartphone, a head-mounted device for advanced reality (AR) or virtual reality (VR), a smart watch, a tablet, a laptop, etc.). The additional device may capture image data including the user's eyes, and may perform line of sight estimation using a processor contained in the additional device. The additional device may then transmit the line of sight estimation data to the mobile device 500. The mobile device 500 uses the line of sight estimation data from the additional mobile device to perform the method disclosed herein. In other examples, the image data for line of sight estimation may not be captured by the mobile device 500, but by the additional mobile device. The image data may include the user's eyes and may be transmitted to the mobile device 500. The mobile device 500 may use the image data captured by the additional mobile device to perform the line of sight estimation in the method disclosed herein.

[0074] Figure 6A mobile device, designated 600, is schematically illustrated and configured to perform the methods disclosed herein. The mobile device 600 has a front surface 602 that is generally directed toward a user and a rear surface 604 that is generally directed toward a scene captured by the user. The front surface 602 includes a front-facing (or "user-facing") camera 610 (similar to the camera 510) having a front-facing camera FOV 612, or more generally, having a first FOV ("FOV 1 ”). Front surface 602 also includes screen 650. Rear surface 604 includes rear camera 620 (similar to camera 520) having rear camera FOV 622, or more generally, having a second FOV (“FOV 2 ”). The front camera 610 and the rear camera 620 are respectively configured to capture front and rear image data that may include the user's eyes. The front and rear image data can be used to estimate the user's gaze.

[0075] Unless otherwise indicated, the use of "and / or" between the last two members of a list of options indicates that selecting one or more of the listed options is appropriate and can be made.

[0076] Although the present application has been described according to specific embodiments and generally related methods, variations and substitutions of the embodiments and methods are obvious to those skilled in the art. The present application should be understood to be not limited to the specific embodiments described herein, but only to the scope of the appended claims.

Claims

1. A mobile device, comprising: a first camera configured to capture first image data and having a first field of view (FOV1) directed toward a first scene; a second camera configured to capture second image data and having a second field of view (FOV2) directed toward a second scene including the user's eyes; as well as a processor comprising an object tracker and an eye tracker, wherein the object tracker is configured to perform object tracking using the first image data and the eye tracker is configured to perform gaze estimation using the second image data, Wherein, the object tracker is configured to use the line of sight estimate to select an object in the first scene for tracking by the object tracker.

2. The mobile device according to claim 1, wherein: The mobile device has a front side with a front surface and a back side with a back surface, wherein the front surface includes a screen, wherein the first camera is located on the back side, and wherein the second camera is located on the front side.

3. The mobile device according to claim 1, wherein: The line of sight estimation is performed directly by the user looking at a location in the scene.

4. The mobile device according to claim 1, wherein: The mobile device comprises a screen, wherein the line of sight estimation is performed indirectly by the user gazing at a position on the screen.

5. The mobile device according to claim 1, wherein: The object tracking provides additional scene information, and wherein the additional scene information is used to control the second camera.

6. The mobile device according to claim 5, wherein: Control of the second camera includes control selected from the group consisting of: control for focusing the second camera, control for personalized object selection, control for improving image quality of an area of ​​interest within FOV2, control for zooming the second camera, control for selecting an image resolution, and any combination thereof.

7. The mobile device according to claim 1, wherein: The object tracking provides additional context information, and wherein the additional context information is used to control the mobile device.

8. The mobile device according to claim 7, wherein: The control of the mobile device includes control of prioritizing the scene segments in FOV1 or FOV2.

9. The mobile device according to claim 1, wherein: The second camera is an active camera.

10. The mobile device according to claim 1, wherein: The mobile device is a smart phone.

11. The mobile device according to claim 1, wherein: The mobile device is a tablet computer.

12. The mobile device according to claim 1, wherein: The mobile device is a head-mounted device.

13. A mobile device, comprising: a first camera configured to capture first image data and having a first field of view (FOV1) directed toward a first scene; a second camera configured to capture second image data and having a second field of view (FOV2) directed toward a second scene including the user's eyes; as well as a processor comprising an object tracker and an eye tracker, wherein the object tracker is configured to perform object tracking using the first image data and the eye tracker is configured to perform gaze estimation using the second image data, The object tracker is configured to use line of sight estimation to verify the tracked object in the first scene.

14. The mobile device according to claim 13, wherein: The mobile device has a front side with a front surface and a back side with a back surface, wherein the front surface includes a screen, wherein the first camera is located on the back side, and wherein the second camera is located on the front side.

15. The mobile device according to claim 13, wherein: The line of sight estimation is performed directly by the user looking at a location in the scene.

16. The mobile device according to claim 13, wherein: The mobile device comprises a screen, wherein the line of sight estimation is performed indirectly by the user gazing at a position on the screen.

17. The mobile device according to claim 13, wherein: The object tracking provides additional scene information, and wherein the additional scene information is used to control the second camera.

18. The mobile device according to claim 17, wherein: Control of the second camera includes control selected from the group consisting of: control for focusing the second camera, control for personalized object selection, control for improving image quality of an area of ​​interest within FOV2, control for zooming the second camera, control for selecting an image resolution, and any combination thereof.

19. The mobile device according to claim 13, wherein: The object tracking provides additional context information, and wherein the additional context information is used to control the mobile device.

20. The mobile device according to claim 19, wherein: The control of the mobile device includes control of prioritizing the scene segments in FOV1 or FOV2.

21. The mobile device of claim 13, wherein: The second camera is an active camera.

22. The mobile device of claim 13, wherein: The mobile device is a smart phone.

23. The mobile device of claim 13, wherein: The mobile device is a tablet computer.

24. The mobile device of claim 1, wherein: The mobile device is a head-mounted device.

25. A mobile device, comprising: a first camera configured to capture first image data and having a first field of view (FOV1) directed toward a first scene; a second camera configured to capture second image data and having a second field of view (FOV2) directed toward a second scene including the user's eyes; as well as a processor comprising an object tracker and an eye tracker, wherein the object tracker is configured to perform object tracking using the first image data and the eye tracker is configured to perform gaze estimation using the second image data, Wherein the object tracker is configured to use the line of sight estimate to re-identify a tracked object in the first scene.

26. The mobile device of claim 25, wherein: The mobile device has a front side with a front surface and a back side with a back surface, wherein the front surface includes a screen, wherein the first camera is located on the back side, and wherein the second camera is located on the front side.

27. The mobile device of claim 25, wherein: The line of sight estimation is performed directly by the user looking at a location in the scene.

28. The mobile device of claim 25, wherein: The mobile device comprises a screen, wherein the line of sight estimation is performed indirectly by the user gazing at a position on the screen.

29. The mobile device of claim 25, wherein: The object tracking provides additional scene information, and wherein the additional scene information is used to control the second camera.

30. The mobile device of claim 29, wherein: Control of the second camera includes control selected from the group consisting of: control for focusing the second camera, control for personalizing object selection, control for improving image quality of an area of ​​interest within FOV2, control for zooming the second camera, control for selecting an image resolution, and any combination thereof.

31. The mobile device of claim 25, wherein: The object tracking provides additional context information, and wherein the additional context information is used to control the mobile device.

32. The mobile device of claim 31 , wherein: The control of the mobile device includes control of prioritizing the scene segments in FOV1 or FOV2.

33. The mobile device of claim 25, wherein: The second camera is an active camera.

34. The mobile device of claim 25, wherein: The mobile device is a smart phone.

35. The mobile device of claim 25, wherein: The mobile device is a tablet computer.

36. The mobile device of claim 25, wherein: The mobile device is a head-mounted device.

37. A system comprising: A first mobile device including a first camera, the first camera configured to capture first image data and having a first field of view (FOV1), the first camera being directed toward a first scene including a user's eyes; a second mobile device including a second camera configured to capture second image data and having a second field of view (FOV2), the second camera being directed toward a second scene; as well as One or more processors, including an object tracker and an eye tracker, wherein the eye tracker is configured to use the first image data to perform gaze estimation, and the object tracker is configured to use the gaze estimation to select an object in the second scene for object tracking and use the second image data to perform the object tracking.

38. The system of claim 37, wherein: The one or more processors are included in the second mobile device.

39. The system of claim 37, wherein: The one or more processors include a first processor and a second processor, wherein the first processor is included in the first mobile device, and wherein the second processor is included in the second mobile device.

40. The system of claim 38, wherein: The first mobile device is configured to transmit the first image data to the second mobile device.

41. The system of claim 39, wherein: The first processor performs the line of sight estimation, wherein the first mobile device is configured to transmit line of sight estimation information to the second mobile device, and wherein the second processor performs the object tracking.

42. The mobile device of claim 37, wherein: The line of sight estimation is performed directly by the user looking at a location in the scene.

43. The mobile device of claim 37, wherein: The first mobile device comprises a screen, and wherein the line of sight estimation is performed indirectly by the user looking at a location on the screen.

44. The mobile device of claim 37, wherein: The object tracking provides additional scene information, and wherein the additional scene information is used to control the second camera.

45. The mobile device of claim 44, wherein: Control of the second camera includes control selected from the group consisting of: control to focus the second camera, control to personalize object selection, control to improve image quality of an area of ​​interest within FOV2, control to zoom the second camera, control to select image resolution, and any combination thereof.

46. ​​The mobile device of claim 37, wherein: The first camera is an active camera.

47. The mobile device of claim 37, wherein: The first mobile device and the second mobile device are smart phones.

48. The mobile device of claim 37, wherein: The first mobile device is a head mounted device and the second mobile device is a smartphone.

49. The mobile device of claim 37, wherein: The first mobile device is a smart watch, and the second mobile device is a smart phone.

Images