Low Power Object Detection

Abstract

A system, method, and computer-readable medium for low power variable focus are provided. An exemplary method includes: acquiring a first image of a scene captured by an image sensor based on a trigger, the first image being captured with a lens of the image sensor in a first configuration of a plurality of available lens configurations; determining whether an object of interest is present in the first image based on the first image and a first detection result; adjusting the lens to a second configuration selected from the plurality of lens configurations in response to determining that the object of interest is not present in the first image; acquiring a second image of the scene by the image sensor while the lens is in the second configuration; and determining that the object of interest is present in the second image based on the second image and a second detection result.

Description

[Technical field]

[0001] This disclosure relates generally to low power variable focus, and more particularly to low power variable focus for object detection. [Background technology]

[0002]

[0002] Electronic devices are increasingly equipped with camera hardware to capture images and / or videos for consumption. For example, a computing device may include a camera to enable the computing device to capture videos or images of scenes, people, objects, etc. (e.g., a mobile device such as a mobile phone or smartphone including one or more cameras). The images or videos may be captured and processed by the computing device (e.g., a mobile device, an IP camera, etc.) and stored or output for consumption (e.g., displayed on the device and / or another device). In some cases, the images or videos may be further processed for effects (e.g., compression, image enhancement, image restoration, scaling, frame rate conversion, etc.) and / or for some applications such as computer vision, extended reality (e.g., augmented reality, virtual reality, etc.), image recognition (e.g., face recognition, object recognition, scene recognition, etc.), feature extraction, autonomous driving, and object detection, among others.

[0003] In some cases, electronic devices can process images to detect objects, faces, and / or any other items captured by the images. Object detection can be useful for various applications, such as, for example, face authentication, gesture recognition, surveillance, automation, among others. In some examples, electronic devices can implement lower power or "always-on" (AON) cameras that operate permanently or periodically to automatically detect some objects in the environment. Lower power cameras can be implemented for various use cases, such as, for example, permanent gesture detection, permanent face recognition for authentication, permanent face or other object (e.g., person, animal, vehicle, device, airplane, etc.) detection, permanent Quick Response (QR) Code scanning, etc. However, permanent and / or more frequent operation of lower power cameras and other camera setups can result in high overall power consumption. Moreover, mobile devices implementing such lower power cameras may suffer from reduced battery life, and fixed devices may require more complex heat dissipation designs and / or exhibit unacceptably low power efficiency during long-term use. Thus, significantly higher power consumption may adversely affect electronic device usage, device performance, and user experience. Summary of the Invention

[0004] In some examples, systems and techniques for reducing power consumed for object detection are described. According to at least one example, a method for lower power variable focus for object detection is provided. The method can include obtaining a first image of a scene captured by an image capture device based on a trigger event, the first image being captured with a lens of the image capture device in a first configuration of a plurality of available lens configurations, determining whether an object of interest is present in the first image based on the first image of the scene and a first detection result, adjusting the lens to a second configuration selected from the plurality of available lens configurations in response to determining that the object of interest is not present in the first image, obtaining a second image of the scene by the image capture device while the lens is in the second configuration, and determining that the object of interest is present in the second image based on the second image of the scene.

[0005] According to at least one example, a non-transitory computer-readable medium for lower power variable focus for object detection is provided. The non-transitory computer-readable medium can include instructions that, when executed by one or more processors, cause the one or more processors to: obtain a first image of a scene captured by an image capture device based on a trigger event, the first image being captured with a lens of the image capture device in a first configuration of a plurality of available lens configurations; determine whether an object of interest is present in the first image based on the first image of the scene and a first detection result; in response to determining that the object of interest is not present in the first image, adjust the lens to a second configuration selected from the plurality of available lens configurations; obtain a second image of the scene by the image capture device while the lens is in the second configuration; and determine that the object of interest is present in the second image based on the second image of the scene and a second detection result.

[0006] According to at least one example, an apparatus for lower power variable focus for object detection is provided. The apparatus may include a memory configured to store data and one or more processors coupled to the memory, the one or more processors configured to: obtain a first image of a scene captured by an image capture device based on a trigger event, the first image being captured with a lens of the image capture device in a first configuration of a plurality of available lens configurations; determine whether an object of interest is present in the first image based on the first image of the scene and a first detection result; adjust the lens to a second configuration selected from the plurality of available lens configurations in response to determining that the object of interest is not present in the first image; obtain a second image of the scene by the image capture device while the lens is in the second configuration; and determine that the object of interest is present in the second image based on the second image of the scene and a second detection result.

[0007] According to at least one example, another apparatus for lower power variable focus for object detection is provided, the apparatus including means for obtaining a first image of a scene captured by an image capture device based on a trigger event, the first image being captured with a lens of the image capture device in a first configuration of a plurality of available lens configurations, determining whether an object of interest is present in the first image based on the first image of the scene and a first detection result, adjusting the lens to a second configuration selected from the plurality of available lens configurations in response to determining that the object of interest is not present in the first image, obtaining a second image of the scene by the image capture device while the lens is in the second configuration, and determining that the object of interest is present in the second image based on the second image of the scene and a second detection result.

[0008]

[0008] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can select a second lens configuration from a plurality of available lens configurations based on an amount of power required by the image capture device to adjust the lens to the second lens configuration relative to one or more different amounts of power required by the image capture device to adjust the lens to one or more other lens configurations from the plurality of available lens configurations.

[0009] In some examples, the first detection result and the second detection result are reliability values.

[0010] In some examples, the first configuration can include a first lens position and the second configuration can include a second lens position that is different from the first lens position.

[0011] In some cases, adjusting the lens to the second configuration can include moving the lens from a first lens position to a second lens position using a focus motor.

[0012]

[0012] In some examples, the multiple available lens configurations may include multiple available lens positions, and the second lens position may be selected from the multiple available lens positions based on an amount of power used by the focus motor to move the lens to the second lens position relative to one or more amounts of power used by the focus motor to move the lens to one or more other positions from the multiple available lens positions.

[0013]

[0013] In some examples, the multiple available lens configurations may include multiple available lens positions, and the second lens position may be selected from the multiple available lens positions based on a reliability value associated with the first detection result.

[0014]

[0014] In some cases, selecting the second lens position may include reducing the number of the plurality of available lens positions by excluding one or more positions as unavailable lens positions for selection of the second lens position from the plurality of available lens positions based on the first reliability value.

[0015]

[0015] In some cases, selecting the second lens position may include comparing the amount of power to one or more amounts of power required by the focus motor to move the lens from the first position to one or more other positions from a plurality of available lens positions.

[0016]

[0016] In some examples, the multiple available lens configurations can include multiple available lens positions, and the second lens position is selected from the multiple available lens positions based on a reliability value associated with lens displacement from the first lens position to the second lens position.

[0017] In some examples, the plurality of available lens configurations comprises a plurality of available lens positions, and the second lens position is selected from the plurality of available lens positions based on a priority of focal lengths associated with the plurality of available lens positions. In some examples, the priority of focal lengths associated with the plurality of available lens positions is based on at least one of a detection reliability associated with the lens displacement from the first lens position to the second lens position, a respective likelihood of detecting the object of interest in an image captured from each of the plurality of available lens positions, and an amount of power used by a focus motor to move the lens to each of the plurality of available lens positions. In some cases, the priority of focal lengths associated with the plurality of available lens positions is based on a default type of object of interest or a type of object detected in the second image.

[0018]

[0018] In some examples, the multiple available lens configurations can include multiple available lens positions, and the second lens position can be selected from the multiple available lens positions based on a relative distance between the first position and each of the multiple available lens positions.

[0019] In some examples, the plurality of available lens configurations can include a plurality of available lens positions, and the second position is selected from the plurality of available lens positions based on one or more characteristics of the lens. In some cases, the one or more characteristics of the lens can include at least one of an aperture associated with the lens, a field of view associated with the lens, and a focus power profile associated with the lens.

[0020] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can select a lens from a plurality of available lenses based on one or more characteristics of the lens and a focal length associated with the object of interest. In some examples, the one or more characteristics can include at least one of an aperture, a field of view, and a focus power profile.

[0021]

[0021] In some examples, a lens is selected from the plurality of available lenses based on a determination that a focus power profile associated with the lens includes a lower focus power than a respective focus power profile of one or more lenses from the plurality of available lenses.

[0022] In some cases, the object of interest may include at least one of a document, a Quick Response (QR) code, a face, a finger, a hand, a device, a product, and an animal. In some cases, the triggering event may include an inertial motion above a threshold, an audio change above a threshold, an ambient light change above a threshold, a change in range to an object above a threshold, a trigger from an application, a depth measurement from an active depth sensing system, a trigger from a global navigation satellite system, a trigger from a global positioning system, a data connection, and a phase detection change above a threshold.

[0023]

[0023] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can adjust the different image capture device in response to determining that the object of interest is present in the second image. In some examples, adjusting the different image capture device can include turning on the different image capture device and / or initializing the different image capture device. In some cases, the different image capture device can include a main camera device, and / or a higher power camera device than the image capture device. In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can process one or more images of the scene via the different image capture device. In some cases, the one or more images can include a second image, and / or a third image captured by a different image capture device.

[0024] In some aspects, the methods, non-transitory computer readable media, and apparatus described above can maintain the lens in the second configuration based on the second detection result.

[0025] In some cases, the first configuration can include an inactive optical image stabilization mode and the second configuration can include an active optical image stabilization mode. In some examples, adjusting the lens to the second configuration can include activating the optical image stabilization mode using a lens stabilization motor.

[0026] In some cases, the first configuration can include a first aperture setting and the second configuration can include a second aperture setting different from the first aperture setting. In some cases, adjusting the lens to the second configuration can include changing an aperture of the lens from the first aperture setting to the second aperture setting using an aperture motor.

[0027] In some aspects, the device may be or include a camera (e.g., an IP camera), a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, an Internet of Things (IoT) device, a smart wearable device, or another device. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the devices described above may include one or more sensors (e.g., one or more accelerometers, gyroscopes, inertial measurement units (IMUs), motion detection sensors, and / or other sensors).

[0028]

[0028] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0029]

[0029] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0030]

[0030] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Brief description of the drawings]

[0031] [Figure 1]

[0031] FIG. 1 illustrates an example of a user using a mobile device configured to perform object detection, in accordance with some examples of the present disclosure. [Figure 2A]

[0032] FIG. 2 illustrates a top-down view of a pixel array configuration of an image sensor in which two side-by-side focus pixels are covered by a 2 pixel by 1 pixel microlens, according to some examples of the present disclosure. [Figure 2B]

[0033] A diagram showing a top-down view of a pixel array configuration of an image sensor in which four neighboring focus pixels are covered by a 2 pixel x 2 pixel microlens, in accordance with some examples of the present disclosure. [Figure 3A]

[0034] FIG. 2 illustrates a top-down view of a pixel array configuration of an image sensor, in which at least one focus pixel has two photodiodes, according to some examples of the present disclosure. [Figure 3B]

[0035] FIG. 2 illustrates a top-down view of a pixel array configuration of an image sensor, with at least one focus pixel having four photodiodes, according to some examples of the present disclosure. [Figure 4]

[0036] FIG. 1 illustrates an example image processing system for low power variable focus object detection, in accordance with some examples of the present disclosure. [Diagram 5]

[0037] 1 illustrates an example process for low power variable focus object detection in accordance with some examples of this disclosure. [Figure 6]

[0038] 1A-1C show example timelines of focus adjustments and associated power profiles, in accordance with some examples of the present disclosure. [Figure 7]

[0039] 1 illustrates an example process for low power variable focus object detection in accordance with some examples of this disclosure. [Figure 8]

[0040] 1A-1C are diagrams illustrating an example use of phase detection autofocus data to estimate the distance of an object of interest within the field of view of a camera system, in accordance with some examples of the present disclosure. [Figure 9]

[0041] 1A-1C illustrate example images captured using different apertures, in accordance with certain examples of the present disclosure. [Figure 10]

[0042] 1 is a flowchart illustrating an example process for object detection using multiple image sensors, in accordance with some examples of this disclosure. [Figure 11]

[0043] 1 is a flowchart illustrating an example process for low power variable focus object detection in accordance with some examples of this disclosure. [Figure 12]

[0044] FIG. 1 illustrates an exemplary computing device architecture, in accordance with some examples of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032]

[0045] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be implemented without these specific details. The figures and descriptions are not limiting.

[0033]

[0046] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0034]

[0047] Some electronic devices (e.g., smartphones, laptop computers, tablets, wearable devices, cameras, etc.) are increasingly leveraging camera hardware for use cases where the camera can operate permanently or periodically when the electronic device implementing the camera is in a certain power state. For example, an electronic device can implement a camera that can capture images while the electronic device is in a lower power state, a locked state, and / or other state. In some examples, the camera can capture images while the electronic device has available battery power, the battery power level is above a threshold, the electronic device is awake, etc. In some cases, the camera can include a lower power camera or a "power sensitive" camera (often referred to as an "always on" (AON) camera) that is configured to detect objects / events periodically, instantly, continuously, or on demand. In some cases, the lower power camera can automatically detect some objects as needed / desired while maintaining a lower power usage footprint. In some examples, the lower power camera can implement lower power hardware and energy-efficient image processing software used to detect events / objects. A lower power camera can remain on or "wake up" to watch movement in a scene and detect events / objects in the scene while using less battery power than other devices, such as higher power / resolution cameras. Upon discovering an object, the camera can trigger one or more actions, such as, for example, object detection, object recognition, face recognition, image processing tasks, among other actions.

[0035]

[0048] Cameras implemented by electronic devices can detect objects for a variety of use cases, such as, but not limited to, persistent gesture detection, persistent face recognition for device authentication / unlocking, persistent face or object (e.g., people, animals, vehicles, devices, etc.) detection, persistent Quick Response (QR) code scanning, etc. By way of example, a camera on a mobile device can detect a QR code when the QR code is within the camera's field of view (FOV), and upon detecting the QR code, can "wake up" the mobile device without the user needing to power on or unlock the mobile device.

[0036]

[0049] Although a lower power camera setup may leverage lower power camera hardware for reduced power consumption, the overall power consumption of the camera setup may still significantly reduce the battery life of a mobile device that generally has a more limited battery life. For example, the camera setup may implement lens focus adjustment techniques to move, reposition, and / or otherwise vary the focus of a lens module to focus on a target and / or increase the sharpness of the target in a captured image, given the characteristic(s) of the target, characteristic(s) of the environment, characteristic(s) of the lens / camera configuration, the relative distance between the target and the camera, and / or any other factors. Non-limiting examples of lens focus adjustment components can include, among others, a voice coil motor (VCM), a piezoelectric motor, a stepper motor, an ultrasonic motor, an electroactive polymer motor, a liquid lens electrowetting, an electromagnetic focus motor, a geared direct current (DC) motor, a direct drive super sonic wave motor, and a solid lens controller.

[0037]

[0050] The lens focus adjustment component may require and / or consume a significant amount of power to change the focus of the lens module. In many cases, the power consumption of the lens focus adjustment component may increase when changing the focus of the lens module away from a neutral (or infinity) position. Such changes in the focus of the lens module may result in a large power draw by the camera setup, which may significantly affect the battery life of the mobile device. Moreover, since the object of interest is often at different distances from the camera, and rear-facing cameras generally include variable focus lenses, the lens focus adjustment component may have to frequently change the lens focus, aperture setting, image stabilization, etc. to focus on the object of interest. Changes in focus, aperture setting, image stabilization, etc., by moving / positioning the lens focus adjustment component away from a neutral or current position may draw a significant amount of power as objects at different distances are observed in the scene. Such power draw may adversely affect the device and / or the performance of the device.

[0038]

[0051] Described herein are systems, apparatuses, processes (or methods), and computer-readable media (collectively referred to as "systems and techniques") for reducing power consumption of lens adjustment components (e.g., focus motors, lens stabilization motors, aperture motors, controllers, drivers, circuits, etc.) implemented in a camera setup. In some examples, the systems and techniques described herein can reduce the number of lens movements required to successfully detect an object of interest. In some cases, the systems and techniques described herein can reduce or minimize the amount of time spent by a lens (or lens module) being focused away from a neutral or current position of the lens focus adjustment components. In some cases, the systems and techniques described herein can increase the amount of time that the lens (or lens module) is maintained in a first configuration (e.g., a non-powered or neutral configuration) despite ongoing object detection attempts, i.e., attempts to detect one or more objects of interest. The neutral (or infinity) configuration of the lens can be the lowest power consumption state of the lens, a non-powered position, a position associated with a lower power draw than a non-neutral or other position, etc. For example, in some cases, the neutral (or infinity) position of the lens configuration may be a minimum power consumption state in which a spring loads the lens to remain in a neutral lens position, e.g., a spring load is used to oppose lens movement during focusing operations. The reduced power consumption of the lens focus adjustment components may be used to reduce the overall power consumption of object detection systems, image capture devices with variable focus lenses, and / or any other image capture systems capable of varying the focus of a lens module.

[0039]

[0052] In some examples, the systems and techniques described herein may test or test several lens positions consecutively for detection of an object of interest. In some cases, the systems and techniques described herein may test or test lens positions intelligently and / or in a specific order to reduce or minimize the amount of power used to detect an object of interest, reduce or minimize the lens changes performed before finding a suitable lens position (and / or achieve a desired focus). For example, the systems and techniques described herein may test or test lens positions in an order determined based on the respective focus motor power requirements for various lens positions. In some cases, the systems and techniques described herein may first test or test the lens position requiring the lowest amount of focus motor power, after which it may progressively test other lens positions as needed in order of increasing focus motor power consumption.

[0040]

[0053] In some examples, the image capture system can initiate the object detection process based on one or more triggering events. The triggering events can include, for example, a measured inertial motion above a threshold, an audio change above a threshold detected by an audio sensor, an ambient light change above a threshold, a detected change in range to an object above a threshold, a phase detection change above a threshold, a trigger from an application (e.g., a command, a request, a parameter, etc.), a trigger from an active depth sensing system (e.g., a threshold measurement, etc.), a trigger from a Global Navigation Satellite System (GNSS) or Global Positioning System (GPS), a trigger from a data connection (e.g., a Bluetooth connection, a Wi-Fi connection, a wireless local area network connection, a wide area network connection, etc.), etc. Upon triggering the object detection process, the image capture system can capture an image and determine whether an object of interest is present or absent in the captured image. In one non-limiting example, the image capture system can calculate a confidence based on the captured image that indicates the certainty / likelihood that an object of interest is present or absent in the captured image. For example, the image capture system can determine a confidence that indicates the likelihood that a QR code is present in the captured image.

[0041]

[0054] The image capture system can calculate the reliability based on the image captured from the first lens position. If the reliability indicates that the object of interest is present in the image, the image processing system can detect the object of interest from the first lens position. The image capture system can maintain the lens at the first lens position, and no lens adjustment to the second lens position is required. Thus, the image processing system can detect the object of interest if the reliability is equal to or above an upper threshold. This does not require that the captured object of interest be in focus in the image. If the reliability is equal to or above an upper threshold, the image processing system can detect the object of interest without a lens adjustment, but rather by maintaining the lens at the first lens position. This contributes to an overall power consumption reduction. If the reliability is below the upper threshold and above the lower threshold, the image capture system can capture another image from the second lens position, and can calculate the reliability based on the image captured from the second lens position. If the reliability indicates that the object of interest is present in the image captured from the second lens position, the image processing system can detect the object of interest from the second lens position. If the confidence falls below the upper threshold and above the lower threshold, the image capture system can capture another image from a third lens position and calculate another confidence based on the captured image. The image capture system can successively check several detection confidences at different lens positions until an object is detected (while satisfying the threshold), or if the confidence falls below the upper threshold and below the lower threshold, a determination can be made that an object is not present.

[0042]

[0055] The image capture system may prioritize some lens positions based on several factors, such as, for example, power consumption associated with different lens positions, FOV and / or depth-of-field characteristics of the lens, current or neutral position of the focus motor, object detectability associated with different lens positions, risk-reward evaluation / comparison calculated for different lens positions, etc. In some cases, the image capture system may begin the object detection check from the neutral or current position of the focus motor. For example, the image capture system may begin with a detection confidence from a neutral position of the focus motor, which in some cases may not require / involve a power draw from the focus motor. If the confidence from the neutral position is below a threshold, the focus motor may move the lens of the image capture device to the next lower power position (e.g., a different position that involves the next lower amount of power draw from the focus motor to move the lens to that position). If the confidence from the next lower power position is below a threshold, the focus motor may move the lens to the next lower power position and check the confidence from that position as well. The image capture system can continue any additional inspections until the object is detected or the process is completed. In this way, the object detection trigger and focus motor power consumption can be used to detect the object from the lowest focus power position required, instead of requiring a full autofocus process at higher power.

[0043]

[0056] In other examples, the image capture system may use a laser range finder to determine a sequence of lens positions to inspect and / or aid in the calculation of detection reliability. Additionally, the image capture system may leverage phase detection autofocus (PDAF) data to guide lens placement / repositioning and / or aid in the calculation of detection reliability. The image capture system may additionally or alternatively leverage other data, such as contrast-based autofocus (AF) search (course or fine), depth information from stereo data, etc., to guide lens placement / repositioning and / or aid in the calculation of detection reliability. In some cases, the image capture system may adjust the lens aperture to increase the lens depth of field to potentially reduce the number of lens positions tested. In some cases, the image capture system may implement one or more other sensors in different setups to reduce or limit lens motor retraction across the system.

[0044]

[0057] 1 is a diagram illustrating an example of a user 100 using a mobile device 102 configured to perform object detection as described herein. In some examples, the mobile device 102 is a mobile phone (e.g., a smartphone with Internet and voice capabilities). In other examples, the mobile device 102 can include any other type of electronic device, such as, for example, but not limited to, a tablet computer, a laptop computer, a camera system, an Internet of Things (IoT) device, a smart wearable device (e.g., a head mounted display, smart glasses, a smart watch, etc.), a smart television, or any other electronic device with an image sensor. In some implementations, the mobile device 102 can have a system architecture similar to the computing system 1200 described below with respect to FIG. 12.

[0045]

[0058] In this example, the mobile device 102 includes a forward-facing camera 104 that is configured and capable of capturing images of a physical scene or environment within a field of view (FOV) of the camera 104. In some cases, the forward-facing camera 104 may include a low-power camera and / or a camera configured to operate permanently or periodically when the mobile device 102 is in a power state. In some examples, the forward-facing camera 104 may capture images while the mobile device 102 is in a lower power state, a locked state, and / or other state. In some examples, the forward-facing camera 104 may capture images while the mobile device 102 has available battery power, a battery power level is above a threshold, the mobile device 102 is awake, etc. In some cases, the forward-facing camera 104 may include a lower power or "power sensitive" camera (often referred to as an "always on" (AON) camera) configured to detect objects / events periodically, continuously, or on demand. In some cases, the lower power camera can automatically detect some objects as needed / desired while maintaining a lower power usage footprint. In some examples, the lower power camera can implement lower power hardware and energy efficient image processing software. The lower power camera can stay on or "wake up" to watch movement in the scene and detect events / objects in the scene while using less battery power than other devices, such as higher power / resolution cameras.

[0046]

[0059] In some examples, the front-facing camera 104 may include a low-power camera that passively captures images without requiring an explicit command (e.g., based on user input) requesting the capture of an image. In some cases, the front-facing camera 104 may have a lower frame rate and capture fewer images than the frame rate of a higher power / resolution camera. In some examples, images captured by the front-facing camera 104 (as a lower-power camera that operates permanently or periodically when the mobile device 104 is in a lower power or locked state) are not stored except as needed to perform the object detection described herein. For example, images captured by the front-facing camera 104 (as a lower-power camera that operates permanently or periodically when the mobile device 104 is in a lower power or locked state) may be temporarily cached for use by one or more processors to perform the object detection described herein.

[0047]

[0060] In some cases, the front-facing camera 104 may be activated to begin capturing images when a trigger is detected. When a trigger is detected, the front-facing camera 104 may capture an image of the user 100, and one or more processors of the mobile device 102 may perform the object detection process described herein. In some cases, the trigger may include, for example, but not limited to, inertial movement above a threshold, an audio change above a threshold, an ambient light change above a threshold, a phase detection-based depth change above a threshold, a change in range to one or more objects above a threshold, a detected presence of one or more objects, a scene change, a stereo-based depth change above a threshold, combinations thereof, or any other configured trigger.

[0048]

[0061] In one illustrative example, a scene change may be detected when a change in pixel data above a scene change threshold is detected. The scene change may trigger activation of the front-facing camera 104 to begin capturing one or more images. The scene change threshold may be based on the amount of pixels in a first image that differ from corresponding pixels (at a common location) in a second image or multiple images. For example, a scene change may be detected when at least 20% of the pixels in a first image differ from corresponding pixels (at a common location) in a second image. In another illustrative example, the front-facing camera 104 may be activated to begin capturing images when motion is detected. In some examples, motion may be detected using optical motion sensors, accelerometers, gyroscopes, inertial measurement units (IMUs), and / or other sensors or components of the mobile device 102.

[0049]

[0062] The front-facing camera 104 may include one or more motors (not shown) that move the lens of the front-facing camera 104 between lens positions corresponding to different states (e.g., forward focus state, rear focus state, in focus state) and one or more motor actuators (not shown) that the mobile device 102 activates to operate the motors. Non-limiting examples of lens motors may include voice coil motors (VCMs), piezoelectric motors, stepper motors, ultrasonic motors, electroactive polymer motors, electromagnetic focus motors, geared direct current (DC) motors, and direct drive supersonic wave motors, among others. The front-facing camera 104 may also include various additional unillustrated components, such as lenses, mirrors, partially reflective (PR) mirrors, prisms, photodiodes, image sensors, processors, and / or other components that may be implemented in a camera or other optical instrument, in some cases.

[0050]

[0063] 2A and 2B show top-down views of an example pixel array configuration of an image sensor. An image sensor of a camera system (e.g., front-facing camera 104) may include an array of pixels, such as pixel array 230 of FIG. 2A or pixel array 240 of FIG. 2B. A pixel array (e.g., pixel array 230, pixel array 240) may include an array of photodiodes and microlenses. Both the 2 pixel by 1 pixel microlens 232 of FIG. 2A and the 2 pixel by 2 pixel microlens 242 of FIG. 2B span multiple adjacent focus pixels (e.g., the microlens covers multiple adjacent focus pixel photodiodes) and both may limit the amount and / or direction of light that strikes the focus pixel photodiodes of those focus pixels.

[0051]

[0064] When a pixel array (e.g., pixel array 230, pixel array 240) captures a frame and thus captures focus pixel data for each focus pixel, the focus pixel data from paired focus pixels may be compared with each other. For example, focus pixel data from a left focus pixel photodiode may be compared with focus pixel data from a right focus pixel photodiode, and focus pixel data from a top focus pixel photodiode may be compared with focus pixel data from a bottom focus pixel photodiode. If the compared focus pixel data values ​​are different, this difference is called a phase disparity, also known as a phase difference, a defocus value, or a separation error. A focus pixel under a 2 pixel by 2 pixel microlens 242, as in FIG. 2B, essentially has two vertically adjacent horizontally oriented pairs of focus pixels and / or two horizontally adjacent vertically oriented pairs of focus pixels. Thus, focus pixel data from the UL focus pixels may be compared with focus pixel data from the BL focus pixels (as a top / bottom pair), focus pixel data from the UR focus pixels may be compared with focus pixel data from the BR focus pixels (as a top / bottom pair), focus pixel data from the UL focus pixels may be compared with focus pixel data from the UR focus pixels (as a left / right pair), focus pixel data from the BL focus pixels may be compared with focus pixel data from the BR focus pixels (as a left / right pair), or some combination thereof.

[0052]

[0065] 3A shows a top-down view of a pixel array configuration of an image sensor, with at least one focus pixel having two photodiodes. In particular, a 4 pixel by 4 pixel pixel array 350 with four focus pixels is shown in FIG. 3A. The four focus pixels shown in pixel array 350 each include two photodiodes, with the left and right photodiodes of each focus pixel's photodiode pair being labeled "L" and "R", respectively. A focus pixel with two photodiodes is sometimes referred to as a dual photodiode (2PD) focus pixel.

[0053]

[0066] One of the 2PD focus pixels in FIG. 3A is labeled as the 2PD focus pixel 352. The left photodiode (L) of the 2PD focus pixel 352 is labeled as the "left photodiode 354L" and the right photodiode (R) of the 2PD focus pixel 352 is labeled as the "right photodiode 354R". For each captured frame, the left photodiode 354L and the right photodiode 354R may capture light received by the 2PD focus pixel 352 from different angles. For a given frame, the data captured by the left photodiode 354L may be referred to as a left image or left image data, and the data captured by the right photodiode 354R may be referred to as a right image or right image data. The left image data and the right image data may be compared to determine a phase disparity. The pixel array 350 shown in FIG. 3A is a "low density" 2PD pixel array in which only some of the pixels in the pixel array 350 include two photodiodes (i.e., focus pixels). The remaining pixels are imaging pixels and contain only a single photodiode. In some cases, a "high density" 2PD pixel array may be used instead, in which every pixel in the pixel array (or a higher percentage of the pixels in the pixel array) contains two photodiodes, and in some cases can act as both a focus pixel and an imaging pixel simultaneously, or can switch between acting as a focus pixel for one frame and an imaging pixel for another frame.

[0054]

[0067] FIG. 3B shows a top-down view of a pixel array configuration of an image sensor in which at least one focus pixel has four photodiodes. The pixel array 360 includes focus pixels, commonly referred to as 4PD focus pixels or Quadrature Phase Detection (QPD) focus pixels, each focus pixel including four diodes. For example, a 4PD focus pixel 362 is labeled in FIG. 3B and includes an upper left photodiode labeled with the letters "UL", an upper right photodiode labeled with the letters "UR", a lower left photodiode labeled with the letters "BL", and a lower right photodiode labeled with the letters "BR". Data from each photodiode of the 4PD focus pixel 362 can be compared to data from adjacent photodiodes of the 4PD focus pixel 362 to determine a phase difference. The pixel array 360 is a "low density" 4PD pixel array in which only some of the pixels in the pixel array 360 include four photodiodes (i.e., focus pixels). The remaining pixels are imaging pixels and include only a single photodiode. However, in some cases, a "high density" 4PD pixel array may be used instead, in which every pixel in the pixel array (or a higher percentage of the pixels in the pixel array) contains four photodiodes, and in some cases can act as both a focus pixel and an imaging pixel simultaneously, or can switch between acting as a focus pixel for one frame and as an imaging pixel for another frame.

[0055]

[0068] 4 illustrates an example image processing system 400 for low power variable focus object detection. The image processing system 400 may be implemented by one or more computing devices, such as, for example, a system on a chip (SoC). In some examples, the image processing system 400 may be part of and / or implemented by the mobile device 102 shown in FIG. 1. The image processing system 400 may include a sensor data processing engine 402, a camera initiator 404, a focus / lens controller 406, and an object detector 408.

[0056]

[0069] In some cases, the image processing system 400 can optionally include applications 410 related to object detection, such as applications accessed based on object detection, applications configured to require authentication based on object detection, applications that use and / or receive object detection results, and / or any other applications. In other cases, the applications 410 can be implemented by a separate system or a separate device(s) within the same system. In some examples, the focus / lens controller 406 can include and / or control a focus motor or controller. For example, the focus / lens controller 406 can include and / or control a voice coil motor (VCM), a piezoelectric motor, a stepper motor, an ultrasonic motor, an electroactive polymer motor, a liquid lens electrowetting device, an electromagnetic focus motor, a geared direct current (DC) motor, a direct drive supersonic wave motor, a solid lens controller, or any other focus motor, controller, driver, actuator, and / or component.

[0057]

[0070] The sensor data processing engine 402 can process sensor data 430 from one or more sensors, which is used to detect a trigger event to initiate the object detection process described herein. The sensor data 430 can include data from one or more sensors, such as, for example, but not limited to, a gyroscope, an accelerometer, an IMU, an audio sensor, an ambient light sensor, a depth sensor and / or a laser range finder, an optical motion sensor, a barometer, a temperature sensor, an altimeter, a radar, a global positioning system (GPS) device, and / or any other type of sensor. In some examples, the sensor data 430 can include motion data, audio data, position data, location data, altitude data, ambient light measurements, temperature measurements, distance measurements, pressure measurements, radar returns, and / or any other type of sensor data.

[0058]

[0071] The sensor data processing engine 402 can process the sensor data 430 and provide the processed sensor data to the camera initiator 404. The camera initiator 404 can use the processed sensor data and image data 432 from the camera 420 to detect a trigger event that can indicate the presence of an object of interest to be detected and / or can trigger the object detection process described herein. In some examples, the camera initiator 404 can collect and / or coalesce a number of optical and / or non-optical trigger events that can indicate the presence of an object of interest to be detected.

[0059]

[0072] Non-limiting examples of triggers that may be configured to trigger the object detection process described herein include inertial movement above a threshold, a gesture that may indicate an attempt / intent to initiate object detection (e.g., a gesture indicating an attempt to scan an item such as a QR code or document, a gesture indicating an attempt to scan / detect a face, a hand or finger gesture, etc.), a detection / indication of the presence of an object of interest (e.g., inertial sensor based presence detection, audio sensor based presence detection, depth sensor based presence detection, etc.), an audio change above a threshold, an object specific trigger (e.g., ultrasonic signaling, automobile noise, animal noise, human voice / voice, etc.), a light change above a threshold, a change in range to an object above a threshold, a PD based depth change above a threshold, a trigger from an application (e.g., a command, request, parameter, etc.), a trigger from an active depth sensing system (e.g., a threshold measurement, etc.), a trigger from a GNSS / GPS, a trigger from a data connection (e.g., a Bluetooth connection, a Wi-Fi connection, a wireless local area network connection, a wide area network connection, etc.), and / or any other predetermined trigger or combination thereof. Additionally, non-limiting examples of objects of interest may include a QR code, a document, a face, a product, a hand, a finger, an animal, a sign, text, a symbol, an image, a landmark, a vehicle, a person, a device, and / or any other item(s) that can be captured and detected in an image.

[0060]

[0073] The camera 420 may be implemented and / or part of the image processing system 400 or a device(s) implementing the image processing system 400. In some examples, the camera 420 may be or be implemented by the front-facing camera 104 shown in FIG. 1. In other examples, the camera 420 may be a different camera implemented by the image processing system 400, the device implementing the image processing system 400, or any other device. In some cases, the camera 420 may be or correspond to a lower power camera, as previously described. In some examples, the camera 420 may be a low-power image sensor that passively captures images without requiring an explicit command based on user input requesting the capture of an image. In some examples, the camera 420 may operate permanently or periodically when the image processing system 400 is in one or more states, such as a low power state, a locked state, an active state, an inactive state, a charging state, and / or any other state.

[0061]

[0074] The camera initiator 404 can analyze the image data 432 and the sensor data received from the sensor data processing engine 402 to detect one or more triggers for object detection. When the camera initiator 404 detects a trigger, the camera initiator 404 can send a signal to the object detector 408 to instruct / trigger the object detector 408 to determine whether an object of interest is present or absent in the scene. In some examples, the camera initiator 404 can also send the image data 432 (and optionally any sensor data from the sensor data processing engine 402) to the object detector 408, and the object detector 408 can use such data to determine whether an object of interest is present or absent.

[0062]

[0075] In some cases, the camera initiator 404 can optionally notify the focus / lens controller 406 of the detected trigger. For example, the camera initiator 404 can send an indication and / or description of the detected trigger to the focus / lens controller 406. The focus / lens controller 406 can optionally use information about the trigger, along with the output from the object detector 408, to control the operation and / or configuration of the camera 420 as described herein. For example, the focus / lens controller 406 can use information about the trigger (e.g., time of trigger, type of trigger, trigger parameters, etc.), along with the output from the object detector 408, to adjust the configuration (e.g., lens position) of the lens associated with the camera 420.

[0063]

[0076] The object detector 408 can analyze images captured by the camera 420 (e.g., image data 432) and generate a detection result that may include an indication of the certainty, determination, prediction, reliability, and / or likelihood of the presence of an object of interest in the image. In some examples, the object detector 408 can analyze images captured by the camera 420 (e.g., image data 432) and generate an indication of the certainty, determination, prediction, reliability, and / or likelihood of the presence and / or absence of the object of interest. For example, the object detector 408 can analyze the captured image to determine whether the object of interest is present in the image and output a reliability value from 0 to 1, where 0 indicates that the object detector 408 is highly or maximally confident that the object of interest is not present in the image, 0.5 indicates that the object detector 408 is not confident or maximally confident that the object of interest is present in the image, and 1 indicates that the object detector 408 is highly or maximally confident that the object of interest is present in the image. The confidence values ​​from 0 to 1 in this example are provided herein for purposes of explanation as just one illustrative example of confidence values ​​generated by the object detector 408. Those skilled in the art will recognize from this disclosure that in other examples, the object detector 408 can generate other types of confidence values ​​and / or the confidence values ​​can be based on other conventions. In other cases, the object detector 408 can instead or additionally generate a probability score or metric, a probabilistic prediction, a classification and / or label, a cost or loss function value, an estimate, a ranking, a binary prediction (e.g., "yes" or "no", "true" or "false", "present" or "not present", etc.), a set of probabilities, etc., to indicate whether the object of interest is present or absent.

[0064]

[0077] In some aspects, the object detector 408 may implement one or more algorithms, networks, and / or statistical models for generating a confidence value based on the captured image. For example, in some cases, the object detector 408 may implement an image processing algorithm, a machine learning algorithm, a Bayesian network, a conditional random field, a Markov chain, a tensor factorization, a neural network, any combination thereof, or any other algorithm, network, and / or statistical model. The object detector 408 may determine that an object of interest is present in the image if the generated confidence value is equal to or above an upper threshold (e.g., a value of 1 in the previous example). The object detector 408 may determine that an object of interest is not present in the image if the confidence value is equal to or below a lower threshold (e.g., a value of 0 in the previous example). The object detector 408 may indicate uncertainty regarding whether an object of interest is present or not present in the image when the generated confidence value is above a lower threshold but below an upper threshold (e.g., above 0 and below 1 in the previous example).

[0065]

[0078] In some cases, if the object detector 408 determines that an object of interest is present in the captured image and detects the object in the image, the object detector 408 may provide a detection output (e.g., a detection result) to the application 410. For example, the object detector 408 may notify the application 410 that an object has been detected in the image (e.g., an object has been determined to be present in the image). As another example, the object detector 408 may provide the detection result to the application 410, such as an indication and / or description of the detected object, a confidence value indicating that the object is present in the image, and / or any other information regarding the detected object and / or the detection of the object. In some examples, if the object detector 408 determines that an object of interest is present in the captured image, the object detector 408, the application 410, or the image processing system 400 may "wake up" or initialize another camera device implemented by the image processing system 400.

[0066]

[0079] The other camera device may include, for example, a higher power / resolution camera device and / or image sensor. In some examples, the other camera device may capture higher resolution images than camera 420, implement a higher frame rate than camera 420, implement or invoke additional and / or more computationally intensive image processing tasks / algorithms than camera 420, implement camera hardware with higher power and / or capabilities than camera 420, etc. Image processing system 400 may use the other camera to capture one or more images of the detected object (and / or other object(s) or portion of the scene) for further processing. In some examples, image processing system 400 may use the other camera to process images captured by camera 420 (and / or by other cameras) at lens positions used to capture images of the detected object, recognize the detected object (e.g., recognize a QR code, recognize a face, recognize a device, etc.), etc.

[0067]

[0080] In some cases, image processing system 400 may switch back to camera 420 from the other camera device when image processing system 400 transitions to a different power state (e.g., a lower power state, a locked state, an inactive state, etc.), when an object of interest is not detected for a threshold time period, when the other camera device is not used (and / or remains inactive) for a threshold time period, etc. For example, image processing system 400 may place the other camera device in an inactive, sleep, off, or lower power state and initialize camera 420 for use until another switch to a different camera device is triggered.

[0068]

[0081] The object detector 408 can communicate with the focus / lens controller 406 to cause the focus / lens controller 406 to adjust a configuration of the camera 420, such as a position of the lens of the camera 420, based on a detection result from the object detector 408, such as a confidence value generated by the object detector 408. For example, if the object detector 408 determines that an object of interest is not present in the image, the object detector 408 can notify the focus / lens controller 406 that an object is not present in the image. The focus / lens controller 406 can determine an action to take based on the notification that an object is not present in the image. For example, the focus / lens controller 406 can command the camera 420 to turn off if an object of interest is not detected in the image.

[0069]

[0082] As another example, if the object detector 408 cannot determine using a certainty threshold that the object of interest is present or absent in the image (e.g., if the confidence value is above a lower threshold but below an upper threshold), the object detector 408 may inform the focus / lens controller 406 of such uncertainty to cause the focus / lens controller 406 to adjust a configuration of the camera 420, such as a lens position and / or lens aperture, and trigger the camera 420 to capture another image using the adjusted configuration. The object detector 408 may use the other image captured using the adjusted configuration (e.g., adjusted lens position, adjusted lens aperture, etc.) to make another determination regarding whether the object of interest is present or absent. Based on the adjusted configuration, the other image captured by the camera 420 may have one or more properties / characteristics (e.g., focus / sharpness, quality, etc.) that are different from the previous image used by the object detector 408 to determine whether the object of interest is present or absent in the image. The other image with one or more different properties / characteristics may enable the object detector 408 to more reliably determine whether the object of interest is present or absent.

[0070]

[0083] The focus / lens controller 406 can adjust one or more configurations associated with the camera 420 based on the detection results (e.g., confidence values, etc.) from the object detector 408, and optionally further based on trigger information from the camera initiator 404. For example, the focus / lens controller 406 can control the image sensor focus, the power settings of the camera 420, and / or one or more other configuration settings, such as, for example, aperture, based on events / information from the object detector 408 (and optionally from the camera initiator 404). To illustrate, in some cases, the focus / lens controller 406 can adjust the lens position to change the focal length for capturing one or more images (e.g., via the camera 420) with different properties / characteristics (e.g., focus / sharpness, quality, etc.) that the object detector 408 can use to make more reliable / accurate object presence determinations and / or object detection.

[0071]

[0084] As another example, the focus / lens controller 406 can adjust the aperture used by the camera 420 to change the depth of field and / or one or more other properties / characteristics of any additional images captured by the camera 420 and used by the object detector 408 to determine if the object of interest is present and / or to detect the object. As another example, the focus / lens controller 406 can generate instructions to turn off the camera 420 (and / or stop capturing additional images until further instructions) if the confidence value from the object detector 408 indicates that the object of interest is not present in the image.

[0072]

[0085] Some focus technologies, such as piezoelectric or stepper motors, may not have a neutral focal length. Moreover, some focus technologies may have nonlinear or time components to their focus power function. In some cases, these focus technologies may also be incorporated into the logic of the focus / lens controller 406. For example, in the case of a piezoelectric focus motor, which may only draw power when moving the lens, the focus / lens controller 406 may prioritize testing focus lengths close to the current focus length to avoid changing focus as much as possible.

[0073]

[0086] As described further herein, the focus / lens controller 406 can intelligently select configuration adjustments (e.g., lens position and focal length, lens aperture, etc.) to reduce or minimize the amount of power consumed by the camera 420 and / or the lens motor / driver associated with the camera 420 when performing object detection and object detection determination. Moreover, the image processing system 400 can continuously perform one or more iterations of the object detection checks / tests and camera / lens adjustments described above until the object detector 408 determines with a threshold confidence that an object of interest is present or not. The image processing system 400 can intelligently perform each iteration in a manner that reduces the power consumed in determining whether an object of interest is present and, if an object of interest is present, detecting the object. In some cases, the image processing system 400 may also intelligently perform each iteration in a manner that increases the probability of detecting an object of interest, increases the probability of making a detection decision that satisfies a confidence threshold (e.g., object is detected, object is present, object is not present), and / or reduces the amount of time (and / or amount of attempts / iterations) it takes to make such a detection decision and / or complete the object detection process.

[0074]

[0087] FIG. 5 illustrates an example process 500 for low power variable focus object detection. In some examples, the process 500 can reduce or limit the amount of power consumed by a focus controller device, such as a focus motor. For example, the process 500 can reduce power consumption by reducing or minimizing the time spent by the lens focused away from a neutral position and / or the amount and / or magnitude of lens position changes implemented by the system. In some examples, the neutral position can be a position that consumes the least amount of power to reach / achieve and / or maintain. In some examples, the neutral position can be a non-powered position and / or a position associated with a lower power draw than a non-neutral or other position.

[0075]

[0088] At block 502, process 500 may begin with the camera system (e.g., front-facing camera 104, camera 420) in an off state (e.g., focus off, power off). At block 504, process 500 may determine whether a trigger event has occurred. The trigger event may indicate that an object of interest may be present in the scene. Moreover, the trigger event may be an event configured to trigger an object detector (e.g., object detector 408) to determine whether an object of interest is present in the scene. Non-limiting examples of objects of interest may include a QR code, a document, a face, a product, a hand, a finger, an animal, a sign, text, a symbol, an image, a landmark, a device, a person, a vehicle, and / or any other item(s) that may be captured and detected in an image.

[0076]

[0089] Additionally, trigger events may include, for example, but are not limited to, detection of inertial movement above a threshold, gestures that may indicate an attempt / intent to initiate object detection (e.g., gestures indicating an attempt to scan an item such as a QR code or document, gestures indicating an attempt to scan / detect a face, hand or finger gestures, etc.), detection / indication of the presence of an object of interest (e.g., inertial sensor based presence detection, audio sensor based presence detection, depth sensor based presence detection, etc.), audio change above a threshold, object specific triggers (e.g., ultrasonic signaling, automobile noise, animal noise, human voice / audio, etc.), light change above a threshold, change in range to an object above a threshold, change in PD based depth above a threshold, and / or any other predetermined trigger or combination thereof.

[0077]

[0090] Process 500 may analyze image data from the camera system to determine if a trigger event is detected. In some cases, process 500 may also analyze other sensor data (e.g., sensor data 430), as previously described. If a trigger event is not detected, process 500 may return to block 502 and maintain the camera system in an off state. If a trigger event is detected, in block 506, process 500 may turn on the camera system with the lens of the camera system in a neutral position.

[0078]

[0091] At block 508, the process 500 may determine whether an object of interest is present or absent based on an image captured by the camera system using the neutral position (e.g., while the lens is in the neutral position). In some examples, the presence determination is based on an object detection metric that indicates a confidence that the object of interest is present or absent in the image.

[0079]

[0092] At block 510, process 500 may determine whether the object detection metric is above or below one or more detection thresholds. In some examples, the one or more confidence thresholds may include an upper threshold corresponding to a certainty / likelihood threshold that the object of interest is present in the image and a lower threshold corresponding to a certainty / likelihood threshold that the object of interest is not present in the image.

[0080]

[0093] If the object detection metric is above or at an upper threshold, then at block 512, process 500 may determine that an object of interest has been detected. In some cases, if an object of interest is detected at block 512, process 500 may "wake up" or initialize a different image sensor and / or camera system with higher / additional capabilities (e.g., higher resolution, higher frame rate, higher power state, etc.). If the object detection metric is below or at a lower threshold, then process 500 may return to block 502. For example, process 500 may power off the camera system if the object detection metric indicates that an object of interest is not present in the image.

[0081]

[0094] If the object detection metric is above the lower threshold but below the upper threshold, in block 514, the process 500 may adjust the focus of the camera system by moving the lens to the next lowest power position (e.g., relative to the neutral power position). The next lowest power position may be a lens position different from the neutral position that requires the least amount of power to achieve relative to other available / possible lens positions. For example, the next lowest power position may be a lens position that is determined to involve the least amount of power draw from the focus motor / controller to achieve. For illustration, the power consumed by the focus motor / controller to move the lens to different positions may be different, with some positions involving more power consumption by the focus motor / controller than other positions. The lens position that requires the least amount of power consumption by the focus motor / controller to move the lens (excluding the neutral position already implemented in block 506) may be the next lowest power position referenced in block 514. Thus, in block 514, process 500 may include moving the lens to a particular lens position that involves a minimal amount of power consumption by the focus motor / controller (excluding the neutral position already implemented in block 506).

[0082]

[0095] At block 516, the process 500 may make an additional determination regarding whether the object of interest is present or absent based on another image captured by the camera system using the next lowest power position. In some examples, the additional presence determination is based on an additional object detection metric that indicates confidence that the object of interest is present or absent in the image.

[0083]

[0096] At block 518, process 500 may determine whether the additional object detection metric is above or below one or more thresholds. In some examples, if the additional object detection metric is above or at an upper threshold, then at block 512, process 500 may determine that an object of interest is detected. If the additional object detection metric is below or at a lower threshold, then process 500 may return to block 502. For example, process 500 may power off the camera system if the additional object detection metric indicates that an object of interest is not present in the image.

[0084]

[0097] If the additional object detection metric is above the lower threshold but below the upper threshold, process 500 may again adjust the focus of the camera system by moving the lens to the next lower power position (e.g., a lens position associated with more power consumption than the next lower power position from block 514, but less power consumption than any other available / possible lens position for that camera system). Process 500 may continuously perform the object presence check described above with incremental increases in power consumption until process 500 detects an object, determines that an object is not present, or finishes testing every available / possible lens position for that camera system. In this manner, process 500 may search for a focus and lens position that results in an image of sufficient quality to detect an object or determine that an object is not present, while reducing or minimizing the amount of power used to find such focus and lens position. For example, before implementing a focus and lens position with a higher focus motor / controller power consumption, process 500 may first check any focus and lens positions with a lower focus motor / controller power consumption, which may lead to a detection that uses a lower amount of power.

[0085]

[0098] FIG. 6 illustrates an exemplary timeline 600 of focus adjustments and associated power profiles. The power profile includes detection power 652, rock bottom sleep (RBS) power 650, sensor focus power 630-634, and image processing power 640. The power profile includes detection power 652, rock bottom sleep (RBS) power 650, sensor focus power 630-634, and image processing power 640, each of which is represented in FIG. 6 by a block, and the relative values ​​of the powers of detection power 652, RBS power 650, sensor focus power 630-634, and image processing power 640 are represented by the relative (exemplary) sizes of the blocks representing the power profile including detection power 652, rock bottom sleep (RBS) power 650, sensor focus power 630-634, and image processing power 640.

[0086]

[0099] In the example shown in FIG. 6, the camera system is in a disabled state 602 between time T0 and time T1. In the disabled state 602, the camera system consumes detection power 652 and RBS power 650. Detection power 652 can include a small amount of power consumed to process sensor data and detect trigger events as previously described. RBS power 650 (or lock bottom sleep current or RBSC) can represent a minimum power level of the camera system (e.g., the minimum amount of power used by the camera system to operate and / or when not completely powered off). RBS power 650 and detection power 652 involve a minimum amount of power consumption represented by the smaller, relative size of the blocks representing RBS power 650 and detection power 652 (relative to the blocks corresponding to sensor focus power and image processing power).

[0087]

[0100] At T1 (or prior to T1), the camera system may detect a trigger event 610, which may indicate that an object of interest may be present, and may initiate an object detection process, such as process 500 described above with respect to FIG. 5. Based on the trigger event 610, the camera system sets the lens to a neutral lens position 604. The neutral lens position 604 may be a non-powered lens position or a lens position associated with the lowest amount of focus motor / controller power from various available / possible lens positions in the camera system. In some examples, the neutral lens position 604 may correspond to an infinite focus (also referred to as a focus to infinity).

[0088]

[0101] The camera system maintains a neutral lens position 604 between time T1 and time T2. In the neutral lens position 604, the camera system consumes a minimum sensor focus power 630 (e.g., represented by a relative size of a block corresponding to the minimum sensor focus power 630) and image processing power 640, in addition to detection power 652 and RBS power 650. Based on an image captured by the camera system while in the neutral lens position 604, the camera system determines an object detection metric 612 that estimates whether an object of interest is present in the captured image. In some examples, the object detection metric 612 may include a confidence value indicating a certainty, likelihood, estimate, etc., regarding whether an object of interest is present in the captured image. The object detection metric 612 in this example is below an upper threshold and above a lower threshold (e.g., as indicated by the camera system not returning to the disabled state 602 after determining the object detection metric 612).

[0089]

[0102] Based on the object detection metric 612 being below the upper threshold and above the lower threshold, the camera system adjusts the lens position to the first lens position 606 to inspect / test whether an object can be detected within the upper threshold confidence when the lens is in the first lens position 606. The camera system maintains the first lens position 606 between time T2 and time T3. The first lens position 606 may include an image processing power 640 and a next lower sensor focus power 632 in addition to a detection power 652 and an RBS power 650. As shown, the next lower sensor focus power 632 associated with the first lens position 606 includes more power consumption than the lowest sensor focus power 630 associated with the neutral position 604, but less power consumption than a higher sensor focus power 634 associated with the second lens position 608. For example, the focus motor / controller of the camera system draws more power to change the lens of the camera system to the first lens position 606 than to the neutral lens position 604, but draws less power to change the lens of the camera system to the first lens position 606 than to the second lens position 608.

[0090]

[0103] Based on the images captured by the camera system at the first lens position 606, the camera system determines an object detection metric 614 (e.g., a second confidence value) that estimates whether an object of interest is present in the captured image. The object detection metric 614 in this example is again below the upper threshold and above the lower threshold (e.g., as indicated by the camera system not returning to the disabled state 602 after determining the object detection metric 614). Thus, based on the object detection metric 614 being below the upper threshold and above the lower threshold, the camera system again adjusts the lens position to the second lens position 608 to inspect / test whether an object can be detected within the upper threshold confidence when the lens position is at the second lens position 608. The camera system maintains the first lens position 606 between time T3 and time T4.

[0091]

[0104] The second lens position 608 can include image processing power 640 and higher sensor focus power 634 in addition to detection power 652 and RBS power 650. As shown, the higher sensor focus power 634 associated with the second lens position 608 includes more power consumption than the lowest sensor focus power 630 associated with the neutral position 604 and the next lowest sensor focus power 632 associated with the first lens position 606. For example, a focus motor / controller draws more power to change the lens of the camera system to the second lens position 608 than to change to the neutral lens position 604 and the first lens position 606.

[0092]

[0105] Based on the image captured by the camera system at the second lens position 608, the camera system determines an object detection metric 616 that estimates whether an object of interest is present in the captured image. In some cases, the object detection metric 616 in this example falls below an upper threshold and above a lower threshold (e.g., as indicated by the camera system not returning to the disabled state 602 after determining the object detection metric 614). Thus, based on the object detection metric 614 falling below the upper and lower thresholds, the camera system transitions to the disabled state 602. In other cases, if the second lens position 608 is the last lens position not inspected / tested by the camera system and the object detection metric 616 is still below the upper threshold, the camera system can transition to the disabled state 602 even if the object detection metric 616 is above the lower threshold.

[0093]

[0106] The camera system remains in the disabled state 602 between time T4 and time T5. While in the disabled state 602, the camera system detects a trigger event 618 and transitions to a neutral lens position 604. The camera system remains in the neutral lens position 604 between time T5 and time T6. The camera system captures an image based on the neutral lens position 604 and can use the captured image to determine an object detection metric 620 that an object is present or absent in the image. In this example, the object detection metric 620 is at or below a lower confidence threshold. Thus, the camera system determines that an object was not detected and / or is not present in the image.

[0094]

[0107] The camera system then transitions to a disabled state 602 and remains in the disabled state 602 between times T6 and T7. When the camera system detects a trigger event 622, the camera system changes to a neutral lens position 604. The camera system maintains the neutral lens position 604 between times T7 and T8. The camera system captures an image based on the neutral lens position 604 and determines an object detection metric 624 that an object is present or absent in the image. In this example, the object detection metric 624 is at or above an upper threshold confidence. Thus, based on the object detection metric 624, the camera system can detect an object and / or determine that an object is present in the image.

[0095]

[0108] In some cases, if the camera system detects an object (e.g., based on the object detection metrics 624) and / or determines that an object is present in an image, the electronic device implementing the camera system (e.g., the mobile device 102) may “wake up” or initialize a different camera system implemented by the electronic device. The different camera system may include a camera system with higher / additional image capture and / or image processing capabilities, such as higher resolution, higher frame rate(s), higher power state / mode(s), etc. In some examples, the different camera system may implement and / or invoke additional and / or more computationally intensive image processing tasks / algorithms. The different camera system may be implemented to capture one or more images of the detected object for further processing and / or one or more images of the scene and / or other objects in the scene. In some examples, the electronic device may use the different camera system to process images captured by the camera system at a focal length used to capture an image of the detected object, recognize the detected object (e.g., recognize a QR code, recognize a face, recognize a device, etc.), etc.

[0096]

[0109] In some cases, the electronic device may switch back to a camera system (e.g., from a different camera system) when the electronic device transitions to a different power state (e.g., a lower power state, a locked state, an inactive state, a different power mode, etc.), when an object of interest is not detected for a threshold time period, when the different camera system has not been used (and / or remains inactive) for a threshold time period, etc. For example, the electronic device may place the different camera system in an inactive, sleep, off, or lower power state to initialize the camera system for use until another switch to the different camera system is triggered.

[0097]

[0110] 6, the camera system may successively test different lens positions / focal lengths starting with the lens position / focal length with the least amount of power consumption, and (if necessary) successively inspect / test any additional lens positions / focal lengths selected in an order from lower to higher power consumption. As described further herein, in other examples, the camera system may select which lens positions / focal lengths to inspect based on a different selection order (e.g., instead of or in addition to an order / criterion from lower to higher power consumption), such as a reward vs. risk selection order, a selection order based on distance and / or power consumption for a current distance / lens position, and / or any other selection order and / or criteria.

[0098]

[0111] 7 illustrates another exemplary process 700 for low power variable focus object detection. In this example, the process 700 may prioritize certain focal lengths based on a risk and reward analysis / decision.

[0099]

[0112] For example, at block 702, process 700 may begin with a camera system (e.g., front facing camera 104, camera 420) in an off state (e.g., focus off, power off). At block 704, process 700 may determine whether a trigger event has occurred. If a trigger event is detected, at block 706, process 700 may turn on the camera system with the lens of the camera system in a selected position, where the position of the lens may be selected based on risk and reward decisions.

[0100]

[0113] For example, the lens position selected in block 706 may be the position with the highest / best reward vs. risk assessment or comparison. The reward associated with the position may include and / or correspond to an object detection metric that provides an indication and / or estimated likelihood / certainty that the object of interest will be detected with the lens at that position. The risk associated with the position may include and / or correspond to an object detection metric that provides an indication and / or estimated likelihood / certainty that the object of interest will not be detected with the lens at that position, and thus moving the lens to that position may risk unnecessary power consumption (e.g., power draw to move the lens to that position that does not result in object detection and / or threshold object detection certainty).

[0101]

[0114] In some cases, the risk / reward determination for various lens positions can be based on an estimated range / distance of the object of interest. The estimated range / distance can provide insight into the likelihood of achieving object detection (or a determination that the object of interest is not present) from various lens positions and / or into which lens position(s) may have a higher / highest likelihood of resulting in detection (or a determination that the object of interest is not present). In some cases, the likelihood of achieving object detection (or a determination that the object of interest is not present) estimated for a particular lens position, if that likelihood is high and / or higher than the likelihood estimated for other lens positions, can indicate, reflect, and / or correspond to a reward of using that particular lens position (and / or can be used to determine a reward value). In some cases, the likelihood of achieving object detection (or a determination that the object of interest is not present) estimated for a particular lens position, if that likelihood is low and / or lower than the likelihood estimated for other lens positions, can indicate, reflect, and / or correspond to a risk of using that particular lens position (and / or can be used to determine a risk value).

[0102]

[0115] Moreover, in some cases, the amount of power consumption used to move the lens to a particular lens position may indicate, reflect, and / or correspond to a risk of using that particular lens position if that amount of power consumption is higher than the amount of power consumption of high and / or other lens positions. In some cases, the amount of power consumption used to move the lens to a particular lens position may also indicate, reflect, and / or correspond to a reward of using that particular lens position if that amount of power consumption is lower than the amount of power consumption of low and / or other lens positions.

[0103]

[0116] In some implementations, the trigger event(s) (and / or information about the trigger event(s)), and / or sensor data used to detect the trigger event(s) (e.g., sensor data 430 and / or image data 432) may be used to determine a risk / reward value for a lens position. For example, the trigger event, information related to the trigger event, and / or sensor data used to detect the trigger event may provide an indication of the distance / range of an object of interest, which may then be used to determine which lens position(s) may have a higher / highest likelihood of resulting in object detection for an object at that distance / range, based on focal length and / or any other focus characteristics associated with various available / possible lens positions for the camera system.

[0104]

[0117] In some examples, the trigger event, information related to the trigger event, and / or sensor data used to detect the trigger event may be used as input to a risk / reward function in the focus / lens controller 406 to determine risk / reward values ​​for various lens positions. In some cases, the focus / lens controller 406 may prioritize a higher power lens position (e.g., focal length) if the risk / reward function provides a higher / higher indication / likelihood of a possible object of interest at the higher power lens position (e.g., focal length). For example, in some cases, the focus / lens controller 406 may prioritize a higher power lens position over a lower power lens position if the reward estimated for the higher power lens position is high (e.g., at or above a certain threshold) and / or higher than the reward estimated for one or more lower power lens positions, and / or if the risk estimated for the higher power lens position is low (e.g., at or below a certain threshold) and / or lower than the risk estimated for one or more lower power lens positions. By way of illustration, the focus / lens controller 406 may prioritize the second lens position 608 shown in FIG. 6 over the first lens position 606 and / or the neutral lens position 604 shown in FIG. 6 if the reward estimated for the second lens position 608 is high (e.g., at or above a certain threshold) and / or higher than the reward estimated for the neutral lens position 604 and / or the first lens position 606, and / or if the risk estimated for the second lens position 608 is low (e.g., at or below a certain threshold) and / or lower than the risk estimated for the neutral lens position 604 and / or the first lens position 606.

[0105]

[0118] At block 708, process 700 may determine an object detection metric (e.g., a confidence value) for the selected lens position based on an image captured by the camera system using the selected lens position (e.g., the lens position selected at block 706). The object detection metric may include a value indicating a confidence that an object of interest is present or absent in the image.

[0106]

[0119] At block 710, process 700 may determine whether the detection metric (e.g., a confidence value) is above or below one or more confidence thresholds. In some examples, the one or more confidence thresholds may include an upper threshold corresponding to a threshold certainty / likelihood that the object of interest is present in the image and a lower threshold corresponding to a threshold certainty / likelihood that the object of interest is not present in the image.

[0107]

[0120] If the detection metric is above or at the upper threshold, then in block 712, process 700 may determine that an object of interest has been detected. In some examples, if process 700 determines that an object of interest has been detected, process 700 may "wake up" or initialize a different camera system. The different camera system may include, for example, a higher power / resolution camera system. In some examples, the different camera system may capture higher resolution images, implement a higher frame rate, implement or invoke additional and / or more computationally intensive image processing tasks / algorithms, implement camera hardware with higher power and / or capabilities, etc. Process 700 may use the different system to capture one or more images of the detected object (and / or other object(s) or part of the scene) for further processing. In some examples, process 700 may use the different camera system to process images captured by the camera system (and / or by the different camera system) at the lens position used to capture an image of the detected object, recognize the detected object (e.g., recognize a QR code, recognize a face, recognize a device, etc.), etc.

[0108]

[0121] In some cases, process 700 may switch back to a camera system (e.g., from a different camera system) when an electronic device implementing the camera system (e.g., mobile device 102) transitions to a different power state (e.g., a lower power state, a locked state, an inactive state, etc.), when an object of interest is not detected for a threshold time period, when the different camera system has not been used (and / or remains inactive) for a threshold time period, etc. For example, process 700 may place the different camera system in an inactive, sleep, off, or lower power state to initialize the camera system for use until another switch to the different camera system is triggered.

[0109]

[0122] If the detection metric is below or at the lower threshold, the process 700 may return to block 702. For example, the process 700 may power off the camera system if the detection metric indicates that the object of interest is not present in the image. If the detection metric is above the lower threshold but below the upper threshold, in block 714, the process 700 may adjust the focus of the camera system by moving the lens to a different selected lens position. The different selected lens position may be similarly selected based on the risk / reward values ​​estimated for the different selected lens position and any other lens positions. In some examples, the different selected lens position may be the lens position with the next best risk / reward value (e.g., having the highest reward-to-risk ratio or evaluation / comparison after the lens position selected in block 706).

[0110]

[0123] At block 716, process 700 may determine an additional detection metric based on another image captured by the camera system using a different selected lens position. At block 718, process 700 may determine whether the additional detection metric is above or below one or more confidence thresholds. In some examples, if the additional detection metric is above or at an upper threshold, process 700 may determine that an object of interest has been detected at block 712. If the additional detection metric is below or at a lower threshold, process 700 may return to block 702 and process 700 may power off the camera system.

[0111]

[0124] If the additional detection metric is above the lower threshold but below the upper threshold, process 700 may again adjust the focus of the camera system by moving the lens to an additional lens position selected based on the risk / reward decision, as previously described. Process 700 may continuously perform the detection confidence checks described above for the lens positions selected based on the risk / reward values, such as risk / reward ratios or risk / reward evaluations / comparisons, until process 700 detects the object, determines that the object is not present, or finishes testing every available / possible lens position for that camera system. In this manner, process 700 may search for a focus and lens position that results in an image of sufficient quality to detect the object or determine that the object is not present, while reducing or minimizing the amount of power used to find such focus and lens position and / or increasing or maximizing the potential reward to risk ratio or evaluation.

[0112]

[0125] 8 is a diagram illustrating an example use of PDAF data to estimate / detect a distance of an object of interest (one or more) within the FOV of a camera system. In some examples, the camera system can use the PDAF data (and / or the distance of an object of interest estimated / detected based on the PDAF data) to detect a trigger event. In some cases, the camera system can use the PDAF data (and / or the distance of an object of interest estimated / detected based on the PDAF data) to determine a risk / reward value for selecting a lens position, as previously described with respect to the process 700 shown in FIG. 7. In some cases, the camera system can use the PDAF data (and / or the distance of an object of interest estimated / detected based on the PDAF data) as part of ongoing autofocus correction.

[0113]

[0126] 8, pixel array 802 includes focus pixels, generally referred to as 4PD focus pixels or four-phase detection (QPD) focus pixels, with each focus pixel including four diodes, as previously described with respect to FIG. 3D. For example, a 4PD focus pixel in pixel array 802 may include a top left photodiode, a top right photodiode, a bottom left photodiode, and a bottom right photodiode. Data from each photodiode of the 4PD focus pixel may be compared to data from adjacent photodiodes of the 4PD focus pixel to determine a phase difference.

[0114]

[0127] The camera system can use partial PDAF data from pixel array 802 as a trigger event to detect the distance of an object within the FOV of the image sensor. The image sensor can sample PDAF data on some lines while skipping most of the lines. The image can have high density in the horizontal direction to enable accurate PDAF processing. In some examples, PD data can provide reliable results even on a single PDAF line. In some cases, a QPD sensor (e.g., pixel array 802) can bin data from two lines to generate L / R interleaved PDAF data. In some examples, binning more lines can enable working at lower light levels.

[0115]

[0128] To illustrate, a camera system may use partial PDAF data from a read out 804 of a pixel array 802 to detect the distance of an object within the FOV of an image sensor. The image sensor may sample PDAF data on a portion 810 of a line while skipping a larger portion 812 of the line. In some examples, a QPD sensor (e.g., pixel array 802) may bin data from two lines to generate L / R interleaved PDAF data 814. In some cases, the portion 810 of a line may be sampled from an edge of the pixel array 802 that is added to the bin, such as a top, bottom, or side edge.

[0116]

[0129] In some cases, the camera system can adjust the lens aperture to aid in object detection as described herein. For example, some camera modules can have an adjustable aperture. In such cases, the camera system can adjust the aperture to achieve different depths of field, which can aid in detecting objects. For illustration, FIG. 9 shows images 900, 902, and 904 captured using different apertures. Here, image 900 was captured using a first aperture 910, image 902 was captured using a second aperture 912 that is smaller than the first aperture 910, and image 904 was captured using a third aperture 914 that is smaller than the second aperture 912.

[0117]

[0130] As shown, different apertures 910, 912, and 914 result in different depths of field in images 900, 902, and 904, with the smallest aperture (e.g., the third aperture 914) resulting in the deepest depth of field and the largest aperture (e.g., the first aperture 910) resulting in the shallowest depth of field.

[0118]

[0131] Thus, the camera system can use different apertures (e.g., first aperture 910, second aperture 912, and third aperture 914) to exploit the different associated depths of field to aid in object detection. For example, the camera system can use a smaller aperture (e.g., third aperture 914 or second aperture 912) to achieve a deeper depth of field. The deeper depth of field can allow the camera system to reduce the number of focal lengths to inspect for an object of interest (or inspect only one focal length).

[0119]

[0132] In some cases, the aperture selected may depend on the light level in the scene / environment. For example, in a low light scenario, a larger aperture (e.g., first aperture 910) may result in a shallower depth of field despite such a larger aperture, potentially requiring more focal length stops / checks.

[0120]

[0133] In some cases, instead of or in addition to sequentially selecting a particular focal length to use / test, an object detection process implemented by a camera system with multiple image sensors can sequentially select a particular image sensor with the lowest power profile at a given focal length. Electronic devices often have multiple cameras, and each of the cameras can have its own unique characteristics, such as, for example, FOV, focus power profile, aperture, etc. Thus, the object detection process can use multiple cameras. For example, the object detection process can select a particular camera for a given focal length because one image sensor may have a lower focus power for a particular focal length and another image sensor may have a lower focus power for a different focal length. In some examples, the focus / lens controller 406 can select from a set of cameras the camera with the lowest power (e.g., with the appropriate settings) for the focal length or the camera with the best risk / reward value for the focal length.

[0121]

[0134] 10 is a flow chart illustrating an example process 1000 for object detection using multiple cameras. In this example, at block 1002, the process 1000 may begin with the camera system (e.g., front-facing camera 104, camera 420) in an off state (e.g., focus off, power off). At block 1004, the process 1000 may determine whether a trigger event has occurred.

[0122]

[0135] If a trigger event is not detected, process 1000 may return to block 1002 and keep the camera system in an off state. If a trigger event is detected, in block 1006, process 1000 may turn on the camera system and select the lowest power camera for a given focal length.

[0123]

[0136] In block 1008, the process 1000 may determine an object detection confidence based on the images captured by the camera system using the selected lowest power camera.

[0124]

[0137] At block 1010, process 1000 may determine whether the detection metric (e.g., a confidence value) is above or below one or more confidence thresholds. In some examples, the one or more confidence thresholds may include an upper threshold corresponding to a threshold certainty / likelihood that the object of interest is present in the image and a lower threshold corresponding to a threshold certainty / likelihood that the object of interest is not present in the image.

[0125]

[0138] If the detection metric is above or at the upper threshold, then in block 1012, the process 1000 may determine that the object of interest has been detected. If the detection reliability is below or at the lower threshold, then the process 1000 may return to block 1002. For example, the process 1000 may power off the camera system if the detection reliability indicates that the object of interest is not present in the image. If the detection reliability is above the lower threshold but below the upper threshold, then in block 1014, the process 1000 may select the next lowest power camera for the given focal length.

[0126]

[0139] At block 1016, the process 1000 may determine an additional detection confidence based on another image captured by the camera system using the next lower power camera. The object detection confidence may include a value indicating the confidence that the object of interest is present or absent in the image.

[0127]

[0140] At block 1018, the process 1000 may determine whether the additional detection metric is above or below one or more confidence thresholds. In some examples, if the additional detection metric is above or at an upper threshold, the process 1000 may determine that an object of interest has been detected at block 1012. If the additional detection metric is below or at a lower threshold, the process 1000 may return to block 1002 and power off the camera system.

[0128]

[0141] If the additional detection metric is above the lower threshold but below the upper threshold, the process 1000 may again select a subsequently lower power camera for the focal length (e.g., if another camera is available). The process 1000 may perform the detection reliability check described above successively for different cameras with different power profiles for a given focal length. The process 1000 may select a particular camera in each iteration based on an order from lower power profile to higher power profile. In another example, the process 1000 may select a particular camera in each iteration based on an order based on the risk / reward values ​​calculated for the different cameras, with cameras with better risk / reward being tested before other cameras with worse risk / reward.

[0129]

[0142] 11 is a flow chart illustrating an example process 1100 for low power variable focus object detection. At block 1102, the process 1100 can include obtaining a first image of a scene captured by an image capture device (e.g., forward-facing camera 104, camera 420) with a lens in a first configuration of a plurality of available lens configurations based on a trigger event.

[0130]

[0143] In some examples, the trigger event may include inertial movement above a threshold, an audio change above a threshold, an ambient light change above a threshold, a change in range to an object above a threshold, a trigger from an application associated with the method, a depth measurement from an active depth sensing system, a trigger from a global navigation satellite system, a trigger from a global positioning system, a data connection, and a phase detection change above a threshold, and / or a phase detection change above a threshold. In some cases, the trigger event may be detected based on data from one or more sensors (e.g., sensor data 430, image data 432). In some examples, the one or more sensors may include an image sensor (e.g., camera 420), an audio sensor, a gyroscope, an accelerometer, an inertial measurement unit, an ambient light sensor, and / or a depth sensor.

[0131]

[0144] At block 1104, the process 1100 may include determining, based on the first image of the scene and the first detection result, whether an object of interest is present in the first image. In some examples, the object of interest may include a document, a Quick Response (QR) code, a face, a finger, a hand, a device, a product, and / or an animal.

[0132]

[0145] At block 1106, the process 1100 may include adjusting the lens to a second configuration selected from a plurality of available lens configurations in response to determining that the object of interest is not present in the first image. In some cases, the determination that the object of interest is not present in the first image may be based on a confidence value (e.g., an object detection metric) in the first detection result being below a threshold. In some examples, the threshold may include an upper confidence threshold. In some cases, the confidence value (e.g., an object detection metric) used to determine that the object of interest is not present in the first image may be below (or at) an upper confidence threshold and above (or at) a lower confidence threshold.

[0133]

[0146] In some examples, the second lens configuration may be selected from the multiple available lens configurations based on an amount of power required by the image capture device to adjust the lens to the second lens configuration. In some cases, the amount of power may be for one or more different amounts of power required by the image capture device to adjust the lens to one or more other lens configurations from the multiple available lens configurations.

[0134]

[0147] In some examples, the first configuration can include a first lens position and the second configuration can include a second lens position different from the first lens position. In some cases, adjusting the lens can include moving the lens from the first lens position to the second lens position using a focus motor. The focus motor can include, for example, but is not limited to, a voice coil motor (VCM), a piezoelectric motor, a stepper motor, an ultrasonic motor, an electroactive polymer motor, an electromagnetic focus motor, a geared direct current (DC) motor, a direct drive supersonic wave motor, a solid lens controller, among others.

[0135]

[0148] In some examples, the multiple available lens configurations can include multiple available lens positions, and the second lens location can be selected from the multiple available lens positions based on an amount of power required by the focus motor to move the lens from the first lens position to the second lens position relative to one or more different amounts of power required by the focus motor to move the lens to one or more other positions from the multiple available lens positions. In some cases, the amount of power required by the focus motor to move the lens from the first lens position to the second lens position can include a next lower amount of power relative to an amount of power associated with the first lens position, and / or one or more amounts of power required by the focus motor to move the lens to the one or more other positions from the multiple available lens positions.

[0136]

[0149] In some examples, the second lens location may be selected from the multiple available lens positions based on a priority of focal lengths associated with the multiple available lens positions. In some examples, the priority of focal lengths associated with the multiple available lens positions may be based on a respective likelihood of detecting an object of interest in an image captured from each of the multiple available lens positions and an amount of power required by a focus motor to move the lens to each of the multiple available lens positions. In some examples, the priority of focal lengths associated with the multiple available lens positions may be based on a default type of object of interest or a type of detected object.

[0137]

[0150] In some cases, the second lens location may be selected from the plurality of available lens positions based on a relative distance between the first position and each of the plurality of available lens positions. In some examples, the second position is selected from the plurality of available lens positions based on a shortest relative distance between the first position and each of the plurality of available lens positions.

[0138]

[0151] In some cases, the second position may be selected from a plurality of available lens positions based on one or more characteristics of the lens. In some examples, the one or more characteristics of the lens may include a current aperture associated with the lens, a field of view associated with the lens, and / or a focus power profile associated with the lens.

[0139]

[0152] At block 1108, process 1100 may include acquiring, by the image capture device, a second image of the scene while the lens is in a second configuration. The first and second configurations may include, for example, but not limited to, different lens positions, different apertures, different depths of field, different power profiles, different focal lengths, etc.

[0140]

[0153] At block 1110, the process 1100 may include determining that the object of interest is present in the second image based on the second image of the scene and the second detection result. In some cases, the first detection result and the second detection result may include a confidence value (e.g., an object detection metric). In some examples, the process 1100 may determine that the confidence value in the second detection result is above (or at) a threshold (or upper threshold) and determine that the object is detected (e.g., present in the second image) based on the confidence value being above (or at) the threshold (or upper threshold).

[0141]

[0154] In some aspects, the process 1100 may include selecting a lens from the plurality of available lenses based on one or more characteristics of the lens and a focal length associated with the object of interest. In some examples, the one or more characteristics may include an aperture, a field of view, and / or a focus power profile. In some cases, the lens may be selected from the plurality of available lenses based on a determination that a focus power profile associated with the lens includes a lower focus power than a respective focus power profile of one or more (or all) lenses from the plurality of available lenses.

[0142]

[0155] In some cases, the second lens position may be selected from a plurality of available lens positions based on a confidence value associated with the first detection result (e.g., an object detection metric from the first detection result).

[0143]

[0156] In some aspects, selecting the second lens position may include excluding one or more positions as unavailable lens positions for selection of the second lens position from the plurality of available lens positions based on a first reliability value associated with the first detection result, to reduce a number of the plurality of available lens positions.

[0144]

[0157] In some examples, selecting the second lens position may include comparing an amount of power required by the focus motor to move the lens from the first lens position to the second lens position to one or more amounts of power required by the focus motor to move the lens from the first position to one or more other positions from the plurality of available lens positions.

[0145]

[0158] In some cases, the second lens position is selected from a plurality of available lens positions based on a confidence value (e.g., an object detection metric) associated with the lens displacement from the first lens position to the second lens position.

[0146]

[0159] In some aspects, the process 1100 may include adjusting a different image capture device associated with the apparatus in response to determining that the object of interest is present in the second image. In some examples, adjusting the different image capture device may include turning on the different image capture device and / or initializing the different image capture device. In some cases, the different image capture device may include a main camera device and / or a higher power camera device than the image capture device. In some aspects, the process 1100 may include processing one or more images of the scene via the different image capture device. In some cases, the one or more images may include a second image and / or a third image captured by a different image capture device.

[0147]

[0160] In some embodiments, the process 100 can include maintaining the lens in the second configuration based on the second detection result.

[0148]

[0161] In some cases, the first configuration can include a non-active optical image stabilization mode and the second configuration can include an active optical image stabilization mode. In some examples, adjusting the lens to the second configuration can include activating the optical image stabilization mode using a lens stabilization motor.

[0149]

[0162] In some examples, the first configuration can include a first aperture setting and the second configuration can include a second aperture setting that is different from the first aperture setting. In some cases, adjusting the lens to the second configuration can include changing the aperture of the lens from the first aperture setting to the second aperture setting using an aperture motor.

[0150]

[0163] In some examples, the processes described herein (e.g., processes 500, 700, 1000, 1100, and / or other processes described herein) may be performed by a computing device or apparatus. In one example, process 1100 may be performed by a computing device (e.g., mobile device 102 in FIG. 1) having the computing architecture of computing system 1200 shown in FIG. 12. The computing device may also include image processing system 400 shown in FIG. 4, which may implement various components described above with respect to FIG. 4.

[0151]

[0164] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device with resource capabilities to perform the processes described herein, including process 1100. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) configured to perform steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0152]

[0165] Components of a computing device may be implemented in circuits. For example, components may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry) and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0153]

[0166] Processes 500, 700, 1000, and 1100 are illustrated as logic flow diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0154]

[0167] Processes 500, 700, 1000, 1100, and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored in a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0155]

[0168] 12 is a diagram illustrating an example of a system for implementing some aspects of the present technology. In particular, FIG. 12 illustrates an example of a computing system 1200, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device that constitutes any of the components of the system, in which the components communicate with each other using a connection 1205. The connection 1205 may be a physical connection to the processor 1210 using a bus, or a direct connection to the processor 1210, such as in a chipset architecture. The connection 1205 may also be a virtual connection, a networked connection, or a logical connection.

[0156]

[0169] In some embodiments, computing system 1200 is a distributed system in which the functionality described in this disclosure may be distributed within a data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the system components described represent many such components, each of which performs some or all of the functionality described for it. In some embodiments, a component may be a physical or virtual device.

[0157]

[0170] The exemplary system 1200 includes at least one processing unit (CPU or processor) 1210 and a connection 1205 that couples various system components, including system memory 1215, such as read only memory (ROM) 1220 and random access memory (RAM) 1225, to the processor 1210. The computing system 1200 may include a cache 1212 of high-speed memory directly connected to the processor 1210, in close proximity to the processor 1210, or integrated as part of the processor 1210.

[0158]

[0171] The processor 1210 may include any general-purpose processor, as well as hardware or software services, such as services 1232, 1234, 1236 stored in a storage device 1230, configured to control the processor 1210, as well as special-purpose processors where software instructions are incorporated into the actual processor design. The processor 1210 may essentially be a fully self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0159]

[0172] To enable user interaction, computing system 1200 includes input device(s) 1245, which may represent any number of input mechanisms, such as a microphone for audio, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 1200 may also include output device(s) 1235, which may be one or more of several output mechanisms. In some instances, a multimodal system may allow a user to provide multiple types of input and output to communicate with computing system 1200. Computing system 1200 may include a communication interface 1240, which may generally govern and manage user input and system output.The communications interface may be an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet® port / plug, a fiber optic port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, a wireless local area network (WLAN) signal transmission, The communication interface 1240 may implement or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing radio signal forwarding, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX®), infrared (IR) communications wireless signal forwarding, public switched telephone network (PSTN) signal forwarding, integrated services digital network (ISDN) signal forwarding, 3G / 4G / 5G / LTE® cellular data network wireless signal forwarding, ad-hoc network signal forwarding, radio signal forwarding, microwave signal forwarding, infrared signal forwarding, visible light signal forwarding, ultraviolet light signal forwarding, wireless signal forwarding along the electromagnetic spectrum, or any combination thereof. The communication interface 1240 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine a location of the computing system 1200 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based Beidou Navigation Satellite System (BDS), and the European-based Galileo GNSS.There is no restriction to operating on any particular hardware configuration, and therefore the basic features herein can be easily substituted for improved hardware or firmware configurations as they are developed.

[0160]

[0173] The storage device 1230 may be a non-volatile and / or non-transitory and / or computer readable memory device, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disk read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disk (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory The memory may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a Stick card, a smart card chip, an EMV chip, a Subscriber Identity Module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0161]

[0174] Storage devices 1230 may include software services, servers, services, etc., which, when code defining such software is executed by processor 1210, cause the system to perform a function. In some embodiments, hardware services performing a particular function may include software components stored in a computer-readable medium in relation to the necessary hardware components, such as processor 1210, connections 1205, output devices 1235, etc., to perform that function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data. Computer-readable media may include non-transitory media on which data may be stored, which does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memories, memories, or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0162]

[0175] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0163]

[0176] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will appreciate that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the technology may be presented as including individual functional blocks comprising devices, device components, steps or routines in a method implemented in software, or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0164]

[0177] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process is terminated when an operation of a process is completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0165]

[0178] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, and the like.

[0166]

[0179] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Common examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executing in a single device, as further examples.

[0167]

[0180] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0168]

[0181] In the above description, aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concept may be embodied and employed in various ways, and the appended claims are to be construed to include such variations, except as limited by the prior art. The various features and aspects of the applications described above may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be considered illustrative rather than restrictive. For purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0169]

[0182] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0170]

[0183] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0171]

[0184] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component, and / or any component that is in communication, either directly or indirectly, with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communications interface).

[0172]

[0185] Claim language or other language in this disclosure reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can further include items not recited in the set of A and B.

[0173]

[0186] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0174]

[0187] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized, at least in part, by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0175]

[0188] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0176]

[0189] Illustrative examples of the present disclosure include the following:

[0177]

[0190] Aspect 1. An apparatus comprising a memory configured to store data and one or more processors coupled to the memory, the one or more processors configured to: obtain a first image of a scene captured by an image capture device based on a trigger event; the first image is captured with a lens of the image capture device in a first configuration among a plurality of available lens configurations; determine whether an object of interest is present in the first image based on the first image of the scene and a first detection result; in response to determining that the object of interest is not present in the first image, adjust the lens to a second configuration selected from the plurality of available lens configurations; obtain a second image of the scene by the image capture device while the lens is in the second configuration; and determine that the object of interest is present in the second image based on the second image of the scene and the second detection result.

[0178]

[0191] Aspect 2. The apparatus of aspect 1, wherein the first detection result and the second detection result are reliability values.

[0179]

[0192] Embodiment 3. The apparatus of any of embodiments 1 to 2, wherein the one or more processors are configured to select a second lens configuration from the plurality of available lens configurations based on an amount of power required by the image capture device to adjust the lens to the second lens configuration relative to one or more different amounts of power required by the image capture device to adjust the lens to one or more other lens configurations from the plurality of available lens configurations.

[0180]

[0193] Aspect 4. The apparatus of any of Aspects 1 to 3, wherein the first configuration comprises a first lens position, and wherein the second configuration comprises a second lens position different from the first lens position, and wherein to adjust the lens to the second configuration, the one or more processors are configured to move the lens from the first lens position to the second lens position using a focus motor.

[0181]

[0194] Aspect 5. The apparatus of aspect 4, wherein the plurality of available lens configurations comprises a plurality of available lens positions, and wherein the second lens position is selected from the plurality of available lens positions based on a reliability value associated with the first detection result.

[0182]

[0195] Aspect 6. The apparatus of Aspect 5, wherein, to select the second lens position, the one or more processors are configured to reduce the number of the plurality of available lens positions by excluding one or more positions as unavailable lens positions for selection of the second lens position from the plurality of available lens positions based on the first reliability value.

[0183]

[0196] Example 7. The apparatus of any of Examples 4 to 6, wherein the multiple available lens configurations comprise multiple available lens positions, and wherein the second lens position is selected from the multiple available lens positions based on an amount of power required by the focus motor to move the lens from the first lens position to the second lens position.

[0184]

[0197] Aspect 8. The apparatus of Aspect 7, wherein to select the second lens position, the one or more processors are configured to compare the amount of power to one or more amounts of power required by the focus motor to move the lens from the first position to one or more other positions from a plurality of available lens positions.

[0185]

[0198] Aspect 9. The apparatus of any of aspects 4 to 8, wherein the plurality of available lens configurations comprises a plurality of available lens positions, and wherein the second lens position is selected from the plurality of available lens positions based on a reliability value associated with lens displacement from the first lens position to the second lens position.

[0186]

[0199] Aspect 10. The apparatus of any of aspects 4 to 9, wherein the multiple available lens configurations comprise multiple available lens positions, and wherein the second lens position is selected from the multiple available lens positions based on a focal length priority associated with the multiple available lens positions.

[0187]

[0200] Aspect 11. The apparatus of aspect 10, wherein a priority of focal lengths associated with a plurality of available lens positions is based on a default type of object of interest or a type of object detected in the first image.

[0188]

[0201] Example 12. An apparatus as described in any of Examples 4 to 11, wherein the plurality of available lens configurations comprises a plurality of available lens positions, and wherein the second lens position is selected from the plurality of available lens positions based on a relative distance between the first position and each of the plurality of available lens positions.

[0189]

[0202] Aspect 13. An apparatus as described in any of aspects 4 to 12, wherein the plurality of available lens configurations comprises a plurality of available lens positions, and wherein the second position is selected from the plurality of available lens positions based on one or more characteristics of the lens, the one or more characteristics of the lens comprising at least one of an aperture associated with the lens, a field of view associated with the lens, and a focus power profile associated with the lens.

[0190]

[0203] Embodiment 14. An apparatus described in any of embodiments 4 to 13, wherein one or more processors are configured to select a lens from a plurality of available lenses based on one or more characteristics of the lens and a focal length associated with the object of interest.

[0191]

[0204] Embodiment 15. The apparatus of embodiment 14, wherein the one or more characteristics comprise at least one of an aperture, a field of view, and a focus power profile.

[0192]

[0205] Aspect 16. The apparatus of aspect 14, wherein the lens is selected from the plurality of available lenses based on a determination that a focus power profile associated with the lens comprises a lower focus power than a respective focus power profile of one or more lenses from the plurality of available lenses.

[0193]

[0206] Aspect 17. The apparatus of any of aspects 1 to 16, wherein the object of interest comprises at least one of a document, a quick response code, a face, a finger, a hand, a device, a product, and an animal.

[0194]

[0207] Aspect 18. The device of any of aspects 1 to 17, wherein the trigger events comprise inertial movement above a threshold, an audio change above a threshold, a change in ambient light above a threshold, a change in range to an object above a threshold, a trigger from an application associated with the device, a depth measurement from an active depth sensing system, a trigger from a global navigation satellite system, a trigger from a global positioning system, a data connection, and a phase detection change above a threshold.

[0195]

[0208] Aspect 19. The apparatus of any of aspects 1 to 18, wherein the one or more processors are configured to adjust a different image capture device associated with the apparatus in response to determining that the object of interest is present in the second image, where adjusting the different image capture device comprises at least one of turning on the different image capture device and initializing the different image capture device, where the different image capture device comprises at least one of a main camera device and a camera device of a higher power than the image capture device.

[0196]

[0209] Aspect 20. The apparatus of aspect 19, wherein the one or more processors are configured to process one or more images of a scene via different image capture devices, the one or more images comprising at least one of a second image and a third image captured by the different image capture devices.

[0197]

[0210] Embodiment 21. An apparatus described in any of embodiments 1 to 20, wherein the one or more processors are configured to maintain the lens in the second configuration based on the second detection result.

[0198]

[0211] Aspect 22. A device described in any of aspects 1 to 21, wherein the first configuration has an inactive optical image stabilization mode and the second configuration has an active optical image stabilization mode, and wherein to adjust the lens to the second configuration, the one or more processors are configured to activate the optical image stabilization mode using a lens stabilization motor.

[0199]

[0212] Aspect 23. An apparatus as described in any of aspects 1 to 22, wherein the first configuration has a first aperture setting, and the second configuration has a second aperture setting that is different from the first aperture setting, and wherein to adjust the lens to the second configuration, the one or more processors are configured to change the aperture of the lens from the first aperture setting to the second aperture setting using an aperture motor.

[0200]

[0213] Example 24. An apparatus as described in any of examples 1 to 23, wherein the apparatus comprises a mobile device.

[0201]

[0214] Aspect 25. An apparatus as described in any of aspects 1 to 24, wherein the apparatus comprises an image capture device, wherein the image capture device comprises a camera.

[0202]

[0215] Embodiment 26. An apparatus according to any of embodiments 1 to 25, wherein the conditions for successful detection include a predefined or user-configured threshold.

[0203]

[0216] Aspect 27. The apparatus of any of aspects 1 to 26, wherein at least one of the first detection result and the second detection result includes at least one of a probability score or metric, a probabilistic prediction, a classification, a label, a cost or loss function value, an estimate, a ranking, and a binary prediction.

[0204]

[0217] Aspect 28. An apparatus as described in any of aspects 1 to 27, wherein the first configuration comprises a first lens position, wherein the second configuration comprises a second lens position different from the first lens position, wherein the plurality of available lens configurations comprises a plurality of available lens positions, wherein the second lens position is selected from the plurality of available lens positions based on a relative distance between the first position and each of the plurality of available lens positions, and wherein the second position is selected from the plurality of available lens positions based on a shortest relative distance between the first lens position and each of the plurality of available lens positions.

[0205]

[0218] Aspect 29. A method comprising: obtaining a first image of a scene captured by an image capture device based on a trigger event; the first image is captured with a lens of the image capture device in a first configuration among a plurality of available lens configurations; determining whether an object of interest is present in the first image based on the first image of the scene and a first detection result; in response to determining that the object of interest is not present in the first image, adjusting the lens to a second configuration selected from the plurality of available lens configurations; obtaining a second image of the scene by the image capture device while the lens is in the second configuration; and determining that the object of interest is present in the second image based on the second image of the scene and the second detection result.

[0206]

[0219] Aspect 30. The method of aspect 29, wherein the first detection result and the second detection result are reliability values.

[0207]

[0220] Aspect 31. The method of any of aspects 29 to 30, further comprising selecting a second lens configuration from the plurality of available lens configurations based on an amount of power required by the image capture device to adjust the lens to the second lens configuration relative to one or more different amounts of power required by the image capture device to adjust the lens to one or more other lens configurations from the plurality of available lens configurations.

[0208]

[0221] Aspect 32. The method of any of aspects 29 to 31, wherein the first configuration comprises a first lens position, and wherein the second configuration comprises a second lens position different from the first lens position, and wherein adjusting the lens to the second configuration further comprises moving the lens from the first lens position to the second lens position using a focus motor.

[0209]

[0222] Aspect 33. The method of aspect 32, wherein the plurality of available lens configurations comprises a plurality of available lens positions, and wherein the second lens position is selected from the plurality of available lens positions based on a reliability value associated with the first detection result.

[0210]

[0223] Aspect 34. The method of aspect 33, wherein selecting the second lens position further comprises excluding one or more positions as unavailable lens positions for selection of the second lens position from the plurality of available lens positions based on the first reliability value to reduce a number of the plurality of available lens positions.

[0211]

[0224] Aspect 35. The method of any of aspects 32 to 34, wherein the multiple available lens configurations comprise multiple available lens positions, and wherein the second lens position is selected from the multiple available lens positions based on an amount of power required by the focus motor to move the lens from the first lens position to the second lens position.

[0212]

[0225] Aspect 36. The method of aspect 35, wherein selecting the second lens position further comprises comparing the amount of power to one or more amounts of power required by the focus motor to move the lens from the first position to one or more other positions from a plurality of available lens positions.

[0213]

[0226] Aspect 37. The method of any of aspects 32 to 36, wherein the plurality of available lens configurations comprises a plurality of available lens positions, and wherein the second lens position is selected from the plurality of available lens positions based on a reliability value associated with a lens displacement from the first lens position to the second lens position.

[0214]

[0227] Aspect 38. The method of any of aspects 32 to 37, wherein the plurality of available lens configurations comprises a plurality of available lens positions, and wherein the second lens position is selected from the plurality of available lens positions based on a focal length priority associated with the plurality of available lens positions.

[0215]

[0228] Aspect 39. The method of aspect 38, wherein a priority of focal lengths associated with a plurality of available lens positions is based on a default type of object of interest or a type of object detected in the first image.

[0216]

[0229] Aspect 40. The method of any of aspects 32 to 39, wherein the plurality of available lens configurations comprises a plurality of available lens positions, and wherein the second lens position is selected from the plurality of available lens positions based on a relative distance between the first position and each of the plurality of available lens positions.

[0217]

[0230] Aspect 41. The method of any of aspects 32 to 40, wherein the plurality of available lens configurations comprises a plurality of available lens positions, and wherein the second position is selected from the plurality of available lens positions based on one or more characteristics of the lens, the one or more characteristics of the lens comprising at least one of an aperture associated with the lens, a field of view associated with the lens, and a focus power profile associated with the lens.

[0218]

[0231] Aspect 42. The method of any of aspects 29 to 41, further comprising selecting a lens from a plurality of available lenses based on one or more characteristics of the lens and a focal length associated with the object of interest, wherein the one or more characteristics comprise at least one of an aperture, a field of view, and a focus power profile.

[0219]

[0232] Aspect 43. The method of aspect 42, wherein the lens is selected from the plurality of available lenses based on a determination that a focus power profile associated with the lens comprises a lower focus power than a respective focus power profile of one or more lenses from the plurality of available lenses.

[0220]

[0233] Aspect 44. The method of any of aspects 29 to 43, wherein the trigger event comprises inertial movement above a threshold, an audio change above a threshold, an ambient light change above a threshold, a change in range to an object above a threshold, a trigger from an application associated with the method, a depth measurement from an active depth sensing system, a trigger from a global navigation satellite system, a trigger from a global positioning system, a data connection, and a phase detection change above a threshold.

[0221]

[0234] Aspect 45. The method of any of aspects 29 to 44, further comprising adjusting a different image capture device associated with the method in response to determining that the object of interest is present in the second image, wherein adjusting the different image capture device comprises at least one of turning on the different image capture device and initializing the different image capture device, wherein the different image capture device comprises at least one of a main camera device and a camera device of a higher power than the image capture device.

[0222]

[0235] Aspect 46. The method of aspect 45, further comprising processing one or more images of the scene via different image capture devices, the one or more images comprising at least one of a second image and a third image captured by the different image capture device.

[0223]

[0236] Aspect 47. The method of any of aspects 29 to 46, wherein the first configuration comprises a non-active optical image stabilization mode and the second configuration comprises an active optical image stabilization mode, and wherein adjusting the lens to the second configuration further comprises activating the optical image stabilization mode using a lens stabilization motor.

[0224]

[0237] Aspect 48. The method of any of aspects 29 to 47, wherein the first configuration has a first aperture setting, and wherein the second configuration has a second aperture setting that is different from the first aperture setting, and wherein adjusting the lens to the second configuration further comprises changing an aperture of the lens from the first aperture setting to the second aperture setting using an aperture motor.

[0225]

[0238] Embodiment 49. The method of any of embodiments 29 to 48, wherein the conditions for successful detection include a predefined or user-configured threshold.

[0226]

[0239] Aspect 50. The method of any of aspects 29 to 49, wherein at least one of the first detection result and the second detection result includes at least one of a probability score or metric, a probabilistic prediction, a classification, a label, a cost or loss function value, an estimate, a ranking, and a binary prediction.

[0227]

[0240] Aspect 51. The method of any of aspects 29 to 50, wherein the first configuration comprises a first lens position, wherein the second configuration comprises a second lens position different from the first lens position, wherein the plurality of available lens configurations comprises a plurality of available lens positions, wherein the second lens position is selected from the plurality of available lens positions based on a relative distance between the first position and each of the plurality of available lens positions, and wherein the second position is selected from the plurality of available lens positions based on a shortest relative distance between the first lens position and each of the plurality of available lens positions.

[0228]

[0241] Aspect 52. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of aspects 29 to 51.

[0229]

[0242] Embodiment 53. An apparatus comprising means for carrying out the method according to any of embodiments 29 to 51.

[0230]

[0243] Aspect 54. An apparatus comprising a memory and one or more processors coupled to the memory, wherein the one or more processors are configured to use a first image captured in a first configuration to make a first determination as to whether an object of interest has been detected, and to use a second image captured in a second configuration to make a second determination as to whether an object of interest has been detected if the detection result of the first determination does not satisfy a configured criterion.

[0231]

[0244] Aspect 55. The apparatus of aspect 54, wherein the first detection result and the second detection result are reliability values.

[0232]

[0245] Embodiment 56. An apparatus according to any of embodiments 54 to 55, wherein the conditions for successful detection include a predefined or user-configured threshold.

[0233]

[0246] Aspect 57. An apparatus as described in any of aspects 54 to 56, wherein the detection result includes one of a probability score or metric, a probabilistic prediction, a classification and / or label, a cost or loss function value, an estimate, a ranking, and / or a binary prediction.

[0234]

[0247] Aspect 58. The apparatus of any of aspects 54 to 57, wherein the first configuration comprises a first lens position of a lens of a camera of the apparatus, and wherein the second configuration comprises a second lens position of the lens, the second lens position being different from the first lens position.

[0235]

[0248] Aspect 59. The apparatus of aspect 58, wherein the one or more processors are configured to move the lens from a first lens position to a second lens position in response to the detection result of the first determination not satisfying a configured criterion.

[0236]

[0249] Aspect 60. The apparatus of aspect 59, wherein the second lens position is selected from the plurality of available lens positions based on an amount of power required by the focus motor to move the lens from the first lens position to the second lens position relative to an amount of power required by the focus motor to move the lens from the first lens position to each other lens position from the plurality of available lens positions.

[0237]

[0250] Aspect 61. The apparatus of any of aspects 54 to 60, wherein the configured criteria comprises an object detection metric being at or above a threshold.

[0238]

[0251] Aspect 62. A method comprising: using a first image captured in a first configuration to make a first determination as to whether an object of interest has been detected; and, if the detection result of the first determination does not satisfy configured criteria, using a second image captured in a second configuration to make a second determination as to whether an object of interest has been detected.

[0239]

[0252] Aspect 63. The method of aspect 62, wherein the first detection result and the second detection result are reliability values.

[0240]

[0253] Embodiment 64. A method according to any of embodiments 62 to 63, wherein the conditions for successful detection include a predefined threshold or a user-configured threshold.

[0241]

[0254] Aspect 65. The method of any of aspects 62 to 64, wherein the detection result comprises one of a probability score or metric, a probabilistic prediction, a classification and / or label, a cost or loss function value, an estimate, a ranking, and / or a binary prediction.

[0242]

[0255] Aspect 66. The method of any of aspects 62 to 65, wherein the first configuration comprises a first lens position of a lens of a camera of the device, and wherein the second configuration comprises a second lens position of the lens, the second lens position being different from the first lens position.

[0243]

[0256] Aspect 67. The method of aspect 66, wherein the one or more processors are configured to move the lens from a first lens position to a second lens position in response to the detection result of the first determination not satisfying a configured criterion.

[0244]

[0257] Aspect 68. The method of aspect 67, wherein the second lens position is selected from the plurality of available lens positions based on an amount of power required by the focus motor to move the lens from the first lens position to the second lens position relative to an amount of power required by the focus motor to move the lens from the first lens position to each other lens position from the plurality of available lens positions.

[0245]

[0258] Aspect 69. The method of any of aspects 62 to 68, wherein the configured criteria comprises an object detection metric being at or above a threshold.

Claims

1. a memory configured to store data; one or more processors coupled to the memory; an apparatus comprising: obtaining a first image of a scene captured by an image capture device based on a trigger event, the first image being captured with a lens of the image capture device in a first configuration of a plurality of available lens configurations; determining whether an object of interest is present in the scene based on the first image of the scene and a first detection result; adjusting the lens to a second configuration selected from the plurality of available lens configurations in response to determining that the object of interest is not present in the scene; capturing, with the image capture device, a second image of the scene while the lens is in the second configuration; and determining that the object of interest is present in the scene based on the second image of the scene and a second detection result; and An apparatus configured to:

2. The apparatus of claim 1 , wherein the first detection result and the second detection result are reliability values.

3. 2. The apparatus of claim 1, wherein the one or more processors are configured to select the second lens configuration from the plurality of available lens configurations based on an amount of power required by the image capture device to adjust the lens to the second lens configuration relative to one or more different amounts of power required by the image capture device to adjust the lens to one or more other lens configurations from the plurality of available lens configurations.

4. the first configuration comprises a first lens position and the second configuration comprises a second lens position different from the first lens position; To adjust the lens to the second configuration, the one or more processors:

2. The apparatus of claim 1 configured to move the lens from the first lens position to the second lens position using a focus motor.

5. the plurality of available lens configurations comprises a plurality of available lens positions; the second lens position is selected from the plurality of available lens positions based on a confidence value associated with the first detection result; Optionally, to select the second lens position, the one or more processors further comprise: eliminating one or more positions as unavailable lens positions for the selection of the second lens position from the plurality of available lens positions based on a first reliability value to reduce a number of the plurality of available lens positions; The apparatus of claim 4 configured to:

6. the plurality of available lens configurations comprises a plurality of available lens positions; the second lens position is selected from the plurality of available lens positions based on an amount of power required by the focus motor to move the lens from the first lens position to the second lens position; Optionally, to select the second lens position, the one or more processors further comprise: comparing the amount of power to one or more amounts of power required by the focus motor to move the lens from the first position to one or more other positions from the plurality of available lens positions; The apparatus of claim 4 configured to:

7. the plurality of available lens configurations comprises a plurality of available lens positions; The apparatus of claim 4 , wherein the second lens position is selected from the plurality of available lens positions based on a reliability value associated with a lens displacement from the first lens position to the second lens position.

8. the plurality of available lens configurations comprises a plurality of available lens positions; the second lens position is selected from the plurality of available lens positions based on a focal length priority associated with the plurality of available lens positions; 5. The apparatus of claim 4, wherein optionally the priority of the focal lengths associated with the plurality of available lens positions is based on a default type of the object of interest or a type of the object detected in the first image.

9. the plurality of available lens configurations comprises a plurality of available lens positions; The second lens position is the relative distance between the first position and each of the plurality of available lens positions; and / or One or more characteristics of the lens. and selecting from the plurality of available lens positions based on 5. The apparatus of claim 4, wherein the one or more characteristics of the lens comprise at least one of an aperture associated with the lens, a field of view associated with the lens, and a focus power profile associated with the lens.

10. the one or more processors are configured to select the lens from a plurality of available lenses based on one or more characteristics of the lens and a focal length associated with the object of interest; 2. The apparatus of claim 1 , optionally wherein the one or more characteristics comprise at least one of an aperture, a field of view, and a focus power profile, or the lens is selected from the plurality of available lenses based on a determination that a focus power profile associated with the lens comprises a lower focus power than a respective focus power profile of one or more lenses from the plurality of available lenses.

11. The apparatus of claim 1 , wherein the object of interest comprises at least one of a document, a quick response (QR) code, a face, a finger, a hand, a device, a product, and an animal.

12. 2. The device of claim 1, wherein the trigger events comprise inertial movement above a threshold, an audio change above a threshold, a change in ambient light above a threshold, a change in range to the object above a threshold, a trigger from an application associated with the device, a depth measurement from an active depth sensing system, a trigger from a global navigation satellite system, a trigger from a global positioning system, a data connection, and a phase detection change above a threshold.

13. The one or more processors: adjusting a different image capture device associated with the apparatus in response to determining that the object of interest is present in the scene; configured to: adjusting the different image capture device comprises at least one of turning on the different image capture device and initializing the different image capture device; the different image capture device comprises at least one of a main camera device and a higher power camera device than the image capture device; Optionally, the one or more processors: processing one or more images of the scene via the different image capture devices; configured to: The apparatus of claim 1 , wherein the one or more images comprise at least one of the second image and a third image captured by the different image capture device.

14. The one or more processors: maintaining the lens in the second configuration based on the second detection result; The apparatus of claim 1 configured to:

15. the first configuration comprises a non-active optical image stabilization mode and the second configuration comprises an active optical image stabilization mode; To adjust the lens to the second configuration, the one or more processors are configured to activate the optical image stabilization mode using a lens stabilization motor.

2. The apparatus of claim 1.

16. the first configuration comprises a first aperture setting and the second configuration comprises a second aperture setting different from the first aperture setting; To adjust the lens to the second configuration, the one or more processors are configured to change an aperture of the lens from the first aperture setting to the second aperture setting using an aperture motor.

2. The apparatus of claim 1.

17. The apparatus of claim 1 , wherein the apparatus comprises a mobile device and / or the image capture device comprises a camera.

18. obtaining a first image of a scene captured by an image capture device based on a trigger event, the first image being captured with a lens of the image capture device in a first configuration of a plurality of available lens configurations; determining whether an object of interest is present in the scene based on the first image of the scene and a first detection result; adjusting the lens to a second configuration selected from the plurality of available lens configurations in response to determining that the object of interest is not present in the scene; capturing, with the image capture device, a second image of the scene while the lens is in the second configuration; and determining that the object of interest is present in the scene based on the second image of the scene and a second detection result; and A method comprising:

19. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: obtaining a first image of a scene captured by an image capture device based on a trigger event, the first image being captured with a lens of the image capture device in a first configuration of a plurality of available lens configurations; determining whether an object of interest is present in the scene based on the first image of the scene and a first detection result; adjusting the lens to a second configuration selected from the plurality of available lens configurations in response to determining that the object of interest is not present in the scene; capturing, with the image capture device, a second image of the scene while the lens is in the second configuration; and determining that the object of interest is present in the scene based on the second image of the scene and a second detection result; and A non-transitory computer-readable medium for causing

Images