Hybrid autofocus system with robust macro object priority focusing

Abstract

An example method includes displaying a zoomed preview of a scene captured by a camera system. The method includes determining a phase detection autofocus (PDAF) depth estimate and a time of flight (ToF) depth estimate of the scene. The method includes determining whether a foreground object in the zoomed preview is in focus for a ToF-based AF mode based on a comparison of the PDAF depth estimate and the ToF depth estimate. The method includes bypassing the PDAF mode and activating the ToF-based AF mode to focus on the foreground object based on determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode. The method includes displaying the focused foreground object as part of a zoomed preview of the scene based on the ToF-based AF mode.

Description

Cross - Reference to Related Applications / Incorporation by Reference

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 378,652, filed on October 6, 2022, which is hereby incorporated by reference in its entirety. Background Art

[0002] Many modern computing devices, including mobile phones, personal computers, and tablet computers, include image - capture devices. Some image - capture devices are configured with multi - camera systems. The camera system is configured to collaboratively meet different image - capture requirements using their respective specifications. A smartphone may integrate multiple types of cameras with multiple focal lengths to take care of objects at different distances and scenes with different fields of view (FOV). Summary of the Invention

[0003] The present disclosure generally relates to switching between multiple cameras. In one aspect, an image - capture device may include multiple cameras. Switching from a normal mode to a macro mode may result in perceivable defocus of a target close - up object. As described herein, a priority - hybrid autofocus strategy is described, whereby perceivable defocus after a camera switch (e.g., switching to an ultra - wide - angle camera) is reduced.

[0004] In a first aspect, a computer - implemented method is provided. The method includes displaying, on a display screen of a camera system, a zoomed - in preview of a scene captured by the camera system. The method includes determining a phase - detection autofocus (PDAF) depth estimate and a time - of - flight (ToF) depth estimate of the scene. The method includes determining whether a foreground object in the zoomed - in preview is in focus for a ToF - based autofocus (AF) mode of the camera system based on a comparison of the PDAF depth estimate with the ToF depth estimate. The method includes, based on determining that the foreground object in the zoomed - in preview is in focus for the ToF - based AF mode, bypassing the PDAF mode and activating the ToF - based AF mode to focus on the foreground object, where the PDAF mode includes focusing by the camera system based on the PDAF depth estimate, and where the ToF - based AF mode includes focusing by the camera system based on the ToF depth estimate. The method includes displaying, on the display screen and based on the ToF - based AF mode, the focused foreground object as part of the zoomed - in preview of the scene.

[0005] In a second aspect, a computing device is provided. The computing device includes: a display screen; a camera system configured to operate at a focal length less than a threshold focal length; one or more processors; and a data storage device, wherein computer-executable instructions are stored on the data storage device, and the computer-executable instructions, when executed by the one or more processors, cause the mobile device to perform functions. The operations include: displaying, on the display screen of the camera system, a zoomed preview of a scene captured by the camera system; receiving, based on the zoomed preview of the scene, a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; determining, based on a comparison of the PDAF depth estimate and the ToF depth estimate, whether a foreground object in the zoomed preview is in focus for a ToF-based AF mode of the camera system; based on determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode, bypassing the PDAF mode and activating the ToF-based AF mode to focus on the foreground object, wherein the PDAF mode includes focusing by the camera system based on the PDAF depth estimate, and wherein the ToF-based AF mode includes focusing by the camera system based on the ToF depth estimate; and displaying, on the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoomed preview of the scene.

[0006] In a third aspect, an article of manufacture is provided. The article of manufacture can include a non-transitory computer-readable medium including program instructions executable by one or more processors to cause the one or more processors to perform operations. The operations include: displaying, on a display screen of a camera system, a zoomed preview of a scene captured by the camera system; determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; determining, based on a comparison of the PDAF depth estimate and the ToF depth estimate, whether a foreground object in the zoomed preview is in focus for a ToF-based AF mode of the camera system; based on determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode, bypassing the PDAF mode and activating the ToF-based AF mode to focus on the foreground object, wherein the PDAF mode includes focusing by the camera system based on the PDAF depth estimate, and wherein the ToF-based AF mode includes focusing by the camera system based on the ToF depth estimate; and displaying, on the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoomed preview of the scene.

[0007] In a fourth aspect, a system is provided. The system includes: means for displaying a zoomed preview of a scene captured by a camera system on a display screen of the camera system; means for determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; means for determining whether a foreground object in the zoomed preview is in focus for a ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate; means for bypassing the PDAF mode and activating the ToF-based AF mode to focus on the foreground object based on determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode, where the PDAF mode includes focusing by the camera system based on the PDAF depth estimate, and where the ToF-based AF mode includes focusing by the camera system based on the ToF depth estimate; and means for displaying the focused foreground object as part of the zoomed preview of the scene on the display screen based on the ToF-based AF mode.

[0008] Other aspects, embodiments, and implementations will become apparent to those of ordinary skill in the art upon reading the following detailed description with appropriate reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 Illustrations of a front view, a right side view, and a rear view of a digital camera device according to an example embodiment.

[0010] Figure 2 Illustration of a preview box showing user-friendly mode switching options according to an example embodiment.

[0011] Figure 3 Illustration of example images with and without macro object focusing based on hybrid AF according to an example embodiment.

[0012] Figure 4 Illustration of example images with and without macro object focusing based on hybrid AF according to an example embodiment.

[0013] Figure 5 Illustration of example images with and without macro object focusing based on hybrid AF according to an example embodiment.

[0014] Figure 6 Illustration of example images with and without macro object focusing based on hybrid AF according to an example embodiment.

[0015] Figure 7 Example workflow of a hybrid autofocus system using robust macro object priority focusing according to an example embodiment.

[0016] Figure 8A is another example workflow of a hybrid autofocus system using robust macro object priority focusing according to an example embodiment.

[0017] Figure 8B is an illustration of multi-grid contrast detection autofocus (CDAF) analysis according to an example embodiment.

[0018] Figure 9 depicts a distributed computing architecture according to an example embodiment.

[0019] Figure 10 is a block diagram of a computing device according to an example embodiment.

[0020] Figure 11 is a flowchart of a method according to an example embodiment.

[0021] Figure 12 shows the difference in image focusing with and without hybrid AF-based macro object focusing according to an example embodiment. DETAILED DESCRIPTION

[0022] Example methods, apparatuses, and systems are described herein. It should be understood that the terms "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration". Any embodiment or feature described herein as "example" or "exemplary" is not necessarily to be construed as preferred or superior to other embodiments or features. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein.

[0023] Thus, the example embodiments described herein are not intended to be limiting. Aspects of the present disclosure, generally described herein and illustrated in the figures, may be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are contemplated herein.

[0024] Further, unless the context otherwise implies, features shown in each figure may be used in combination with each other. Thus, the figures should generally be regarded as constituent aspects of one or more overall embodiments, but it should be understood that not all shown features are necessary for each embodiment. OVERVIEW

[0025] A smart phone or other mobile device that supports image and / or video capture may be equipped with multiple cameras that use corresponding specifications to collaboratively meet different image capture requirements. A smart phone may integrate multiple types of cameras with multiple focal lengths to display and / or capture objects at different distances and scenes in different fields of view (FOV).

[0026] A camera is a device for capturing an image of a scene. Some cameras (e.g., film cameras) capture an image chemically on film. Other cameras (e.g., digital cameras) capture image data electrically (e.g., using a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) sensor). To capture a scene most accurately, a camera can focus on one or more subjects in the scene. There are various ways to focus a camera. For example, the lens of the camera can be moved relative to the image sensor of the camera to adjust the focus of the camera (e.g., to focus on one or more subjects). Similarly, the image sensor of the camera can be moved relative to the lens of the camera to adjust the focus of the camera.

[0027] The adjustment of the focus of the camera can be performed manually (e.g., by a photographer). Alternatively, an autofocus process can be performed to adjust the focus of the camera before capturing an image (e.g., a payload image). The autofocus process can use one or more images (captured by the main image sensor of the camera or one or more auxiliary sensors in the camera) to determine an appropriate focus setting for the camera. Then, based on the determined focus setting, the camera is adjusted to meet that focus setting. For example, a motor can adjust the relative position of the lens and / or the image sensor to meet the determined focus setting.

[0028] There are two traditional types of autofocus processes: an active autofocus process and a passive autofocus process.

[0029] In the active autofocus process, a rangefinder (e.g., a laser rangefinder, a radar device, or a sonar device) is used to determine the distance to one or more objects within the scene. Then, based on the determined distance, a focus setting is determined, and the camera is adjusted to meet the determined focus setting.

[0030] There are two main passive autofocus processes: phase detection autofocus and contrast detection autofocus.

[0031] In phase detection autofocus (PDAF), incident light from the scene is split (e.g., by a beam splitter) such that light from the scene entering one side of the lens of the camera is physically separated on the image sensor (e.g., the main image sensor or an auxiliary image sensor of the camera) from light from the scene entering the other side of the lens. Based on camera characteristics (e.g., lens size, lens focal length, and the position of the lens relative to the image sensor) and the light intensity distribution across various positions on the image sensor, a focus setting can be determined. As with the active autofocus process, the camera can be adjusted to meet the determined focus setting.

[0032] In contrast detection autofocus (CDAF), the camera captures a series of frames at a corresponding series of different focus settings. Then, the contrast between the high intensity and the low intensity in each of the captured frames is determined. Based on the determined contrast, the focus setting is determined (e.g., based on the frame with the highest contrast and / or based on a regression analysis using the contrast of the captured frames). Similar to the active autofocus process and phase detection autofocus, the camera can be adjusted to meet the determined focus setting.

[0033] Different from the active autofocus process, passive autofocus processes (e.g., phase detection autofocus and contrast detection autofocus) do not use additional rangefinding devices. Therefore, passive autofocus processes can be employed in a camera system to save costs (e.g., in a mobile phone or a digital single-lens reflex (DSLR) camera). However, passive autofocus processes may not be very successful under low light conditions (e.g., because the contrast generated between frames used in contrast detection autofocus is insufficient, or because there are not enough bright objects in the scene for comparison when using phase detection autofocus).

[0034] A mobile phone can be configured with: a main camera with a medium focal length to meet ordinary photo / video capture needs; a telephoto camera with a longer focal length to capture distant objects; and a wide-angle or ultra-wide-angle camera with a shorter focal length to capture a larger field of view (FOV). During the photo / video capture phase, when the user continues to zoom in to focus on a distant object, a switch from the main camera to the telephoto camera may occur, while when the user continues to zoom out to capture a larger field of view, a switch from the main camera to the ultra-wide-angle camera may occur. A multi-camera system provides a wider range of focal lengths than a single camera. However, focusing on foreground objects can be challenging when switching from a wide-angle camera to an ultra-wide-angle camera, especially against a high-contrast background.

[0035] For example, in a smartphone camera with a macro mode feature, users typically expect to be able to focus closely on small objects. Close-ups of objects usually have a high-contrast background. In such scenarios using traditional hybrid autofocus (AF) schemes, there may tend to be back-focus, and it may not be possible to automatically achieve the desired focus for the user.

[0036] Typically, each image that a user wishes to capture is desired to be detailed, in focus, worth saving and sharing. It is also desired that the user controls when they wish to enable the functionality, such that it benefits their camera experience. Thus, automatically switching the camera at the appropriate time, dynamically selecting the appropriate lens for the user to obtain the best image quality, providing useful tips to the user, and enhancing the image after capture are important aspects for this feature to provide optimal functionality. Additionally, for example, traditional camera systems typically embed the macro mode deeply in the hardware configuration, and this feature is only available to advanced users (such as professional photographers). Thus, using the macro mode in a user-friendly manner is another aspect of the process described herein.

[0037] To make the process more user-friendly, a hybrid autofocus strategy that prioritizes macro objects can be deployed. For example, traditional hybrid AF strategy hierarchies involve applying a phase detection autofocus (PDAF) algorithm, followed by a time-of-flight (ToF)-based algorithm and a contrast detection autofocus (CDAF) algorithm. However, although there is a high confidence level for PDAF-based depth estimation, the PDAF algorithm can cause the camera lens to focus on the background, and thus the foreground object can remain out of focus. Therefore, it is necessary to override the PDAF algorithm in order to be able to focus the foreground object more clearly, as described herein.

[0038] In addition to focusing on foreground objects, the techniques described herein also enable objects as close as 3 cm to be photographed. When combined with HDR+ quality processing, small objects (such as raindrops, individual flower petals, pollen grains, etc.) can be clearly focused. For example, the techniques described herein can enable objects such as small organisms, such as plants, pets, insects, human eyes, animal eyes, feathers, mushrooms, etc. to be photographed. The techniques described herein also enable clearer image capture of unique textures, such as jeans, leather, cotton, any type of fabric, stone, brick, rough surfaces, smooth surfaces, rust, paint, tissue paper fabric, mouse pads, tin foil, ice cubes, foam, and bubbles. Additionally, for example, natural objects such as fruits, vegetables, water droplets, trees, moss, grass, snowflakes, shells, seeds, etc. can be more clearly focused. Generally, any object that can have a unique appearance and / or display new image information when viewed up close can be clearly focused. Such objects can include, for example, coins, wax tips, pencil tips, matches, needles, cotton swabs, musical instruments, handwriting, paper, fingerprints, buttons, jewelry, floor tiles, etc. The macro mode of an ultra-wide-angle camera can be seamlessly integrated with other camera systems that provide a zoom ratio from to the zoom ratio. Example camera system

[0039] As image capture devices such as cameras become increasingly popular, they can be used as standalone hardware devices or integrated into various other types of devices. For example, static and video cameras are now commonly included in wireless computing devices (e.g., mobile devices such as mobile phones), tablet computers, laptop computers, video game interfaces, home automation devices, and even in cars and other types of vehicles.

[0040] The physical components of a camera can include one or more apertures through which light enters, one or more recording surfaces for capturing an image represented by the light, and lenses located in front of each aperture to focus at least a portion of the image onto the recording surface. The aperture can be of a fixed size or adjustable. In an analog camera, the recording surface can be photographic film. In a digital camera, the recording surface can include an electronic image sensor (e.g., a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor) to transfer and / or store the captured image in a data storage unit (e.g., memory).

[0041] One or more shutters can be coupled to the lens or near the recording surface or both. Each shutter can be in a closed position (blocking light from reaching the recording surface) or an open position (allowing light to reach the recording surface). The position of each shutter can be controlled by a shutter button. For example, the shutter may default to the closed position. When triggered (e.g., pressed), the shutter button causes the shutter to change from the closed position to the open position for a period of time, which is referred to as the shutter cycle. During the shutter cycle, an image can be captured on the recording surface. At the end of the shutter cycle, the shutter can return to the closed position.

[0042] Alternatively, the shuttering process can be electronic. For example, before the electronic shutter of a CCD image sensor "opens", the sensor can be reset to eliminate any residual signal in its photodiodes. While the electronic shutter remains open, the photodiodes can accumulate charge. When the shutter closes or after it closes, this charge can be transferred to a more long-term data storage device. A combination of a mechanical shutter and electronic shuttering is also possible.

[0043] Regardless of type, the shutter can be activated and / or controlled by something other than a shutter button. For example, the shutter can be activated by a soft key, a timer, or other triggers. In this document, the term "image capture" can refer to any mechanical and / or electronic shuttering process that results in the recording of one or more images, regardless of how the shuttering process is triggered or controlled.

[0044] The exposure of the captured image can be determined by a combination of the size of the aperture, the brightness of the light entering the aperture, and the duration of the shutter cycle (also referred to as shutter duration, exposure duration, or exposure time). Additionally, digital and / or analog gain (e.g., based on ISO settings) can be applied to the image, thereby affecting the exposure. In some embodiments, the terms "exposure duration", "exposure time", or "exposure time interval" can refer to the shutter duration multiplied by the gain for a particular aperture size. Thus, these terms can be used somewhat interchangeably and should be interpreted as the shutter duration, exposure time, and / or any other metric that controls the amount of signal response due to light reaching the recording surface.

[0045] In some implementations or operating modes, whenever an image capture is triggered, the camera can capture one or more still images. In other implementations or operating modes, as long as the image capture remains triggered (e.g., when the shutter button is held down), the camera can capture video images by continuously capturing images at a particular rate (e.g., 24 frames per second). Some cameras can open the shutter when the camera device or application is activated while operating in a mode to capture still images, and the shutter can remain in that position until the camera device or application is deactivated. When the shutter is open, the camera device or application can capture the scene and display a representation of the scene on the viewfinder (sometimes referred to as showing a "preview box"). When an image capture is triggered, one or more different payload images of the current scene can be captured.

[0046] Cameras (including digital cameras and analog cameras) can include software for controlling one or more camera functions and / or settings such as aperture size, exposure time, gain, etc. Additionally, some cameras can include software for digitally processing the image during or after image capture. While the above description generally refers to cameras, it can be particularly applicable to digital cameras. A digital camera can be a stand-alone device (e.g., a DSLR camera) or can be integrated with other devices.

[0047] Figure 1Illustrations of a front view, a right side view, and a rear view of a digital camera device 100 according to an example embodiment. The digital camera device 100 may be, for example, a mobile device (e.g., a mobile phone), a tablet computer, or a wearable computing device. Other embodiments are possible. The digital camera device 100 may include various elements such as a body 102, a front camera 104, a multi-element display 106, a shutter button 108, and other buttons 110. The digital camera device 100 may further include a rear camera 112. The front camera 104 may be located on a side of the body 102 that is typically facing the user during operation or on the same side as the multi-element display 106. The rear camera 112 may be located on a side of the body 102 opposite to the front camera 104. Referring to the cameras as front and rear is arbitrary, and the digital camera device 100 may include multiple cameras located on each side of the body 102.

[0048] The multi-element display 106 may represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal display (LCD), a plasma display, or any other type of display known in the art. In some embodiments, the multi-element display 106 may display a current image captured by the front camera 104 and / or the rear camera 112 or a digital representation of an image that may be or has been recently captured by either or both of these cameras. Thus, the multi-element display 106 may serve as a viewfinder for either camera. The multi-element display 106 may also support touchscreen and / or presence-sensitive functionality that may be capable of adjusting settings and / or configurations of any aspect of the digital camera device 100.

[0049] The front camera 104 may include an image sensor and associated optical elements such as a lens. The front camera 104 may provide zoom capabilities or may have a fixed focal length. In other embodiments, interchangeable lenses may be used with the front camera 104. The front camera 104 may have a variable mechanical aperture and mechanical and / or electronic shutters. The front camera 104 may also be configured to capture still images, video images, or both. Additionally, the front camera 104 may represent a single field of view, stereo, or multi-field of view camera. The rear camera 112 may be similarly or differently arranged. Additionally, the front camera 104, the rear camera 112, or both may be an array of one or more cameras.

[0050] Either or both of the front camera 104 and the rear camera 112 may include or be associated with an illumination component that provides a light field for illuminating a target object. For example, the illumination component may provide a flash or constant illumination for the target object (e.g., using one or more LEDs). The illumination component may also be configured to provide a light field that includes one or more of structured light, polarized light, and light having a specific spectral content. Other types of light fields known and used to recover a three-dimensional (3D) model of an object are also possible in the context of the embodiments herein.

[0051] Either or both of the front camera 104 and the rear camera 112 may include or be associated with an ambient light sensor that can continuously or periodically determine the ambient brightness of the scene that the camera can capture. In some devices, the ambient light sensor may be used to adjust the display brightness of a screen (e.g., a viewfinder) associated with the camera. When the determined ambient brightness is high, the brightness level of the screen may be increased to make the screen easier to view. When the determined ambient brightness is low, the brightness level of the screen may be decreased to also make the screen easier to view and potentially save power. Additionally, the input of the ambient light sensor may be used to determine the exposure duration of the associated camera or to assist in such determination.

[0052] The digital camera device 100 may be configured to capture an image of a target object (i.e., a subject within a scene) using the multi-element display 106 and the front camera 104 or the rear camera 112. The captured image may be a plurality of still images or video images (e.g., a series of still images captured in quick succession with or without accompanying audio captured by a microphone). Image capture may be triggered by activating the shutter button 108, pressing a soft key on the multi-element display 106, or by some other mechanism. Depending on the implementation, images may be automatically captured at specific time intervals (e.g., after pressing the shutter button 108, after appropriate illumination conditions of the target object, after moving the digital camera device 100 a predetermined distance, or according to a predetermined capture schedule).

[0053] As described above, the functionality of the digital camera device 100 (or another type of digital camera) may be integrated into a computing device (such as a wireless computing device, a mobile phone, a tablet computer, a laptop computer, etc.). For example, a camera controller may be integrated with the digital camera device 100 to control one or more functions of the digital camera device 100.

[0054] Figure 2 is an illustration of a preview box 202 that displays user-friendly mode switching options according to an example embodiment. The preview box 202 may display the captured frame to the user based on the current scene captured using the current camera system settings (e.g., aperture settings, exposure settings, etc.). When the preview box looks the same as Figure 2When the preview frame 202 is similar, the techniques described in this document can be used.

[0055] In some embodiments, when a previous autofocus (e.g., based on a traditional PDAF algorithm) is unsuccessful, the hybrid autofocus process described in this document can be triggered. For example, Figure 2 the preview frame 202 shown in may not sufficiently focus the payload image, so a hybrid autofocus process can be performed. Whether the previous autofocus is unsuccessful can be determined based on an indication from the user (i.e., the previous autofocus was not appropriate). In other embodiments, the hybrid autofocus algorithm (e.g., the PDAF algorithm used in preview mode) can provide an indication of autofocus failure. For example, the autofocus algorithm can provide a PDAF confidence value indicating the probability of autofocus success, and if this confidence value is below a certain threshold (e.g., the PDAF confidence threshold), it can be determined that autofocus has failed. The indication that autofocus has failed can be provided by an API (e.g., the API of the camera module of the mobile device).

[0056] Whether the hybrid autofocus is successful can be based on the hybrid autofocus algorithm (e.g., the time-of-flight (ToF) algorithm used in preview mode), which can provide an indication of autofocus success. For example, the autofocus algorithm can provide a ToF confidence value indicating the probability of autofocus success, and if this confidence value is above a certain threshold (e.g., the ToF confidence threshold), it can be determined that autofocus has been successful. The indication that autofocus has been successful can be provided by an API (e.g., the API of the camera module of the mobile device).

[0057] For example, selectable virtual objects can be provided to the user (e.g., during a camera transition from the main camera to the ultra-wide-angle camera, or during the operation of the ultra-wide-angle camera) to indicate whether to enable or disable the hybrid autofocus mode described in this document. For example, a toggle switch can be displayed on the multi-component display 106 of the digital camera device 100 to enable or disable the hybrid autofocus mode.

[0058] Figure 3 is an illustration of example images using and not using hybrid AF-based macro object focusing according to an example embodiment. Figure 3 With Figure 1 and Figure 2 share one or more common aspects. The digital camera device 300A shows an image captured using a traditional PDAF method. As shown, one or more foreground objects may be out of focus. The digital camera device 300B shows a case where hybrid AF-based macro object focusing is used. Therefore, applying reference Figure 7the algorithm described with reference to FIGS. 8, and can focus one or more foreground objects. For example, a ToF-based AF mode can be applied (e.g., a multi-grid, multi-directional, multi-frequency CDAF scan based on ToF distance as described in block 830 of FIG. 8).

[0059] Figure 4 is an illustration of example images with and without macro object focusing based on hybrid AF according to an example embodiment. Figure 4 and Figures 1 to 3 share one or more common aspects. Digital camera device 400A shows an image captured using a conventional PDAF method. As shown, one or more foreground objects may be out of focus. Digital camera device 400B shows a case where an image is captured using macro object focusing based on hybrid AF. Thus, the algorithm described with reference to Figure 7 and FIGS. 8 is applied, and one or more foreground objects can be focused. For example, a ToF-based AF mode can be applied (e.g., a multi-grid, multi-directional, multi-frequency CDAF scan based on ToF distance as described in block 830 of FIG. 8).

[0060] Figure 5 is an illustration of example images with and without macro object focusing based on hybrid AF according to an example embodiment. Figure 5 and Figures 1 to 4 share one or more common aspects. Digital camera device 500A shows an image captured using a conventional PDAF method. As shown, one or more foreground objects may be out of focus. Digital camera device 500B shows a case where an image is captured using macro object focusing based on hybrid AF. Thus, the algorithm described with reference to Figure 7 and FIGS. 8 is applied, and one or more foreground objects can be focused. For example, a ToF-based AF mode can be applied (e.g., a multi-grid, multi-directional, multi-frequency CDAF scan based on ToF distance as described in block 830 of FIG. 8).

[0061] Figure 6 is an illustration of example images with and without macro object focusing based on hybrid AF according to an example embodiment. Figure 6 and Figures 1 to 5 share one or more common aspects. Digital camera device 600A shows an image captured using a conventional PDAF method. As shown, one or more foreground objects may be out of focus. Digital camera device 600B shows a case where an image is captured using macro object focusing based on hybrid AF. Thus, the algorithm described with reference to Figure 7The algorithm described in FIGS. 8 and can focus one or more foreground objects. For example, a ToF-based AF mode can be applied (e.g., multi-grid, multi-direction, multi-frequency CDAF scans based on ToF distance as described in block 830 of FIG. 8). Hybrid AF algorithm for macro mode

[0062] Traditional hybrid AF schemes prioritize PDAF->TOF->CDAF based on PDAF confidence. PDAF is an efficient method for continuous focusing of a camera as it relies on parallax information from the image sensor and directly controls the lens to optimize the circle of confusion projected on the image sensor for the AF region of interest (ROI). If PDAF is reliable, it is used to drive focusing. In some cases, such as in low light conditions, due to noise from lower SNR, PDAF parallax estimation may fail. In such situations, ToF can be a good complement for focusing using metric depth estimation that is converted to a focusing lens position via depth-to-position mapping. However, there may be accuracy issues with this mapping, so some contrast scans (e.g., CDAF) may be needed to narrow the accuracy gap. Finally, if ToF is unreliable or ineffective, the AF system will rely on CDAF scans as a last resort. Generally, CDAF requires focusing scans and is not optimal for fast-moving subjects.

[0063] Thus, in a traditional hybrid AF system, if PDAF is reliable, it is directly used to drive focusing and the decision to set the focus is not overridden by other available focus data. However, for macro mode, this scheme may not be optimal in cases where small and / or thin objects contrast against a high-contrast background, resulting in an unwanted back focus. Additionally, for example, the image quality based on the PDAF mode may not be sufficient to view close objects. For example, sparse PDAF may not be able to detect close objects (e.g., foreground objects).

[0064] Phase detection autofocus is a passive autofocus technique that attempts to determine the appropriate focus setting (e.g., the position of the lens and / or the position of the image sensor) of a camera system based on a subject within the scene of the surrounding environment that will ultimately be captured in the payload image. The phase detection autofocus function is achieved by splitting the light entering the camera system into two or more parts. Those parts can be captured and then compared to each other. The two or more parts are compared to determine the relative positions of intensity peaks and valleys across corresponding frames. If the relative positions within the frame match, the subject of the scene is in focus. If the relative positions do not match, the subject of the scene is out of focus. Based on the distance between the corresponding peaks and valleys and the position of the optics (e.g., the lens, the image sensor, etc.) within the camera system, an adjustment can be determined that will move the subject into focus.

[0065] Further, in some embodiments, one or more objects in the scene may be in focus while other objects are still out of focus. Thus, determining whether the scene is out of focus can include selecting one or more subjects within the scene for which to make the determination. The region of interest for focus determination can be selected based on, for example, the user (of the mobile device). For example, the user can select an object (e.g., a building, a person, a face, a car, etc.) within a preview frame that the user wishes to be in focus. Alternatively, an object recognition algorithm can be employed to determine what types of objects are in the scene and to determine which object should be in focus based on a list sorted by importance (e.g., if there is a face in the scene, the face should be the object in focus, then a dog, then a building, etc.). In still other embodiments, determining whether the scene is in focus or out of focus can include identifying whether an object moving within the scene (e.g., as determined by the preview frame) is in focus. Alternatively, determining the "in focus" camera settings can include determining such a lens setting under which the maximized region (e.g., by pixel area) of the frame is in focus, or the maximized number of subjects (e.g., one discrete object, two discrete objects, three discrete objects, four discrete objects, etc.) within the frame is in focus.

[0066] Generally, ToF focusing is an active autofocus technique where the camera can measure the target distance by actively illuminating the object. The illumination can be performed using a light source such as an LED or a laser. The ToF sensor captures the light reflected by the object. Generally, the ToF sensor is configured to be sensitive to different wavelengths, and the ToF sensor can measure the time delay of the light reflected back to the sensor, and can determine the ToF depth estimate based on that time delay. For example, the time delay is typically proportional to twice the distance from the camera to the object (corresponding to the round-trip distance that the light leaves the camera, is reflected, and returns to the camera). Thus, the ToF depth estimate can be determined as where is a proportionality constant.

[0067] As described herein, the hybrid AF macro object priority scheme prioritizes the ToF-based AF mode over the PDAF mode to assist a user in focusing on a close object in macro mode. This method remains applicable even in cases where the brightness level of the scene exceeds a threshold level and the PDAF estimation is valid and reliable. In such cases, a conventional camera system uses the PDAF mode to drive focusing.

[0068] However, in some embodiments, ToF focusing may lose ranging accuracy in bright light conditions and spatial parallax shifts may occur in the camera FOV at closer object distances. Thus, when the confidence of the ToF depth estimation is not sufficient to drive focusing alone, the ToF-based AF mode can be configured to use a multi-grid CDAF search to constrain the focus scan around the focus position estimated by TOF to assist in precisely locating the focus position and thereby overcome the ambiguity of the focus data available to conventional AF algorithms.

[0069] Figure 7 is an example workflow 700 of a hybrid autofocus system using robust macro object priority focusing according to an example embodiment. As shown, a PDAF bypass determination module 705 can be initiated. For example, this can be initiated based on a user indication or automatically determined based on camera sensor data (e.g., invoked by the hybrid AF macro object priority module 805 of FIG. 8). As used herein, the term "ToF-based AF mode" generally refers to the operations performed at step 825 and / or step 830 of FIG. 8. Additionally, for example, the term "PDAF mode" as used herein generally refers to the mode that runs a conventional PDAF algorithm.

[0070] At step 710, the process involves determining whether the PDAF depth estimation is valid and whether the PDAF depth estimation exceeds a PDAF confidence threshold. Generally, PDAF pixels work by capturing two slightly different views of a scene. The parallax effect (where the object remains stationary while the background level moves) can be used to estimate the PDAF depth. For example, parallax is a function of the distance of a point from the camera and the distance between the two viewpoints. Thus, PDAF depth estimation can be performed by matching each point in one view with its corresponding point in the other view. However, finding these correspondences in a PDAF image (i.e., determining depth from a stereo image) can be a challenging task because scene points move very little between views. Additionally, for example, stereo techniques can involve aperture problems (i.e., when observing a scene through a small aperture, it may not always be possible to find correspondences for lines parallel to the stereo baseline (i.e., the line connecting the two cameras)). Thus, PDAF estimations can sometimes be incorrect.

[0071] When it is determined that the PDAF depth estimate is invalid (i.e., incorrect) or the PDAF depth estimate does not exceed the PDAF confidence threshold, the process proceeds to step 715, and the "bypass PDAF" parameter is set to "FALSE", indicating that the PDAF mode remains active (e.g., maintaining the PDAF mode, not bypassing the PDAF mode, etc.), and the ToF-based AF mode is not activated.

[0072] When it is determined that the PDAF depth estimate exceeds the PDAF confidence threshold, the process proceeds to step 720.

[0073] At step 720, the process involves comparing the PDAF depth estimate with the ToF depth estimate to determine whether the foreground object in the zoomed preview is in focus for the camera system's ToF-based AF mode (and may be out of focus for the PDAF mode). For example, the comparison of the PDAF depth estimate with the ToF depth estimate involves determining whether the differential depth estimate based on the difference between the PDAF depth estimate and the ToF depth estimate exceeds a depth threshold, and determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode is based on determining that the differential depth estimate exceeds the depth threshold.

[0074] In some embodiments, based on determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode, the PDAF mode is bypassed and the ToF-based AF mode is activated to focus on the foreground object, where the PDAF mode includes focusing by the camera system based on the PDAF depth estimate, and where the ToF-based AF mode includes focusing by the camera system based on the ToF depth estimate.

[0075] When it is determined that the differential depth estimate does not exceed the depth threshold, the process proceeds to step 715, and the "bypass PDAF" parameter is set to "false", indicating that the PDAF mode remains active, and the ToF-based AF mode is not activated.

[0076] When it is determined that the differential depth estimate exceeds the depth threshold, the process proceeds to step 725.

[0077] At step 725, the process involves determining whether the ToF depth estimate is valid. For example, the ToF sensor may determine that the object is very close, but the estimate may be inaccurate. This can result in an invalid ToF depth estimate.

[0078] When it is determined that the ToF depth estimate is invalid, the process proceeds to step 715, and the "bypass PDAF" parameter is set to "false", indicating that the PDAF mode remains active, and the ToF-based AF mode is not activated.

[0079] When it is determined that the ToF depth estimation is valid, the process proceeds to step 730.

[0080] At step 730, the process involves determining whether the luminance intensity of the background exceeds a luminance threshold.

[0081] When it is determined that the luminance intensity of the background does not exceed the luminance threshold, the process proceeds to step 715 and the "bypass PDAF" parameter is set to "false", indicating that the PDAF mode remains active and the ToF-based AF mode is not activated.

[0082] When it is determined that the luminance intensity of the background exceeds the luminance threshold, the process proceeds to step 735 and the "bypass PDAF" parameter is set to "TRUE", indicating that the PDAF mode is bypassed and the ToF-based AF mode is activated.

[0083] Figure 8A Another example workflow 800A of a hybrid autofocus system using robust macro object priority focusing according to an example embodiment is shown. As shown, the hybrid AF macro object priority module 805 can be initiated. In some embodiments, the hybrid AF macro object priority module 805 can implement the autofocus aspects of the camera system.

[0084] At step 810, the process involves determining whether to bypass the PDAF mode. For example, the hybrid AF macro object priority module 805 can trigger Figure 7 the PDAF bypass determination module 705 shown.

[0085] When it is determined not to bypass the PDAF mode (e.g., the workflow 700 terminates at step 715), the process proceeds to step 815. At step 815, the camera system uses a traditional hybrid AF strategy hierarchy, which involves applying a phase detection autofocus (PDAF) algorithm, followed by a time-of-flight (ToF)-based algorithm and a contrast detection autofocus (CDAF) algorithm.

[0086] When it is determined to bypass the PDAF mode (e.g., the workflow 700 terminates at step 735), the process proceeds to step 820.

[0087] At step 820, the process involves determining whether the ToF depth estimation is valid and whether the ToF depth estimation exceeds a ToF confidence threshold.

[0088] When it is determined that the ToF depth estimation is valid and the ToF depth estimation exceeds the ToF confidence threshold, the process proceeds to step 825.

[0089] At step 825, the process involves focusing the camera system in ToF-based AF mode based on a distance-to-position mapping based on ToF depth estimation for the foreground object.

[0090] In some embodiments, the ToF sensor may determine that an object is very close, but the estimate may be inaccurate or may be untrustworthy in bright light environments. When it is determined that the ToF depth estimate is invalid and the ToF depth estimate does not exceed the ToF confidence threshold, the process proceeds to step 830.

[0091] At step 830, the process involves focusing the camera system in ToF-based AF mode based on a multi-grid contrast detection autofocus (CDAF) search based on the ToF depth estimate. In some embodiments, the multi-grid CDAF search is based on one or more of a spatial grid, one or more directions, or one or more spatial frequencies. For example, two directions may be used: a horizontal direction and a vertical direction. Additionally, for example, the grid may be an array.

[0092] Figure 8B is an illustration of a multi-grid contrast detection autofocus (CDAF) analysis 800B according to an example embodiment. As shown, a 5×5 grid 835 of an image is shown. Based on the 5×5 grid 835, one or more spatial frequencies (e.g., high frequency, medium frequency, etc.) may be determined, and one or more spatial directions (e.g., horizontal, vertical, etc.) may be used. For example, a 5×5 array of FV high frequencies 840 may be determined. Additionally, for example, a 5×5 array of FV medium frequencies 845 may be determined. For example, each sub-grid in the grid 835 corresponds to a high frequency distribution and a medium frequency distribution. Thus, each of the high frequencies 840 and the medium frequencies 845 includes a corresponding array of focus value (FV) curves. In some embodiments, the FV curve may be determined as the sum of one or more directions (e.g., the sum of the horizontal direction and the vertical direction). However, other combinations may be used to generate the FV curve. The actual FV curve is not relevant to this discussion, and the curves shown are for illustrative purposes only. Additionally, for example, although high frequencies and medium frequencies are shown, other frequencies may also be utilized.

[0093] In some embodiments, the CDAF search may also include determining, for the grid 835, a 5×5 array of peak signal-to-noise ratio (PSNR) 850, a 5×5 sharpness ratio array 855, a 5×5 array of peak focus positions 860, and so on. Different intensities of the PSNR 850, sharpness ratio 855, and peak focus positions 860 may be represented by different colors, shades, and so on. For example, the final focus position may be determined based on a weighted histogram analysis that takes into account interpolated peaks from each grid, the final focus position being weighted by its FV curve quality metric, which may be represented as the PSNR 850 or the sharpness ratio 855. Additional and / or alternative quality factors may be used to determine the quality metric. For example, the quality metric may be based on one or more factors such as unimodality, accuracy, repeatability, defined range, general applicability, and robustness. In some embodiments, the final position may be determined based on percentiles in a histogram that takes into account depth of field (e.g., 33% for the rule of thirds). Example data network

[0094] Figure 9 Depicts a distributed computing architecture 900 in accordance with an example embodiment. The distributed computing architecture 900 includes server devices 908, 910 configured to communicate with programmable devices 904a, 904b, 904c, 904d, 904e via a network 906. The network 906 may correspond to a local area network (LAN), wide area network (WAN), WLAN, WWAN, enterprise intranet, public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 906 may also correspond to a combination of one or more LANs, WANs, enterprise intranets, and / or the public Internet.

[0095] Although Figure 9Only five programmable devices are shown, but a distributed application architecture can serve dozens, hundreds, or thousands of programmable devices. Additionally, the programmable devices 904a, 904b, 904c, 904d, 904e (or any additional programmable devices) can be any kind of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mounted device (HMD), a network terminal, a mobile computing device, and so on. In some examples, as shown by the programmable devices 904a, 904b, 904c, 904e, the programmable devices can be directly connected to the network 906. In other examples, as shown by the programmable device 904d, the programmable device can be indirectly connected to the network 906 via an associated computing device (such as the programmable device 904c). In this example, the programmable device 904c can act as the associated computing device for relaying electronic communications between the programmable device 904d and the network 906. In other examples, as shown by the programmable device 904e, the computing device can be part of and / or located inside a vehicle (such as a car, a truck, a bus, a boat or a ship, an airplane, etc.). In Figure 9 In other examples not shown, the programmable device can be both directly and indirectly connected to the network 906.

[0096] The server devices 908, 910 can be configured to perform one or more services requested by the programmable devices 904a through 904e. For example, the server devices 908 and / or 910 can provide content to the programmable devices 904a through 904e. The content can include, but is not limited to, web pages, hypertext, scripts, binary data (such as compiled software), images, audio, and / or video. The content can include compressed and / or uncompressed content. The content can be encrypted and / or decrypted. Other types of content are also possible.

[0097] As another example, the server devices 908 and / or 910 can provide the programmable devices 904a through 904e with access to software for databases, search, computing, graphics, audio, video, World Wide Web / Internet utilization, and / or other functions. Many other examples of server devices are also possible. Computing Device Architecture

[0098] Figure 10 is a block diagram of an example computing device 1000 according to an example embodiment. Specifically, Figure 10 the illustrated computing device 1000 can be configured to perform at least one function of method 1100 and / or at least one function related to the method.

[0099] By way of example and not limitation, computing device 1000 can be a cellular mobile phone (e.g., a smart phone), a still camera, a video camera, a fax machine, a computer (such as a desktop computer, a laptop computer, a tablet computer, or a handheld computer), a personal digital assistant (PDA), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, or some other type of device equipped with at least some image capture and / or image processing capabilities. It should be understood that computing device 1000 can represent a physical image processing device, such as a digital camera, a specific physical hardware platform on which a camera application operates in software, or some other combination of hardware and software configured to perform camera functions.

[0100] As Figure 10 shown, computing device 1000 can include a user interface module 1001, a network communication module 1002, one or more processors 1003, a data storage device 1004, one or more cameras 1018, one or more sensors 1020, and a power system 1022, all of which can be linked together via a system bus, a network, or some other connection mechanism 1005.

[0101] User interface module 1001 may be operable to send data to and / or receive data from external user input / output devices. For example, user interface module 1001 can be configured to send data to and / or receive data from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, and / or other similar devices. User interface module 1001 can also be configured to provide output to a user display device, such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, whether now known or later developed. User interface module 1001 can also be configured to generate audible output using devices such as speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. User interface module 1001 can further be configured with one or more haptic devices that can generate haptic output, such as vibrations, and / or other output detectable by touch and / or physical contact with computing device 1000. In some examples, user interface module 1001 can be used to provide a graphical user interface (GUI) for utilizing computing device 1000.

[0102] In some embodiments, the user interface module 1001 may include a display that serves as a viewfinder for static and / or video camera functions supported by the computing device 1000. Additionally, the user interface module 1001 may include one or more buttons, switches, knobs, and / or dials that facilitate the configuration and focusing of the camera functions and the capture of images (e.g., taking pictures). Some or all of these buttons, switches, knobs, and / or dials may be implemented via a presence-sensitive panel.

[0103] The network communication module 1002 may include one or more devices that provide one or more wireless interfaces 1007 and / or one or more wired interfaces 1008 that are configurable to communicate via a network. The wireless interface 1007 may include one or more wireless transmitters, receivers, and / or transceivers, such as Bluetooth™ transceivers, Zigbee® transceivers, Wi-Fi™ transceivers, WiMAX™ transceivers, LTE™ transceivers, and / or other types of wireless transceivers that are configurable to communicate via a wireless network. The wired interface 1008 may include one or more wired transmitters, receivers, and / or transceivers, such as Ethernet transceivers, universal serial bus (USB) transceivers, or similar transceivers that can be configured to communicate via twisted pair, coaxial cable, fiber optic link, or similar physical connections to a wired network.

[0104] In some examples, the network communication module 1002 may be configured to provide reliable, protected, and / or authenticated communication. For each communication described herein, information may be provided to facilitate reliable communication (e.g., ensuring message delivery), which may be part of a message header and / or message tail (e.g., packet / message sequencing information, encapsulation headers and / or encapsulation tails, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity values). One or more cryptographic protocols and / or algorithms may be used to secure (e.g., encode or encrypt) the communication and / or decrypt / decode the communication, such as but not limited to the Data Encryption Standard (DES), Advanced Encryption Standard (AES), Rivest-Shamir-Adelman (RSA) algorithm, Diffie-Hellman algorithm, secure socket protocols such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms may be used to secure (and then decrypt / decode) the communication in addition to those listed herein.

[0105] One or more processors 1003 may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). The one or more processors 1003 may be configured to execute computer-readable instructions 1006 contained in the data storage device 1004 and / or other instructions as described herein.

[0106] The data storage device 1004 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 1003. The one or more computer-readable storage media may include volatile and / or non-volatile storage components that are integrated, in whole or in part, with at least one of the one or more processors 1003, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the data storage device 1004 may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage unit), while in other examples, the data storage device 1004 may be implemented using two or more physical devices.

[0107] The data storage device 1004 may include computer-readable instructions 1006 and possibly additional data. In some examples, the data storage device 1004 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, the data storage device 1004 may include storage for the hybrid AF module 1012 (e.g., a module that performs a hybrid AF macro object prioritization process and calculates PDAF algorithms, ToF algorithms, CDAF algorithms, etc. and performs one or more operations related to the hybrid autofocus system with robust macro object prioritized focusing described herein). Specifically, in these examples, the computer-readable instructions 1006 may include instructions that, when executed by the processor 1003, cause the computing device 1000 to provide some or all of the functionality of the hybrid AF 1012.

[0108] In some examples, computing device 1000 may include one or more cameras 1018. The cameras 1018 may include one or more image capture devices, such as a still camera and / or a video camera, that are configured to capture light and record the captured light in one or more images; that is, the cameras 1018 may generate images of the captured light. The one or more images may be one or more still images and / or one or more images used in a video stream. The cameras 1018 may capture light and / or electromagnetic radiation that is emitted as visible light, infrared radiation, ultraviolet light, and / or as light and / or electromagnetic radiation at one or more other frequencies. The cameras 1018 may include a wide-angle camera, a telephoto camera, an ultra-wide-angle camera, and the like. Additionally, for example, relative to the computing device 1000, the cameras 1018 may be a front-facing camera or a rear-facing camera. The cameras 1018 may include camera components such as, but not limited to, an aperture, a shutter, a recording surface (e.g., photographic film and / or an image sensor), a lens, and / or a shutter button. The camera components may be controlled at least in part by software executed by one or more processors 1003.

[0109] In some examples, computing device 1000 may include one or more sensors 1020. The sensors 1020 may be configured to measure conditions within the computing device 1000 and / or conditions in the environment of the computing device 1000 and provide data regarding such conditions. For example, sensors 1020 may include one or more of the following: (i) sensors for obtaining data regarding the computing device 1000, such as but not limited to a thermometer for measuring the temperature of the computing device 1000, a battery sensor for measuring the power of one or more batteries of the power system 1022, and / or other sensors for measuring the conditions of the computing device 1000; (ii) identification sensors for identifying other objects and / or devices, such as but not limited to radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcodes (e.g., Quick Response (QR) codes) readers, and laser trackers, where the identification sensors may be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or objects configured to be read and provide at least identification information; (iii) sensors for measuring the position and / or movement of the computing device 1000, such as but not limited to tilt sensors, gyroscopes, accelerometers, Doppler sensors, GPS devices, sonar sensors, radar devices, laser displacement sensors, and compasses; (iv) environmental sensors for obtaining data indicative of the environment of the computing device 1000, such as but not limited to infrared sensors, optical sensors, light sensors, biosensors, capacitance sensors, touch sensors, temperature sensors, wireless sensors, radio sensors, motion sensors, microphones, sound sensors, ultrasonic sensors, and / or smoke sensors; and / or (v) force sensors for measuring one or more forces acting around the computing device 1000 (e.g., inertial forces and / or G-forces), such as but not limited to one or more sensors for measuring one or more of the following: force in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor for identifying the ZMP and / or the position of the ZMP. Many other examples of sensors 1020 are possible.

[0110] The power supply system 1022 may include one or more batteries 1024 and / or one or more external power interfaces 1026 for providing power to the computing device 1000. Each of the one or more batteries 1024 can act as a source of stored power for the computing device 1000 when electrically coupled to the computing device 1000. The one or more batteries 1024 of the power supply system 1022 can be configured to be portable. Some or all of the one or more batteries 1024 can be easily removed from the computing device 1000. In other examples, some or all of the one or more batteries 1024 can be inside the computing device 1000 and may thus not be easily removable from the computing device 1000. Some or all of the one or more batteries 1024 can be rechargeable. For example, a rechargeable battery can be recharged via a wired connection between the battery and another power source, such as one or more power sources external to the computing device 1000 and connected to the computing device 1000 via one or more external power interfaces. In other examples, some or all of the one or more batteries 1024 can be non-rechargeable batteries.

[0111] One or more external power interfaces 1026 of the power supply system 1022 can include one or more wired power interfaces, such as a USB cable and / or a power cord, that implement a wired power connection to one or more power sources external to the computing device 1000. One or more external power interfaces 1026 can include one or more wireless power interfaces, such as a Qi wireless charger, that implement a wireless power connection to one or more external power supply devices, such as a Qi wireless charger. Once a power connection to an external power source is established using one or more external power interfaces 1026, the computing device 1000 can draw power from the external power source of the established power connection. In some examples, the power system 1022 can include associated sensors, such as battery sensors associated with one or more batteries or other types of power sensors. Example operating methods

[0112] Figure 11 Method 1100 according to an example embodiment is shown. Method 1100 can include various blocks or steps. These blocks or steps can be performed individually or in combination. These blocks or steps can be performed in any order and / or serially or in parallel. Further, blocks or steps can be omitted or added to method 1100.

[0113] The blocks of method 1100 can be performed by various elements of the computing device 1000, as shown and described with reference to Figure 10 shown and described.

[0114] Block 1110 includes displaying a zoomed preview of a scene captured by a camera system on a display screen of the camera system.

[0115] Frame 1120 includes determining phase detection autofocus (PDAF) depth estimation and time-of-flight (ToF) depth estimation for the scene.

[0116] Frame 1130 includes determining whether a foreground object in the zoomed preview is in focus for a ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimation and the ToF depth estimation.

[0117] Frame 1140 includes bypassing the PDAF mode and activating the ToF-based AF mode to focus on the foreground object based on determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode, where the PDAF mode includes focusing by the camera system based on the PDAF depth estimation, and where the ToF-based AF mode includes focusing by the camera system based on the ToF depth estimation.

[0118] Frame 1150 includes displaying, by the display screen, the focused foreground object as part of the zoomed preview of the scene based on the ToF-based AF mode.

[0119] Some embodiments relate to determining that a second foreground object in a second zoomed preview of a scene is not in focus for a ToF-based AF mode based on a second comparison of a second PDAF depth estimation and a second ToF depth estimation. Such embodiments relate to maintaining the PDAF mode and not activating the ToF-based AF mode based on determining that the second foreground object in the second zoomed preview of the scene is not in focus for the ToF-based AF mode, and where the display includes displaying the second zoomed preview based on the PDAF mode.

[0120] In some embodiments, the comparison of the PDAF depth estimation and the ToF depth estimation involves determining whether a differential depth estimation based on a difference between the PDAF depth estimation and the ToF depth estimation exceeds a depth threshold, and where determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode is based on determining that the differential depth estimation exceeds the depth threshold.

[0121] In some embodiments, receiving the PDAF depth estimation involves determining whether the PDAF depth estimation exceeds a PDAF confidence threshold, and where bypassing the PDAF mode is based on determining whether the PDAF depth estimation exceeds the PDAF confidence threshold.

[0122] Such embodiments relate to a second PDAF depth estimate that receives a scene-based second zoomed preview. Such embodiments relate to determining that the second PDAF depth estimate does not exceed a PDAF confidence threshold. Such embodiments also relate to maintaining the PDAF mode and not activating the ToF-based AF mode, and wherein the display includes displaying the second zoomed preview based on the PDAF mode.

[0123] In some embodiments, receiving a ToF depth estimate involves determining whether the ToF depth estimate exceeds a ToF confidence threshold, and wherein focusing in the ToF-based AF mode of the camera system is based on determining whether the ToF depth estimate exceeds the ToF confidence threshold.

[0124] Such embodiments relate to determining that the ToF depth estimate exceeds the ToF confidence threshold, and wherein focusing in the ToF-based AF mode of the camera system is based on a distance-to-position mapping based on the ToF depth estimate for a foreground object.

[0125] Some embodiments relate to determining that the ToF depth estimate does not exceed the ToF confidence threshold, and wherein focusing in the ToF-based AF mode of the camera system is based on a multi-grid contrast detection autofocus (CDAF) search based on the ToF depth estimate. In some embodiments, the multi-grid CDAF search is based on one or more of a spatial grid, one or more directions, or one or more spatial frequencies.

[0126] Such embodiments relate to a second ToF depth estimate that receives a scene-based second zoomed preview. Such embodiments relate to determining that the second ToF depth estimate does not exceed the ToF confidence threshold. Such embodiments also relate to maintaining the PDAF mode and not activating the ToF-based AF mode, and wherein the display includes displaying the second zoomed preview based on the PDAF mode.

[0127] Some embodiments relate to determining that the PDAF depth estimate exceeds the PDAF confidence threshold. Such embodiments relate to determining that the ToF depth estimate exceeds the ToF confidence threshold. Such embodiments also relate to determining whether the brightness intensity of the background exceeds a brightness threshold based on a scene-based zoomed preview, and wherein bypassing the PDAF mode is based on determining whether the brightness intensity of the background exceeds the brightness threshold.

[0128] Some embodiments relate to determining that the brightness intensity of the background exceeds the brightness threshold. Such embodiments relate to bypassing the PDAF mode and activating the ToF-based AF mode.

[0129] Some embodiments relate to determining that a second luminance intensity of a second background in a second zoomed preview does not exceed a luminance threshold. Such embodiments relate to maintaining the PDAF mode and not activating the ToF-based AF mode, and wherein the display includes displaying the second zoomed preview based on the PDAF mode.

[0130] Some embodiments relate to receiving an indication to disable the ToF-based AF mode by a user interface of a display screen. Such embodiments relate to maintaining the PDAF mode and not activating the ToF-based AF mode in response to the indication, and wherein the display includes displaying the second zoomed preview based on the PDAF mode.

[0131] Some embodiments relate to displaying, by a display screen, an initial preview of a scene captured by another camera system operating at another focal length greater than or equal to a threshold focal length. Such embodiments relate to detecting a zoom operation that causes a transition from the second camera system to the camera system. Some embodiments also relate to providing, by a user interface of the display screen, a selectable virtual object to receive an indication of whether to enable or disable the ToF-based AF mode. In such embodiments, the camera system is configured to provide an ultra-wide field of view (FOV), and wherein the other camera system is configured to provide a wide FOV.

[0132] For example, when the user moves to within less than 15 centimeters (cm) of an object, the camera system can automatically switch to an ultra-wide-angle (UWA) camera (e.g., limited to ). In some embodiments, a button representing the macro mode can appear and be highlighted in the user interface. If the button is pressed in the macro mode (e.g., at a distance less than 18 cm), the UWA camera can be disengaged and the camera system can revert to the main sensor. Additionally, for example, pressing the button (e.g., ) can disengage the macro mode and switch back to the normal UWA view. If the user moves away from the object (e.g., greater than 18 cm), the camera system automatically switches back to the main sensor.

[0133] In some embodiments, focusing of the camera system includes adjusting at least one lens of the camera system.

[0134] In some embodiments, focusing of the camera system includes determining an exposure time of the camera system based on a motion blur tolerance of the camera system.

[0135] In some embodiments, the camera system is a component of a mobile device.

[0136] Figure 12Shows the difference in image focusing with and without macro object focusing based on hybrid AF according to an example embodiment. Images 1200A and 1200B correspond to a camera system in which macro object focusing based on hybrid AF is activated. When switching to an ultra-wide-angle camera (e.g., from a main camera), the camera system is able to detect intensity peaks corresponding to close-up objects. However, as shown in Image 1200C, for a camera system in which macro object focusing based on hybrid AF is not activated, when switching to an ultra-wide-angle camera, the camera system focuses on the background and fails to focus on the foreground object.

[0137] The specific arrangements shown in the drawings should not be considered restrictive. It should be understood that other embodiments may include more or fewer of each element shown in a given drawing. Further, some of the elements shown may be combined or omitted. Still further, an illustrative embodiment may include elements not shown in the figures.

[0138] The steps or blocks representing the processing of information may correspond to circuits that can be configured to perform the specific logical functions described herein for the method or technique. Alternatively or additionally, the steps or blocks representing the processing of information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions that can be executed by a processor to implement the specific logical functions or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a disk, a hard disk drive, or other storage medium.

[0139] The computer-readable medium may also include non-transitory computer-readable media, such as computer-readable media that store data for a short period of time, such as register memory, processor cache, and random access memory (RAM). The computer-readable medium may also include non-transitory computer-readable media that store program code and / or data for a long term. Thus, the computer-readable medium may include auxiliary or persistent long-term storage devices, such as, for example, read-only memory (ROM), optical disks or magnetic disks, compact disc read-only memory (CD-ROM). The computer-readable medium may also be any other volatile or non-volatile storage system. The computer-readable medium may be considered, for example, a computer-readable storage medium or a tangible storage device.

[0140] Although various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various examples and embodiments disclosed are for illustrative purposes and are not intended to be limiting, where the true scope is indicated by the appended claims.

Claims

1. A computer-implemented method, comprising: displaying, on a display screen of a camera system, a zoomed preview of a scene captured by the camera system; determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; determining, based on a comparison of the PDAF depth estimate and the ToF depth estimate, whether a foreground object in the zoomed preview is in focus for a ToF-based AF mode of the camera system; based on determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode, bypassing the PDAF mode and activating the ToF-based AF mode to focus on the foreground object, wherein the PDAF mode includes focusing by the camera system based on the PDAF depth estimate, and wherein the ToF-based AF mode includes focusing by the camera system based on the ToF depth estimate; and displaying, on the display screen, the focused foreground object as part of the zoomed preview of the scene based on the ToF-based AF mode.

2. The method of claim 1, further comprising: determining, based on a second comparison of a second PDAF depth estimate and a second ToF depth estimate, that a second foreground object in a second zoomed preview of the scene is not in focus for the ToF-based AF mode; and based on determining that the second foreground object in the second zoomed preview of the scene is not in focus for the ToF-based AF mode, maintaining the PDAF mode and not activating the ToF-based AF mode, and wherein the displaying includes displaying the second zoomed preview based on the PDAF mode.

3. The method of claim 1, wherein the comparison of the PDAF depth estimate and the ToF depth estimate comprises: determining whether a differential depth estimate based on a difference between the PDAF depth estimate and the ToF depth estimate exceeds a depth threshold, and wherein determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode is based on determining that the differential depth estimate exceeds the depth threshold.

4. The method of claim 1, wherein receiving the PDAF depth estimate further comprises: determining whether the PDAF depth estimate exceeds a PDAF confidence threshold, and wherein bypassing the PDAF mode is based on determining whether the PDAF depth estimate exceeds the PDAF confidence threshold.

5. The method of claim 4, further comprising: receiving a second PDAF depth estimate based on a second zoomed preview of the scene; determining that the second PDAF depth estimate does not exceed the PDAF confidence threshold; and maintaining the PDAF mode and not activating the ToF-based AF mode, and wherein the displaying includes displaying the second zoomed preview based on the PDAF mode.

6. The method of claim 1, wherein receiving the ToF depth estimate further comprises: Determine whether the ToF depth estimate exceeds a ToF confidence threshold, and wherein the focusing of the camera system in the ToF-based AF mode is based on determining whether the ToF depth estimate exceeds the ToF confidence threshold.

7. The method according to claim 6, further comprising: Determine that the ToF depth estimate exceeds the ToF confidence threshold, and wherein the focusing of the camera system in the ToF-based AF mode is based on a distance-to-position mapping based on the ToF depth estimate for the foreground object.

8. The method according to claim 6, further comprising: Determine that the ToF depth estimate does not exceed the ToF confidence threshold, and wherein the focusing of the camera system in the ToF-based AF mode is based on a multi-grid contrast detection autofocus (CDAF) search based on the ToF depth estimate.

9. The method according to claim 8, wherein the multi-grid CDAF search is based on one or more of a spatial grid, one or more directions, or one or more spatial frequencies.

10. The method according to claim 1, further comprising: Receiving a second ToF depth estimate of a second zoomed preview based on the scene; Determine that the second ToF depth estimate does not exceed the ToF confidence threshold; And Maintain the PDAF mode and do not activate the ToF-based AF mode, and wherein the display includes displaying the second zoomed preview based on the PDAF mode.

11. The method according to claim 1, further comprising: Determine that the PDAF depth estimate exceeds the PDAF confidence threshold; Determine that the ToF depth estimate exceeds the ToF confidence threshold; Determine whether the brightness intensity of the background exceeds a brightness threshold based on the zoomed preview of the scene, and wherein bypassing the PDAF mode is based on determining whether the brightness intensity of the background exceeds the brightness threshold.

12. The method according to claim 11, further comprising: Determine that the brightness intensity of the background exceeds the brightness threshold; And Bypass the PDAF mode and activate the ToF-based AF mode.

13. The method according to claim 11, further comprising: Determine that the second brightness intensity of the second background in the second zoomed preview does not exceed the brightness threshold; And Maintain the PDAF mode and do not activate the ToF-based AF mode, and wherein the display includes displaying the second zoomed preview based on the PDAF mode.

14. The method according to claim 1, further comprising: Receiving an indication to disable the ToF-based AF mode from a user interface of the display screen; And In response to the indication, maintain the PDAF mode and do not activate the ToF-based AF mode, and wherein the display includes displaying a second zoomed preview based on the PDAF mode.

15. The method according to claim 1, further comprising: Display an initial preview of the scene being captured by another camera system on the display screen; Detect a zoom operation that causes a transition from the second camera system to the camera system; And Provide a selectable virtual object via a user interface of the display screen to receive an indication of whether to enable or disable the ToF-based AF mode.

16. The method according to claim 15, wherein the camera system is configured to provide an ultra-wide field of view (FOV), and wherein the other camera system is configured to provide a wide FOV.

17. The method according to claim 1, wherein the focusing of the camera system includes adjusting at least one lens of the camera system.

18. The method according to claim 1, wherein the focusing of the camera system includes determining an exposure time of the camera system based on a motion blur tolerance of the camera system.

19. The method according to claim 1, wherein the camera system is a component of a mobile device.

20. A computing device, comprising: A display screen; A camera system configured to operate at a focal length less than a threshold focal length; One or more processors; And A data storage device, wherein computer-executable instructions are stored on the data storage device, and when executed by the one or more processors, the computer-executable instructions cause the mobile device to perform functions, the functions including: Display a zoomed preview of a scene captured by the camera system on the display screen of the camera system; Determine a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; Determine whether a foreground object in the zoomed preview is in focus for the ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate; Based on determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode, bypass the PDAF mode and activate the ToF-based AF mode to focus on the foreground object, wherein the PDAF mode includes focusing of the camera system based on the PDAF depth estimate, and wherein the ToF-based AF mode includes focusing of the camera system based on the ToF depth estimate; and Display the focused foreground object as part of the zoomed preview of the scene on the display screen based on the ToF-based AF mode.

21. A non-transitory computer-readable medium, comprising program instructions that can be executed by one or more processors to cause the one or more processors to perform operations, the operations including: Display a zoomed preview of a scene captured by a camera system on a display screen of the camera system; Determine a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; Determine whether a foreground object in the zoomed preview is in focus for the ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate; Based on determining that the foreground object in the zoomed preview is in focus for the ToF-based AF mode, bypass the PDAF mode and activate the ToF-based AF mode to focus on the foreground object, where the PDAF mode includes focusing by the camera system based on the PDAF depth estimate, and where the ToF-based AF mode includes focusing by the camera system based on the ToF depth estimate; And Based on the ToF-based AF mode, display the focused foreground object by the display screen as part of the zoomed preview of the scene.

Images