Hybrid autofocus system with robust macro-object priority focus

Abstract

An exemplary method includes displaying a zoom preview of a scene captured by a camera system. The method includes determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene. The method includes determining whether a foreground object in the zoom preview is in focus for a ToF-based AF mode based on a comparison of the PDAF and ToF depth estimates. The method includes bypassing the PDAF mode and activating a ToF-based AF mode to focus on the foreground object based on a determination that the foreground object in the zoom preview is in focus for the ToF-based AF mode. The method includes displaying the in-focus foreground object as part of the zoom preview of the scene based on the ToF-based AF mode.

Description

[Technical Field]

[0001] CROSS-REFERENCE / INCORPORATION BY REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 378,652, filed October 6, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Many modern computing devices, including mobile phones, personal computers, and tablets, include image capture devices. Some image capture devices are configured with multiple camera systems. The camera systems are configured to cooperatively meet different image capture requirements using their respective specifications. Smartphones can integrate multiple types of cameras with various focal lengths to address objects at different distances and scenes within different fields of view (FOV). Summary of the Invention

[0003]

[0001] The present disclosure generally relates to transitions between multiple cameras. In one aspect, an image capture device may include multiple cameras. A transition from normal mode to macro mode may result in a perceptible loss of focus of a target close-up object. As described herein, a prioritized hybrid autofocus strategy is described, which reduces the perceptible loss of focus after switching cameras (e.g., to an ultra-wide-angle camera).

[0004] In a first aspect, a computer-implemented method is provided. The method includes displaying, via a display screen of the camera system, a zoom preview of a scene captured by the camera system. The method includes determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene. The method includes determining whether a foreground object in the zoom preview is in focus for a ToF-based autofocus (AF) mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate. The method includes bypassing the PDAF mode and activating a ToF-based AF mode to focus on the foreground object based on a determination that the foreground object in the zoom preview is in focus for the ToF-based AF mode, the PDAF mode including focusing the camera system based on the PDAF depth estimate, and the ToF-based AF mode including focusing the camera system based on the ToF depth estimate. The method includes displaying, via the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoom preview of the scene.

[0005] In a second aspect, a computing device is provided, the computing device including a display screen, a camera system configured to operate at a focal length less than a threshold focal length, one or more processors, and data storage, the data storage storing computer-executable instructions that, when executed by the one or more processors, cause the mobile device to perform functions. The operations include displaying, by a display screen of the camera system, a zoom preview of a scene captured by the camera system; receiving a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene based on the zoom preview of the scene; determining whether a foreground object in the zoom preview is in focus for a ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate; and based on a determination that the foreground object in the zoom preview is in focus for the ToF-based AF mode, bypassing the PDAF mode and activating a ToF-based AF mode to focus on the foreground object, the PDAF mode including focusing the camera system based on the PDAF depth estimate and the ToF-based AF mode including focusing the camera system based on the ToF depth estimate; and the operations further include displaying, by the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoom preview of the scene.

[0006] In a third aspect, an article of manufacture is provided. The article of manufacture may include a non-transitory computer-readable medium containing program instructions executable by one or more processors, the program instructions causing the one or more processors to perform operations. The operations include displaying, by a display screen of the camera system, a zoom preview of a scene captured by the camera system; determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; determining whether a foreground object in the zoom preview is in focus for a ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate; and based on a determination that the foreground object in the zoom preview is in focus for the ToF-based AF mode, bypassing the PDAF mode and activating a ToF-based AF mode to focus on the foreground object, the PDAF mode including focusing the camera system based on the PDAF depth estimate and the ToF-based AF mode including focusing the camera system based on the ToF depth estimate; and the operations further include displaying, by the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoom preview of the scene.

[0007] In a fourth aspect, a system is provided, including: means for displaying, by a display screen of the camera system, a zoom preview of a scene captured by the camera system; means for determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; means for determining whether a foreground object in the zoom preview is in focus for a ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate; and means for bypassing the PDAF mode and activating a ToF-based AF mode to focus on the foreground object based on a determination that the foreground object in the zoom preview is in focus for the ToF-based AF mode, the PDAF mode including focusing the camera system based on the PDAF depth estimate and the ToF-based AF mode including focusing the camera system based on the ToF depth estimate; and means for displaying, by the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoom preview of the scene.

[0008] Other aspects, embodiments, and implementations will become apparent to those skilled in the art from a reading of the following detailed description, with appropriate reference to the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] 1A-1C are illustrations of front, right, and back views of a digital camera device according to an exemplary embodiment. [Figure 2] 10 is a diagram of a preview frame displaying user-friendly mode switching options, according to an exemplary embodiment. [Figure 3] 1A-1C are diagrams of example images with and without hybrid AF-based macro-object focusing, in accordance with an exemplary embodiment; [Figure 4]1A-1C are diagrams of example images with and without hybrid AF-based macro-object focusing, in accordance with an exemplary embodiment; [Figure 5] 1A-1C are diagrams of example images with and without hybrid AF-based macro-object focusing, in accordance with an exemplary embodiment; [Figure 6] 1A-1C are diagrams of example images with and without hybrid AF-based macro-object focusing, in accordance with an exemplary embodiment; [Figure 7] 1 is an exemplary workflow of a hybrid autofocus system with robust macro-object-first focusing, according to an exemplary embodiment. [Figure 8A] 10 is another exemplary workflow of a hybrid autofocus system with robust macro-object-first focusing, according to an exemplary embodiment. [Figure 8B] FIG. 1 is an illustration of a multi-grid contrast detection autofocus (CDAF) analysis, according to an example embodiment. [Figure 9] 1 illustrates a distributed computing architecture in accordance with an exemplary embodiment. [Figure 10] FIG. 1 is a block diagram of a computing device in accordance with an exemplary embodiment. [Figure 11] 1 is a flowchart of a method according to an example embodiment. [Figure 12] 10 illustrates the difference between image focusing with and without hybrid AF-based macro-object focusing, according to an example embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Exemplary methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as an "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or features. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0011] Accordingly, the exemplary embodiments described herein are not intended to be limiting. The aspects of the present disclosure, as generally described and illustrated in the Figures herein, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0012] Furthermore, unless the context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be viewed generally as component aspects of one or more overall embodiments, with the understanding that not all of the illustrated features are required for each embodiment.

[0013] overview A smartphone or other mobile device that supports image and / or video capture may include multiple cameras that use different specifications to cooperatively meet different image capture requirements. A smartphone may integrate multiple types of cameras with various focal lengths to view and / or capture objects at different distances and scenes within different fields of view (FOVs).

[0014] A camera is a device used to capture images of a scene. Some cameras (e.g., film cameras) capture images chemically on film. Other cameras (e.g., digital cameras) capture image data electrically (e.g., using a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) sensor). To most accurately capture a scene, the camera may be focused on one or more objects within the scene. There are multiple ways to focus a camera. For example, the camera's lens can be moved relative to the camera's image sensor to adjust the camera's focus (e.g., to focus on one or more objects). Similarly, the camera's image sensor can be moved relative to the camera's lens to adjust the camera's focus.

[0015] Adjusting the focus of the camera can be performed manually (e.g., by a photographer). Alternatively, an autofocus procedure can be performed to adjust the focus of the camera before capturing an image (e.g., a payload image). The autofocus procedure may use one or more images (captured by either the camera's primary image sensor or one or more auxiliary sensors in the camera) to determine an appropriate focus setting for the camera. Then, based on the determined focus setting, the camera adjusts to meet that focus setting. For example, motors may adjust the relative position of the lens and / or image sensor to meet the determined focus setting.

[0016] There are two conventional types of autofocus procedures: active autofocus procedures and passive autofocus procedures.

[0017] In an active autofocus procedure, a rangefinder (e.g., a laser rangefinder, a radar device, or a sonar device) is used to determine the distance to one or more objects in a scene. Then, based on the determined distance, a focus setting is determined and the camera is adjusted to meet the determined focus setting.

[0018] There are two main types of passive autofocus procedures: phase detection autofocus and contrast detection autofocus.

[0019] In phase-detection autofocus (PDAF), incident light from a scene is split (e.g., by a beam splitter) so that light from the scene entering one side of a camera's lens is physically separated at an image sensor (e.g., the camera's primary or auxiliary image sensor) from light from the scene entering the other side of the lens. A focus setting can be determined based on the camera's characteristics (e.g., lens size, lens focal length, and position of the lens relative to the image sensor) and the light intensity distribution across various locations on the image sensor. Similar to an active autofocus procedure, the camera can be adjusted to meet the determined focus setting.

[0020] In contrast detection autofocus (CDAF), a series of frames are captured by a camera at a corresponding series of different focus settings. The contrast between high and low luminance is then determined for each of the captured frames. Based on the determined contrast, a focus setting is determined (e.g., based on the frame with the highest contrast and / or based on a regression analysis using the contrast of the captured frames). Similar to active autofocus procedures and phase detection autofocus, the camera can be adjusted to meet the determined focus setting.

[0021] Unlike active autofocus procedures, passive autofocus procedures (e.g., phase-detection autofocus and contrast-detection autofocus) do not use additional ranging equipment. Therefore, passive autofocus procedures may be employed in camera systems (e.g., in mobile phones or digital single-lens reflex (DSLR) cameras) to save costs. However, passive autofocus procedures may be less successful in low-light conditions (e.g., because insufficient contrast occurs between frames to be used for contrast-detection autofocus, or because there are not enough bright objects in the scene for comparison when using phase-detection autofocus).

[0022] A mobile phone may be configured with a main camera with a moderate focal length to meet normal photo / video capture requirements, a telephoto camera with a long focal length to capture distant objects, and a wide-angle or ultra-wide-angle camera with a short focal length to capture a larger FOV. During a photo / video capture session, a switch from the main camera to the telephoto camera may occur as the user continues to zoom in to focus on distant objects, and a switch from the main camera to the ultra-wide-angle camera may occur as the user continues to zoom out to capture a larger field of view. Multi-camera systems offer a much larger focal length range than a single camera. However, when switching from the wide-angle camera to the ultra-wide-angle camera, it may be difficult to focus on foreground objects, especially against a high-contrast background.

[0023] For example, with smartphone cameras that have a macro mode feature, users generally expect to be able to focus closely on small objects. Close-ups of objects often have high-contrast backgrounds. In scenes like these, using a conventional hybrid autofocus (AF) scheme tends to adjust the back focus and may not automatically achieve the desired focus point for the user.

[0024] Generally, it is desirable for each image captured by a user to be detailed, in focus, and worthy of saving and sharing. It is also desirable for the user to have control over when they want to enable this feature, which would be beneficial to the user's camera experience. Therefore, automatically switching cameras at the appropriate time, dynamically selecting the appropriate lens for the user to obtain the best image quality, providing helpful prompts to the user, and enhancing the image after capture are important aspects of this feature to provide optimal functionality. Also, for example, conventional camera systems typically have macro modes deeply integrated into their hardware configurations, making these features available to advanced users, such as professional photographers. Therefore, making macro mode available in a user-friendly manner is another aspect of the procedures described herein.

[0025] To make this process user-friendly, a hybrid autofocus strategy that prioritizes macro objects may be deployed. For example, a conventional hybrid AF strategy hierarchy includes applying a phase-detection autofocus (PDAF) algorithm, followed by a time-of-flight (ToF)-based algorithm, and then a contrast-detection autofocus (CDAF) algorithm. However, despite the high reliability of the PDAF-based depth estimate, the PDAF algorithm may cause the camera lens to focus on the background, leaving the foreground object out of focus. Therefore, as described herein, it is necessary to override the PDAF algorithm so that the foreground object can be sharply focused.

[0026] In addition to focusing on foreground objects, the techniques described herein enable the capture of objects as close as 3 cm away. When combined with HDR+ quality processing, minute objects such as raindrops, individual petals, and pollen grains can be sharply focused. For example, the techniques described herein enable the capture of objects such as plants, pets, insects, human eyes, animal eyes, feathers, and small living organisms such as mushrooms. The techniques described herein also enable sharper image capture of unique textures such as jeans, leather, cotton, fabrics of all kinds, stone, brick, rough surfaces, smooth surfaces, rust, paint, tissue paper, mouse pads, tin foil, ice cubes, foam, and bubbles. Natural objects such as fruits, vegetables, water droplets, trees, moss, grass, snowflakes, shells, and seeds can also be sharply focused. In general, any object that may have a unique appearance and / or reveal new image information when viewed up close may be sharply focused. Such objects may include, for example, coins, crayon tips, pen tips, matches, needles, Q-tips, musical instruments, handwriting, paper, fingerprints, buttons, jewelry, floor tiles, etc. The macro mode with the ultra-wide camera can be seamlessly integrated with other camera systems providing zoom ratios ranging from 0.5x to 30x.

[0027] Exemplary Camera System As image capture devices such as cameras become more prevalent, they may be used as standalone hardware devices or integrated into various other types of devices. For example, still and video cameras are now commonly included in wireless computing devices (e.g., mobile devices such as mobile phones), tablet computers, laptop computers, video game interfaces, home automation devices, and even automobiles and other types of vehicles.

[0028] The physical components of a camera may include one or more apertures through which light enters, one or more recording surfaces for capturing images represented by the light, and a lens positioned in front of each aperture to focus at least a portion of the image onto the recording surface(s). The apertures may be fixed size or adjustable. In an analog camera, the recording surface may be photographic film. In a digital camera, the recording surface may include an electronic image sensor (e.g., a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) sensor) for transferring and / or storing the captured image in a data storage unit (e.g., memory).

[0029] One or more shutters may be coupled to or near the lens or recording surface. Each shutter may be in either a closed position, blocking light from reaching the recording surface, or an open position, allowing light to reach the recording surface. The position of each shutter may be controlled by a shutter button. For example, the shutter may be in a closed position by default. When the shutter button is triggered (e.g., pressed), the shutter may change from the closed position to the open position for a period known as a shutter cycle. During the shutter cycle, an image may be captured on the recording surface. At the end of the shutter cycle, the shutter may change back to the closed position.

[0030] Alternatively, the shuttering process may be electronic. For example, before the electronic shutter of a CCD image sensor "opens," the sensor may be reset to remove any residual signal from its photodiodes. While the electronic shutter remains open, the photodiodes may accumulate charge. When or after the shutter closes, these charges may be transferred to longer-term data storage. A combination of mechanical and electronic shutters may also be possible.

[0031] Regardless of type, the shutter may be activated and / or controlled by something other than a shutter button. For example, the shutter may be activated by a soft key, a timer, or some other trigger. As used herein, the term "image capture" may refer to any mechanical and / or electronic shutter process that results in one or more images being recorded, regardless of how the shutter process is triggered or controlled.

[0032] The exposure of a captured image may be determined by a combination of the size of the aperture, the brightness of the light entering the aperture, and the length of the shutter cycle (also referred to as shutter length, exposure length, or exposure time). Additionally, digital and / or analog gain (e.g., based on ISO settings) may be applied to the image to affect exposure. In some embodiments, the terms "exposure length," "exposure time," or "exposure time interval" may refer to the shutter length multiplied by the gain for a particular aperture size. Thus, these terms may be used somewhat interchangeably and, in some cases, should be interpreted as shutter length, exposure time, and / or any other metric that controls the amount of signal response due to light reaching the recording surface.

[0033] In some implementations or modes of operation, the camera may capture one or more still images each time image capture is triggered. In other implementations or modes of operation, the camera may capture video images by continuously capturing images at a particular rate (e.g., 24 frames per second) for as long as image capture remains triggered (e.g., while the shutter button is pressed). When operating in a mode that captures still images, some cameras may open the shutter when the camera device or application becomes active, and the shutter may remain in this position until the camera device or application becomes inactive. While the shutter is open, the camera device or application may capture and display a representation of the scene on the viewfinder (sometimes referred to as displaying a "preview frame"). When image capture is triggered, one or more separate payload images of the current scene may be captured.

[0034] Cameras, including digital and analog cameras, may include software for controlling one or more camera functions and / or settings, such as aperture size, exposure time, gain, etc. Additionally, some cameras may include software that digitally processes images during or after image capture. While the above description refers generally to cameras, it may be particularly relevant to digital cameras. Digital cameras may be standalone devices (e.g., DSLR cameras) or may be integrated with other devices.

[0035] FIG. 1 illustrates front, right, and back views of a digital camera device 100 according to an exemplary embodiment. Digital camera device 100 may be, for example, a mobile device (e.g., a mobile phone), a tablet computer, or a wearable computing device. However, other embodiments are possible. Digital camera device 100 may include various elements, such as a body 102, a front-facing camera 104, a multi-element display 106, a shutter button 108, and other buttons 110. Digital camera device 100 may further include a rear-facing camera 112. Front-facing camera 104 may be positioned on the side of body 102 that typically faces the user during operation or on the same side as multi-element display 106. Rear-facing camera 112 may be positioned on the side of body 102 opposite front-facing camera 104. Referring to cameras as front and back is arbitrary, and digital camera device 100 may include multiple cameras positioned on various sides of body 102.

[0036] Multi-element display 106 may represent a cathode ray tube (CRT) display, a light-emitting diode (LED) display, a liquid crystal display (LCD), a plasma display, or any other type of display known in the art. In some embodiments, multi-element display 106 may display a digital representation of the current image being captured by front-facing camera 104 and / or rear-facing camera 112, or an image that may be captured or was recently captured by one or both of these cameras. Thus, multi-element display 106 may function as a viewfinder for either camera. Multi-element display 106 may also support touchscreen and / or presence sensing capabilities that allow settings and / or configurations of any aspect of digital camera device 100 to be adjusted.

[0037] The front-facing camera 104 may include an image sensor and associated optical elements, such as a lens. The front-facing camera 104 may provide zoom capabilities or have a fixed focal length. In other embodiments, interchangeable lenses may be used by the front-facing camera 104. The front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. The front-facing camera 104 may also be configured to capture still images, video images, or both. Furthermore, the front-facing camera 104 may represent a monoscopic, stereoscopic, or multiscopic camera. The rear-facing camera 112 may be similarly or differently positioned. Furthermore, the front-facing camera 104, the rear-facing camera 112, or both, may be an array of one or more cameras.

[0038] One or both of the front camera 104 and the rear camera 112 may include or be associated with an illumination component that provides a bright field that illuminates the target object. For example, the illumination component may provide flash or constant illumination of the target object (e.g., using one or more LEDs). The illumination component may also be configured to provide a bright field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of bright fields known and used to recover three-dimensional (3D) models from objects are possible within the context of the embodiments herein.

[0039] One or both of the front camera 104 and the rear camera 112 may include or be associated with an ambient light sensor that may continuously or occasionally determine the ambient brightness of a scene that the camera may capture. In some devices, the ambient light sensor may be used to adjust the display brightness of a screen associated with the camera (e.g., a viewfinder). If the determined ambient brightness is high, the screen brightness level may be increased to make the screen easier to view. If the determined ambient brightness is low, the screen brightness level may be decreased not only to make the screen easier to view, but also potentially to save power. Additionally, the ambient light sensor input may be used to determine or assist in determining the exposure time of the associated camera.

[0040] Digital camera device 100 may be configured to capture images of a target object (i.e., a subject in a scene) using multi-element display 106 and either front-facing camera 104 or rear-facing camera 112. The captured images may be multiple still images or video images (e.g., a series of still images captured in rapid succession with or without accompanying audio captured by a microphone). Image capture may be triggered by activating shutter button 108, pressing a soft key on multi-element display 106, or some other mechanism. Depending on the implementation, images may be captured automatically at specific time intervals, for example, upon pressing shutter button 108, in suitable lighting conditions for the target object, upon moving digital camera device 100 a predetermined distance, or according to a predetermined capture schedule.

[0041] As noted above, the functionality of digital camera device 100 (or other types of digital cameras) may be integrated into a computing device such as a wireless computing device, a mobile phone, a tablet computer, a laptop computer, etc. For example, a camera controller may be integrated with digital camera device 100 to control one or more functions of digital camera device 100.

[0042] 2 is an illustration of a preview frame 202 that displays user-friendly mode switching options, according to an example embodiment. The preview frame 202 can display a captured frame to a user based on a current scene captured using current camera system settings (e.g., aperture settings, exposure settings, etc.). The techniques described herein can be used when a preview frame appears similar to the preview frame 202 of FIG. 2.

[0043] In some embodiments, the hybrid autofocus procedure described herein may be triggered when a previous autofocus attempt (e.g., based on a conventional PDAF algorithm) was unsuccessful. For example, the preview frame 202 shown in FIG. 2 may have insufficient focus for the payload image, so the hybrid autofocus procedure may be performed. Whether the previous autofocus attempt was unsuccessful may be based on an indication from a user that the previous autofocus attempt was insufficient. In other embodiments, the hybrid autofocus algorithm (e.g., a PDAF algorithm used for preview mode) may provide an indication that the autofocus failed. For example, the autofocus algorithm may provide a PDAF confidence value indicating the probability that the autofocus was successful, and may determine that the autofocus failed if the confidence value is below a certain threshold (e.g., a PDAF confidence threshold). The indication that the autofocus failed may be provided by an API (e.g., an API of a camera module of a mobile device).

[0044] Whether hybrid autofocus is successful may be based on a hybrid autofocus algorithm (e.g., a time-of-flight (ToF) algorithm used in preview mode), which may provide an indication that autofocus was successful. For example, the autofocus algorithm may provide a ToF confidence value indicating the probability that autofocus was successful, and may determine that autofocus was successful if the confidence value exceeds a certain threshold (e.g., a ToF confidence threshold). The indication that autofocus was successful may be provided by an API (e.g., an API of a camera module of a mobile device).

[0045] For example, a selectable virtual object can be provided to a user (e.g., during a camera transition from the main camera to the ultra-wide camera or during operation of the ultra-wide camera) to indicate whether to enable or disable the hybrid autofocus mode described herein. For example, a toggle switch can be displayed on multi-element display 106 of digital camera device 100 to enable or disable the hybrid autofocus mode.

[0046] FIG. 3 is a diagram of an example of an image with and without hybrid AF-based macro object focusing, according to an exemplary embodiment. FIG. 3 shares one or more aspects in common with FIGS. 1 and 2. Digital camera device 300A illustrates an image in which a conventional PDAF approach is used to capture the image. As shown, one or more foreground objects may be out of focus. Digital camera device 300B illustrates a situation in which hybrid AF-based macro object focusing is used. Accordingly, the algorithms described with reference to FIGS. 7 and 8 may be applied to focus on one or more foreground objects. For example, a ToF-based AF mode may be applied (e.g., a ToF distance-based multi-grid, multi-directional, multi-frequency CDAF scan, described with reference to block 830 of FIG. 8).

[0047] FIG. 4 is a diagram of example images with and without hybrid AF-based macro object focusing, according to an exemplary embodiment. FIG. 4 shares one or more aspects in common with FIGS. 1-3. Digital camera device 400A illustrates an image in which a conventional PDAF approach is used to capture the image. As shown, one or more foreground objects may be out of focus. Digital camera device 400B illustrates a situation in which hybrid AF-based macro object focusing is used to capture the image. Accordingly, the algorithms described with reference to FIGS. 7 and 8 may be applied to focus on one or more foreground objects. For example, a ToF-based AF mode may be applied (e.g., a ToF distance-based multi-grid, multi-directional, multi-frequency CDAF scan, described with reference to block 830 of FIG. 8).

[0048] FIG. 5 is a diagram of example images with and without hybrid AF-based macro object focusing, according to an exemplary embodiment. FIG. 5 shares one or more aspects in common with FIGS. 1-4. Digital camera device 500A illustrates an image in which a conventional PDAF approach is used to capture the image. As shown, one or more foreground objects may be out of focus. Digital camera device 500B illustrates a situation in which hybrid AF-based macro object focusing is used to capture the image. Accordingly, the algorithms described with reference to FIGS. 7 and 8 may be applied to focus on one or more foreground objects. For example, a ToF-based AF mode may be applied (e.g., a ToF distance-based multi-grid, multi-directional, multi-frequency CDAF scan, described with reference to block 830 of FIG. 8).

[0049] FIG. 6 is a diagram of an example of images with and without hybrid AF-based macro object focusing, according to an exemplary embodiment. FIG. 6 shares one or more aspects in common with FIGS. 1-5. Digital camera device 600A illustrates an image in which a conventional PDAF approach is used to capture the image. As shown, one or more foreground objects may be out of focus. Digital camera device 600B illustrates a situation in which hybrid AF-based macro object focusing is used to capture the image. Accordingly, the algorithms described with reference to FIGS. 7 and 8 may be applied to focus on one or more foreground objects. For example, a ToF-based AF mode may be applied (e.g., a ToF distance-based multi-grid, multi-directional, multi-frequency CDAF scan, described with reference to block 830 of FIG. 8).

[0050] Hybrid AF algorithm for macro mode Conventional hybrid AF schemes prioritize PDAF->TOF->CDAF based on PDAF confidence. PDAF is an efficient method for continuous camera focusing because it relies on disparity information derived from the image sensor and directly controls the lens to optimize the blur circle of the AF region of interest (ROI) projected onto the image sensor. In situations where PDAF is reliable, PDAF is used to achieve focus. In some cases, such as low-light conditions, PDAF disparity estimation can fail due to noise from lower SNRs. In these situations, ToF can be a good complement to use for focusing, using a metric depth estimate of the scene converted to focus lens position via a depth-to-position mapping. However, this mapping can have accuracy issues, so it may be followed up with some kind of contrast scanning (e.g., CDAF) to fill the accuracy gap. Finally, if ToF is unreliable or not feasible, the AF system relies on CDAF scanning as a last resort. Generally, CDAF requires a focus scan and is not optimal for fast-moving subjects.

[0051] Therefore, in a conventional hybrid AF system, if PDAF is reliable, it is used directly to focus, and the decision to set the focus point is not overridden by other available focus point data. However, this scheme may not be optimal for macro mode with small and / or thin objects against a high-contrast background, leading to undesirable back focus. Also, for example, the image quality based on the PDAF mode may not be sufficient to display nearby objects. For example, sparse PDAF may not be able to detect nearby objects (e.g., foreground objects).

[0052] Phase detection autofocus is a passive autofocus technique that attempts to determine the appropriate focus setting (e.g., lens position and / or image sensor position) of the camera system based on the objects in the surrounding environmental scene that will ultimately be captured in the payload image. Phase detection autofocus works by splitting the light entering the camera system into two or more portions. These portions can be captured and then compared to each other. The two or more portions are compared to determine the relative positions of the peaks and valleys in brightness across each frame. If the relative positions within the frame match, the object(s) in the scene are in focus. If the relative positions do not match, the object(s) in the scene are out of focus. Based on the distance between each peak and each valley and the position of the optics (e.g., lens, image sensor(s), etc.) within the camera system, adjustments can be determined to move the object(s) into focus.

[0053] Further, in some embodiments, one or more objects in a scene may be in focus, while other objects remain out of focus. Thus, determining whether a scene is out of focus may include selecting one or more objects in the scene on which to make the determination. The region of interest for the focus determination may be selected based on a user (e.g., of a mobile device). For example, the user may select an object in the preview frame on which the user wishes to focus (e.g., a building, a person, a person's face, a car, etc.). Alternatively, an object identification algorithm may be used to determine what types of objects are in the scene and determine which of the objects need to be focused on based on a ranked list by importance (e.g., if a person's face is in the scene, that is the object to focus on, followed by a dog, then a building, etc.). In still other embodiments, whether a scene is in focus or out of focus may include identifying whether a moving object in the scene (e.g., as determined by the preview frame) is in focus or out of focus. Alternatively, determining the "in focus" camera setting may include determining a lens setting that focuses on the largest area of ​​the frame (e.g., by pixel area), or the largest number of objects in the frame (e.g., one distinct object, two distinct objects, three distinct objects, four distinct objects, etc.).

[0054] Generally, ToF focusing is an active autofocus technique, whereby a camera can measure target distance by actively illuminating an object. The illumination can be performed using a light source such as an LED or a laser. Light reflected by the object is captured by a ToF sensor. Generally, the ToF sensor is configured to be sensitive to different wavelengths, and the ToF sensor can measure the time delay as light reflects back to the sensor, and a ToF depth estimate can be determined based on this time delay. For example, the time delay ΔT is generally proportional to twice the distance from the camera to the object, which corresponds to the round-trip distance for light to leave the camera, be reflected, and return to the camera. Therefore, a ToF depth estimate d can be determined as d=kΔT / 2, where k is a proportionality constant.

[0055] As described herein, a hybrid AF macro-object priority scheme prioritizes a ToF-based AF mode over a PDAF mode to help users focus on nearby objects in macro mode. This approach also applies when the scene brightness level is above a threshold level, the PDAF estimate is reasonable, and there is confidence in the PDAF estimate. In these cases, conventional camera systems use the PDAF mode to achieve focus.

[0056] However, in some embodiments, ToF focusing may have reduced ranging accuracy in bright light conditions, and there may be spatial parallax shifts in the camera FOV at closer object distances. Therefore, if the TOF depth estimate is not sufficiently reliable to focus on its own, a ToF-based AF mode may be configured to overcome ambiguities in the focus data available to conventional AF algorithms by constraining the focus scan around the TOF-estimated focus position with a multi-grid CDAF search to help pinpoint the focus position.

[0057] 7 is an example workflow 700 of a hybrid autofocus system with robust macro-object-priority focusing, according to an example embodiment. As shown, a PDAF bypass decision module 705 may be initiated. For example, this may be initiated based on a user indication or may be automatically determined based on camera sensor data (e.g., invoked by hybrid AF macro-object-priority module 805 of FIG. 8). As used herein, the term "ToF-based AF mode" generally refers to the operations performed in step 825 and / or step 830 of FIG. 8. Also, for example, as used herein, the term "PDAF mode" generally refers to a mode that performs a conventional PDAF algorithm.

[0058] In step 710, the process includes determining whether the PDAF depth estimate is reasonable and whether the PDAF depth estimate exceeds a PDAF confidence threshold. Generally, PDAF pixels work by capturing two slightly different views of a scene. The parallax effect can be used to estimate PDAF depth when the background moves horizontally but the object remains stationary. For example, parallax is a function of the distance of the point from the camera and the distance between the two viewpoints. Therefore, a PDAF depth estimate can be performed by matching each point in one view with its corresponding point in the other view. However, because scene points move very little between views, finding these correspondences in the PDAF image (i.e., determining depth from stereo) can be a challenging task. Also, for example, stereo techniques can suffer from aperture issues (i.e., when a scene is viewed through a small aperture, it is not always possible to find correspondences for lines parallel to the stereo baseline, i.e., the lines connecting the two cameras). Therefore, the PDAF estimate may be erroneous.

[0059] Upon determining that the PDAF depth estimate is not valid (i.e., erroneous) or that the PDAF depth estimate fails to exceed the PDAF confidence threshold, the process proceeds to step 715, where the "Bypass PDAF" parameter is set to "False" indicating that the PDAF mode remains active (e.g., PDAF mode is maintained and not bypassed) and the ToF-based AF mode is not activated.

[0060] Upon determining that the PDAF depth estimate exceeds the PDAF confidence threshold, the process proceeds to step 720 .

[0061] In step 720, the process includes comparing the PDAF depth estimate and the ToF depth estimate to determine whether the foreground object in the zoom preview is in focus in the ToF-based AF mode of the camera system (and likely out of focus in the PDAF mode). For example, comparing the PDAF depth estimate and the ToF depth estimate includes determining whether a delta depth estimate based on a difference between the PDAF depth estimate and the ToF depth estimate exceeds a depth threshold, and determining that the foreground object in the zoom preview is in focus for the ToF-based AF mode is based on determining that the delta depth estimate exceeds the depth threshold.

[0062] In some embodiments, based on determining that the foreground object in the zoom preview is in focus due to the ToF-based AF mode, bypassing the PDAF mode for focusing on the foreground object and activating the ToF-based AF mode, the PDAF mode includes focusing the camera system based on the PDAF depth estimate, and the ToF-based AF mode includes focusing the camera system based on the ToF depth estimate.

[0063] Upon determining that the delta depth estimate cannot exceed the depth threshold, the process proceeds to step 715, where the "Bypass PDAF" parameter is set to "False" indicating that the PDAF mode remains active and the ToF-based AF mode will not be activated.

[0064] Upon determining that the delta depth estimate is greater than the depth threshold, the process proceeds to step 725 .

[0065] In step 725, the process includes determining whether the ToF depth estimate is valid. For example, the ToF sensor may determine that an object is close, but the estimate may not be accurate. This may cause the ToF depth estimate to be invalid.

[0066] Upon determining that the ToF depth estimate is not valid, the process proceeds to step 715, where the "Bypass PDAF" parameter is set to "False" indicating that the PDAF mode remains active and the ToF-based AF mode will not be activated.

[0067] Upon determining that the ToF depth estimate is valid, the process proceeds to step 730 .

[0068] In step 730, the process includes determining whether the brightness intensity of the background exceeds a brightness threshold.

[0069] Upon determining that the background brightness intensity cannot exceed the brightness threshold, the process proceeds to step 715, where the "Bypass PDAF" parameter is set to "False" indicating that the PDAF mode remains active and the ToF-based AF mode will not be activated.

[0070] Upon determining that the background brightness intensity is above the brightness threshold, the process proceeds to step 735, where the "Bypass PDAF" parameter is set to "True" indicating that the PDAF mode is bypassed and the ToF-based AF mode is activated.

[0071] 8A is another example workflow 800A of a hybrid autofocus system with robust macro-object-priority focusing, according to an example embodiment. As shown, a hybrid AF macro-object-priority module 805 may be initiated. In some embodiments, the hybrid AF macro-object-priority module 805 may implement the autofocus aspects of the camera system.

[0072] In step 810, the process includes determining whether PDAF mode needs to be bypassed. For example, the hybrid AF macro object priority module 805 may trigger the PDAF bypass determination module 705 shown in FIG.

[0073] Upon determining that PDAF mode is not bypassed (e.g., workflow 700 ends at step 715), the process proceeds to step 815. In step 815, the camera system uses a conventional hybrid AF strategy hierarchy that includes applying a phase detection autofocus (PDAF) algorithm, followed by a time-of-flight (ToF)-based algorithm, and then a contrast detection autofocus (CDAF) algorithm.

[0074] Upon determining that the PDAF mode is to be bypassed (eg, when the workflow 700 ends at step 735), the process proceeds to step 820.

[0075] In step 820, the process includes determining whether the ToF depth estimate is valid and whether the ToF depth estimate is above a ToF confidence threshold.

[0076] If a determination is made that the ToF depth estimate is valid and that the ToF depth estimate is above the ToF confidence threshold, the process proceeds to step 825 .

[0077] In step 825, the process includes focusing the camera system in a ToF-based AF mode based on a distance-to-position mapping of the foreground object based on the ToF depth estimate.

[0078] In some embodiments, the ToF sensor may determine that an object is nearby, but this estimate may not be accurate, or if a determination is made that the ToF depth estimate is not valid or that the ToF depth estimate cannot exceed the ToF confidence threshold, the estimate may not be reliable in bright light settings, and the process proceeds to step 830.

[0079] In step 830, the process includes focusing the camera system in a ToF-based AF mode based on a multi-grid contrast detection autofocus (CDAF) search based on the ToF depth estimate. In some embodiments, the multi-grid CDAF search is based on one or more of a spatial grid, one or more directions, or one or more spatial frequencies. For example, two directions, one horizontal and one vertical, may be used. Also, for example, the grid may be an M×N array.

[0080] FIG. 8B is a diagram of a multi-grid contrast detection autofocus (CDAF) analysis 800B according to an exemplary embodiment. As shown, a 5×5 grid 835 is shown in the image. Based on the 5×5 grid 835, one or more spatial frequencies (e.g., high, mid, etc.) may be determined, and one or more spatial directions (e.g., horizontal, vertical, etc.) may be used. For example, a 5×5 array of FV high frequencies 840 may be determined. Also, for example, a 5×5 array of FV mid frequencies 845 may be determined. For example, each subgrid within the grid 835 corresponds to a high-frequency distribution and a mid-frequency distribution. Thus, each of the high frequencies 840 and mid frequencies 845 includes a focus value (FV) curve for the respective array. In some embodiments, the FV curve may be determined as a summation of one or more directions (e.g., a summation of the horizontal and vertical directions). However, other combinations may be used to generate the FV curve. The actual FV curve is not relevant to this discussion, and the curve shown is for illustrative purposes. Also, for example, although high and medium frequencies are shown, other frequencies may be utilized as well.

[0081] In some embodiments, the CDAF search may also include determining, for the grid 835, a peak signal-to-noise ratio (PSNR) 850 of the 5x5 array, a sharpness ratio 855 of the 5x5 array, a peak focus position 860 of the 5x5 array, etc. Different intensities in the PSNR 850, sharpness ratio 855, and peak focus position 860 may be represented by different colors, shading, etc. For example, the final focus position may be determined based on a weighted histogram analysis that considers the interpolated peaks from each grid, weighted by its FV curve quality metric, which may be represented as PSNR 850 or sharpness ratio 855. Additional and / or alternative quality factors may be used to determine the quality metric. For example, the quality metric may be based on one or more factors, such as unimodality, accuracy, reproducibility, definition range, general applicability, and robustness. In some embodiments, the final position may be determined based on a percentile from the histogram (e.g., 33% for the rule of thirds), taking into account depth of field.

[0082] Data Network Example 9 illustrates a distributed computing architecture 900, according to an example embodiment. The distributed computing architecture 900 includes server devices 908, 910 configured to communicate with programmable devices 904a, 904b, 904c, 904d, and 904e via a network 906. The network 906 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 906 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.

[0083] While FIG. 9 shows only five programmable devices, the distributed application architecture can serve tens, hundreds, or thousands of programmable devices. Furthermore, programmable devices 904a, 904b, 904c, 904d, and 904e (or any additional programmable devices) may be any type of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mountable device (HMD), a network terminal, a mobile computing device, etc. In some examples, such as shown by programmable devices 904a, 904b, 904c, and 904e, the programmable devices may be directly connected to network 906. In other examples, such as shown by programmable device 904d, the programmable devices may be indirectly connected to network 906 through an associated computing device, such as programmable device 904c. In this example, programmable device 904c can serve as an associated computing device for passing electronic communications between programmable device 904d and network 906. In other examples, as shown by programmable device 904e, the computing device may be part of and / or within a vehicle, such as a car, truck, bus, boat or watercraft, airplane, etc. In other examples not shown in Figure 9, the programmable device may be connected both directly and indirectly to the network 906.

[0084] Server devices 908, 910 may be configured to perform one or more services requested by programmable devices 904a-904e. For example, server devices 908 and / or 910 may provide content to programmable devices 904a-904e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or decrypted. Other types of content are possible as well.

[0085] As another example, server devices 908 and / or 910 may provide programmable devices 904a-904e with access to software for database, search, calculation, graphics, audio, video, World Wide Web / Internet usage, and / or other functions. Many other examples of server devices are possible as well.

[0086] Computing Device Architecture 10 is a block diagram of an exemplary computing device 1000, according to an exemplary embodiment. In particular, the computing device 1000 shown in FIG. 10 may be configured to perform at least one function of and / or related to method 1100.

[0087] By way of example, and not limitation, computing device 1000 may be a cellular mobile phone (e.g., a smartphone), a still camera, a video camera, a fax machine, a computer (such as a desktop, notebook, tablet, or handheld computer), a personal digital assistant (PDA), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, or some other type of device having at least some image capture and / or image processing capabilities. It should be understood that computing device 1000 may represent a physical camera device, such as a digital camera, a particular physical hardware platform on which a camera application runs in software, or other combinations of hardware and software configured to perform camera functions.

[0088] As shown in FIG. 10, computing device 1000 may include a user interface module 1001, a network communication module 1002, one or more processors 1003, data storage 1004, one or more cameras 1018, one or more sensors 1020, and a power system 1022, all of which may be linked to each other via a system bus, network, or other connection mechanism 1005.

[0089] The user interface module 1001 may be operable to transmit data to and / or receive data from external user input / output devices. For example, the user interface module 1001 may be configured to transmit data to and / or receive data from user input devices such as a touchscreen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 1001 may also be configured to provide output to a user display device such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices now known or later developed. The user interface module 1001 may also be configured to generate audible output using devices such as speakers, speaker jacks, audio output ports, audio output devices, earphones, and / or other similar devices. User interface module 1001 may further comprise one or more haptic devices that may generate tactile output, such as vibration and / or other output detectable by touch and / or physical contact with computing device 1000. In some examples, user interface module 1001 may be used to provide a graphical user interface (GUI) for utilizing computing device 1000.

[0090] In some embodiments, user interface module 1001 may include a display that functions as a viewfinder for still camera and / or video camera functions supported by computing device 1000. Additionally, user interface module 1001 may include one or more buttons, switches, knobs, and / or dials that facilitate configuring and focusing camera functions and capturing images (e.g., capturing photographs). Some or all of these buttons, switches, knobs, and / or dials may be capable of being implemented by a presence-sensing panel.

[0091] The network communication module 1002 may include one or more devices providing one or more wireless interfaces 1007 and / or one or more wired interfaces 1008 configurable to communicate over a network. The wireless interface(s) 1007 may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth® transceiver, a Zigbee® transceiver, a Wi-Fi® transceiver, a WiMAX® transceiver, an LTE® transceiver, and / or other type of wireless transceiver configurable to communicate over a wireless network. The wired interface(s) 1008 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet® transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate over twisted pair wire, coaxial cable, fiber optic link, or similar physical connection to a wired network.

[0092] In some examples, the network communication module 1002 can be configured to provide reliable, secure, and / or authenticated communications. For each communication described herein, information to facilitate reliable communications (e.g., guaranteed message delivery) can be provided, perhaps as part of the message header and / or footer (e.g., packet / message ordering information, encapsulation header and / or footer, size / time information, and transmission verification information such as a cyclic redundancy check (CRC) and / or parity check value). Communications may be secured (e.g., encoded or encrypted) and / or decrypted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA), the Diffie-Hellman algorithm, a secure socket protocol such as the Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or the Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms may be used to secure (and subsequently decrypt / decode) communications, similar to or in addition to those enumerated herein.

[0093] The one or more processors 1003 may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application-specific integrated circuits, etc.) The one or more processors 1003 may be configured to execute computer-readable instructions 1006 contained in data storage 1004 and / or other instructions described herein.

[0094] Data storage 1004 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 1003. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical storage, magnetic storage, organic storage, or other memory or disk storage, which may be integrated, in whole or in part, with at least one of the one or more processors 1003. In some examples, data storage 1004 may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage unit), while in other examples, data storage 1004 may be implemented using two or more physical devices.

[0095] The data storage 1004 may include computer-readable instructions 1006 and possibly additional data. In some examples, the data storage 1004 may include storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functionality of the devices and networks described herein. In some examples, the data storage 1004 may include storage for a hybrid AF module 1012 (e.g., a module that executes a hybrid AF macro-object-first procedure, calculates a PDAF algorithm, a ToF algorithm, a CDAF algorithm, etc., and performs one of many operations related to the hybrid autofocus system with robust macro-object-first focusing described herein). In particular, in these examples, the computer-readable instructions 1006 may include instructions that, when executed by the processor(s) 1003, enable the computing device 1000 to provide some or all of the functionality of the hybrid AF 1012.

[0096] In some examples, computing device 1000 may include one or more cameras 1018. Camera(s) 1018 may include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light into one or more images. That is, camera(s) 1018 may generate image(s) of the captured light. The one or more images may be one or more still images and / or one or more images utilized in video capture. Camera(s) 1018 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or one or more other frequencies of light. Camera(s) 1018 may include wide-angle cameras, telephoto cameras, ultra-wide-angle cameras, etc. Also, for example, camera(s) 1018 may be front-facing or rear-facing cameras relative to computing device 1000. The camera(s) 1018 may include camera components such as, but not limited to, an aperture, a shutter, a recording surface (e.g., photographic film and / or an image sensor), a lens, and / or a shutter button. The camera components may be controlled at least in part by software executed by the one or more processors 1003.

[0097] In some examples, computing device 1000 may include one or more sensors 1020. Sensors 1020 may be configured to measure conditions within computing device 1000 and / or conditions in the environment of computing device 1000 and provide data regarding these conditions.For example, the sensors 1020 may be (i) sensors for acquiring data about the computing device 1000, such as, but not limited to, a thermometer for measuring the temperature of the computing device 1000, a battery sensor for measuring the power of one or more batteries of the power supply system 1022, and / or other sensors for measuring the state of the computing device 1000; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, a radio frequency identification (RFID) reader, a proximity sensor, a one-dimensional barcode reader, a two-dimensional barcode (e.g., a quick response (QR) code) reader, and a laser tracker, which may be configured to read identifiers such as RFID tags, barcodes, QR codes, and / or other devices and / or objects configured to read and provide at least identification information; (iii) tilt sensors, gyroscopes, accelerometers, The sensors 1020 may include one or more of the following: (i) sensors that measure the position and / or movement of the computing device 1000, such as, but not limited to, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser displacement sensor, and a compass; (ii) environmental sensors that acquire data indicative of the environment of the computing device 1000, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a movement sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (iii) force sensors that measure one or more forces (e.g., inertial forces and / or G-forces) acting about the computing device 1000, such as, but not limited to, one or more sensors that measure force, torque, ground force, friction in one or more dimensions, and / or a zero moment point (ZMP) sensor that identifies the ZMP and / or the location of the ZMP. Many other examples of sensors 1020 are possible as well.

[0098] The power supply system 1022 may include one or more batteries 1024 and / or one or more external power interfaces 1026 for providing power to the computing device 1000. Each battery of the one or more batteries 1024, when electrically coupled to the computing device 1000, can serve as a source of stored power for the computing device 1000. The one or more batteries 1024 of the power supply system 1022 may be configured to be portable. Some or all of the one or more batteries 1024 may be easily removable from the computing device 1000. In other examples, some or all of the one or more batteries 1024 may be internal to the computing device 1000 and therefore may not be easily removable from the computing device 1000. Some or all of the one or more batteries 1024 may be rechargeable. For example, rechargeable batteries may be recharged via a wired connection between the battery and another power source, such as by one or more power sources external to the computing device 1000 and connected to the computing device 1000 via one or more external power interfaces. In other examples, some or all of the one or more batteries 1024 may be non-rechargeable batteries.

[0099] The one or more external power interfaces 1026 of the power system 1022 may include one or more wired power interfaces, such as a USB cable and / or a power cord, that enable a wired power connection to one or more power sources external to the computing device 1000. The one or more external power interfaces 1026 may include one or more wireless power interfaces, such as a Qi wireless charger, that enable a wireless power connection to one or more external power sources, such as via a Qi wireless charger. Once a power connection to an external power source is established using the one or more external power interfaces 1026, the computing device 1000 may draw power from the external power source via the established power connection. In some examples, the power system 1022 may include associated sensors, such as battery sensors or other types of power sensors associated with one or more batteries.

[0100] Exemplary Methods of Operation 11 illustrates a method 1100 according to an example embodiment. The method 1100 may include various blocks or steps. The blocks or steps may be performed individually or in combination. The blocks or steps may be performed in any order and / or sequentially or in parallel. Additionally, blocks or steps may be omitted or added to the method 1100.

[0101] The blocks of the method 1100 may be performed by various elements of the computing device 1000 as illustrated and described with reference to FIG.

[0102] Block 1110 involves displaying a zoomed preview of the scene captured by the camera system via a display screen of the camera system.

[0103] Block 1120 involves determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene.

[0104] Block 1130 includes determining whether a foreground object in the zoom preview is in focus for the ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate.

[0105] Block 1140 includes, based on a determination that the foreground object in the zoom preview is in focus due to the ToF-based AF mode, bypassing the PDAF mode for focusing on the foreground object and activating the ToF-based AF mode, the PDAF mode including focusing the camera system based on the PDAF depth estimate, and the ToF-based AF mode including focusing the camera system based on the ToF depth estimate.

[0106] Block 1150 includes displaying, via the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoom preview of the scene.

[0107] Some embodiments include determining, based on a second comparison of the second PDAF depth estimate and the second ToF depth estimate, that a second foreground object in the second zoomed preview of the scene is out of focus for the ToF-based AF mode. Such embodiments include maintaining the PDAF mode and not activating the ToF-based AF mode based on a determination that a second foreground object in the second zoomed preview of the scene is out of focus for the ToF-based AF mode, and displaying includes displaying the second zoomed preview based on the PDAF mode.

[0108] In some embodiments, comparing the PDAF depth estimate and the ToF depth estimate includes determining whether a delta depth estimate based on a difference between the PDAF depth estimate and the ToF depth estimate exceeds a depth threshold, and determining that a foreground object in the zoom preview is in focus for the ToF-based AF mode is based on determining that the delta depth estimate exceeds the depth threshold.

[0109] In some embodiments, receiving the PDAF depth estimate includes determining whether the PDAF depth estimate exceeds a PDAF confidence threshold, and bypassing the PDAF mode is based on determining whether the PDAF depth estimate exceeds the PDAF confidence threshold.

[0110] Such embodiments include receiving a second PDAF depth estimate based on the second zoom preview of the scene. Such embodiments include determining that the second PDAF depth estimate does not exceed a PDAF confidence threshold. Such embodiments also include maintaining the PDAF mode and not activating the ToF-based AF mode, and displaying includes displaying the second zoom preview based on the PDAF mode.

[0111] In some embodiments, receiving the ToF depth estimate includes determining whether the ToF depth estimate is above a ToF confidence threshold, and focusing the camera system in the ToF-based AF mode is based on determining whether the ToF depth estimate is above the ToF confidence threshold.

[0112] Such embodiments include determining that the ToF depth estimate exceeds a ToF confidence threshold, and focusing the camera system in a ToF-based AF mode is based on a distance-to-position mapping of the foreground object based on the ToF depth estimate.

[0113] Some embodiments include determining that the ToF depth estimate does not exceed a ToF confidence threshold, and focusing the camera system in a ToF-based AF mode is based on a multi-grid contrast detection autofocus (CDAF) search based on the ToF depth estimate. In some embodiments, the multi-grid CDAF search is based on one or more of a spatial grid, one or more directions, or one or more spatial frequencies.

[0114] Some embodiments include receiving a second ToF depth estimate based on a second zoom preview of the scene. Such embodiments include determining that the second ToF depth estimate does not exceed a ToF confidence threshold. Such embodiments also include maintaining a PDAF mode and not activating a ToF-based AF mode, and displaying includes displaying the second zoom preview based on the PDAF mode.

[0115] Some embodiments include determining that the PDAF depth estimate exceeds a PDAF confidence threshold. Such embodiments include determining that the ToF depth estimate exceeds a ToF confidence threshold. Such embodiments also include determining whether a background brightness intensity exceeds a brightness threshold based on a zoom preview of the scene, and bypassing the PDAF mode is based on determining whether the background brightness intensity exceeds the brightness threshold.

[0116] Some embodiments include determining that the brightness intensity of the background is above a brightness threshold, such embodiments include bypassing the PDAF mode and activating a ToF-based AF mode.

[0117] Some embodiments include determining that a second brightness intensity of a second background in the second zoom preview does not exceed a brightness threshold. Such embodiments include maintaining the PDAF mode and not activating the ToF-based AF mode, and displaying includes displaying the second zoom preview based on the PDAF mode.

[0118] Some embodiments include receiving, via a user interface on the display screen, an indication to disable the ToF-based AF mode. Such embodiments include, in response to the indication, maintaining the PDAF mode and not activating the ToF-based AF mode, and displaying includes displaying a second zoom preview based on the PDAF mode.

[0119] Some embodiments include displaying, via a display screen, an initial preview of a scene to be captured by another camera system, the other camera system operating at another focal length equal to or greater than a threshold focal length. Such embodiments include detecting a zoom operation that triggers a transition from the second camera system to the camera system. Some embodiments also include providing, via a user interface on the display screen, a selectable virtual object to receive an indication of whether to enable or disable the ToF-based AF mode. In such embodiments, the camera system is configured to provide an ultra-wide field of view (FOV) and the other camera system is configured to provide a wide FOV.

[0120] For example, the camera system may automatically switch to an ultra-wide-angle (UWA) camera (e.g., cropped to 1×) when the user moves closer than 15 centimeters (cm) to the object. In some embodiments, a button in the user interface representing macro mode may appear and be highlighted. In the event that the button is pressed while in macro mode (e.g., less than 18 cm away), the UWA camera may be disengaged and the camera system may return to the main sensor. Also, for example, pressing a button (e.g., 0.7×) may disengage macro mode or switch back to the normal UWA view. In the event that the user moves away from the object (e.g., more than 18 cm), the camera system automatically switches back to the main sensor.

[0121] In some embodiments, focusing the camera system includes adjusting at least one lens of the camera system.

[0122] In some embodiments, focusing the camera system includes determining an exposure time for the camera system based on a motion blur tolerance of the camera system.

[0123] In some embodiments, the camera system is a component of a mobile device.

[0124] 12 illustrates the difference between image focusing with and without hybrid AF-based macro object focusing, according to an exemplary embodiment. Images 1200A and 1200B correspond to a camera system in which hybrid AF-based macro object focusing is activated. Upon transitioning to the ultra-wide-angle camera (e.g., from the main camera), the camera system is able to detect a brightness peak corresponding to a close-up of an object. However, as shown by image 1200C, for a camera system in which hybrid AF-based macro object focusing is not activated, upon transitioning to the ultra-wide-angle camera, the camera system focuses on the background and is unable to focus on the foreground object.

[0125] The particular configuration shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given figure. Furthermore, some of the illustrated elements may be combined or omitted. Furthermore, example embodiments may include elements not shown in the figures.

[0126] Steps or blocks representing the processing of information may correspond to circuitry that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, steps or blocks representing the processing of information may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor to implement specific logical functions or operations in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device, including a disk, hard drive, or other storage medium.

[0127] Computer-readable media may also include non-transitory computer-readable media, such as register memory, processor cache, and computer-readable media that store data for a short period of time, such as random access memory (RAM). Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for a long period of time. Thus, computer-readable media may include, for example, secondary or persistent long-term storage, such as read-only memory (ROM), optical or magnetic disks, compact disk read-only memory (CD-ROM), etc. Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media may be considered, for example, to be a computer-readable storage medium or a tangible storage device.

[0128] While various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various examples and embodiments disclosed are for purposes of illustration and are not intended to be limiting, the true scope being indicated by the following claims.

Claims

1. 1. A computer-implemented method, the method comprising: displaying a zoom preview of a scene captured by a camera system on a display screen of the camera system; determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; determining whether a foreground object in the zoom preview is in focus for a ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate; and for the ToF-based AF mode, based on determining that the foreground object in the zoom preview is in focus, bypassing a PDAF mode for focusing on the foreground object and activating the ToF-based AF mode, the PDAF mode including focusing the camera system based on the PDAF depth estimate, and the ToF-based AF mode including focusing the camera system based on the ToF depth estimate, the method further comprising: displaying, by the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoom preview of the scene.

2. The method further comprises: determining, based on a second comparison of the second PDAF depth estimate and the second ToF depth estimate, that a second foreground object in the second zoom preview of the scene is out of focus for the ToF-based AF mode; and maintaining the PDAF mode and not activating the ToF-based AF mode based on the determination that the second foreground object in the second zoom preview of the scene is not in focus due to the ToF-based AF mode; The method of claim 1 , wherein the displaying includes displaying the second zoom preview based on the PDAF mode.

3. the comparing the PDAF depth estimate and the ToF depth estimate includes determining whether a delta depth estimate based on a difference between the PDAF depth estimate and the ToF depth estimate exceeds a depth threshold; 2. The method of claim 1, wherein for the ToF-based AF mode, the determination that the foreground object in the zoom preview is in focus is based on a determination that the delta depth estimate is above the depth threshold.

4. The receiving of the PDAF depth estimate further includes determining whether the PDAF depth estimate exceeds a PDAF confidence threshold; The method of claim 1 , wherein the bypass of the PDAF mode is based on the determination of whether the PDAF depth estimate exceeds the PDAF confidence threshold.

5. The method further comprises: receiving a second PDAF depth estimate based on a second zoom preview of the scene; determining that the second PDAF depth estimate does not exceed the PDAF confidence threshold; maintaining the PDAF mode and not activating the ToF-based AF mode; The method of claim 4 , wherein the displaying includes displaying the second zoom preview based on the PDAF mode.

6. The receiving of the ToF depth estimate further includes determining whether the ToF depth estimate exceeds a ToF confidence threshold; The method of claim 1 , wherein the focusing of the camera system in the ToF-based AF mode is based on the determination of whether the ToF depth estimate is above the ToF confidence threshold.

7. The method further includes determining that the ToF depth estimate exceeds the ToF confidence threshold; The method of claim 6 , wherein the focusing of the camera system in the ToF-based AF mode is based on a distance-to-position mapping of the foreground object based on the ToF depth estimate.

8. The method further includes determining that the ToF depth estimate does not exceed the ToF confidence threshold; The method of claim 6 , wherein the focusing of the camera system in the ToF-based AF mode is based on a multi-grid contrast detection autofocus (CDAF) search based on the ToF depth estimate.

9. The method of claim 8 , wherein the multi-grid CDAF search is based on one or more of a spatial grid, one or more directions, or one or more spatial frequencies.

10. The method further comprises: receiving a second ToF depth estimate based on a second zoom preview of the scene; determining that the second ToF depth estimate does not exceed a ToF confidence threshold; maintaining the PDAF mode and not activating the ToF-based AF mode; The method of claim 1 , wherein the displaying includes displaying the second zoom preview based on the PDAF mode.

11. The method further comprises: determining that the PDAF depth estimate exceeds a PDAF confidence threshold; determining that the ToF depth estimate exceeds a ToF confidence threshold; determining whether a background brightness intensity exceeds a brightness threshold based on the zoom preview of the scene; The method of claim 1 , wherein the bypass of the PDAF mode is based on the determination of whether the brightness intensity of the background exceeds the brightness threshold.

12. The method further comprises: determining that the brightness intensity of the background is above the brightness threshold; and bypassing the PDAF mode and activating the ToF-based AF mode.

13. The method further comprises: determining that a second brightness intensity of a second background within a second zoom preview does not exceed the brightness threshold; maintaining the PDAF mode and not activating the ToF-based AF mode; The method of claim 11 , wherein the displaying includes displaying the second zoom preview based on the PDAF mode.

14. The method further comprises: receiving, via a user interface of the display screen, an indication to disable the ToF-based AF mode; in response to the indication, maintaining the PDAF mode and not activating the ToF-based AF mode; The method of claim 1 , wherein the displaying includes displaying a second zoom preview based on the PDAF mode.

15. The method further comprises: displaying, by the display screen, an initial preview of the scene captured by another camera system; detecting a zoom movement causing a transition from the second camera system to the camera system; 10. The method of claim 1, comprising providing, via a user interface on the display screen, a virtual object selectable to receive an indication of whether to enable or disable the ToF-based AF mode.

16. The method of claim 15 , wherein the camera system is configured to provide an ultra-wide field of view (FOV) and the other camera system is configured to provide a wide FOV.

17. The method of claim 1 , wherein the focusing of the camera system comprises adjusting at least one lens of the camera system.

18. The method of claim 1 , wherein the focusing of the camera system includes determining an exposure time of the camera system based on a motion blur tolerance of the camera system.

19. The method of claim 1 , wherein the camera system is a component of a mobile device.

20. 1. A computing device, comprising: A display screen; a camera system configured to operate at a focal length less than a threshold focal length; one or more processors; and a data storage storing computer-executable instructions that, when executed by the one or more processors, cause the mobile device to perform functions, the functions including: displaying, via a display screen of the camera system, a zoomed preview of the scene captured by the camera system; determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; determining whether a foreground object in the zoom preview is in focus for a ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate; and for the ToF-based AF mode, based on determining that the foreground object in the zoom preview is in focus, bypassing a PDAF mode for focusing on the foreground object and activating the ToF-based AF mode, the PDAF mode including focusing the camera system based on the PDAF depth estimate, and the ToF-based AF mode including focusing the camera system based on the ToF depth estimate, the function further comprising: and displaying, by the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoom preview of the scene.

21. A non-transitory computer-readable medium containing program instructions executable by one or more processors, the program instructions causing the one or more processors to perform operations, the operations including: displaying a zoom preview of a scene captured by a camera system on a display screen of the camera system; determining a phase detection autofocus (PDAF) depth estimate and a time-of-flight (ToF) depth estimate of the scene; determining whether a foreground object in the zoom preview is in focus for a ToF-based AF mode of the camera system based on a comparison of the PDAF depth estimate and the ToF depth estimate; and for the ToF-based AF mode, based on determining that the foreground object in the zoom preview is in focus, bypassing a PDAF mode for focusing on the foreground object and activating the ToF-based AF mode, the PDAF mode including focusing the camera system based on the PDAF depth estimate, and the ToF-based AF mode including focusing the camera system based on the ToF depth estimate, the operations further comprising: and displaying, by the display screen and based on the ToF-based AF mode, the focused foreground object as part of the zoom preview of the scene.

Images