Camera switching control technique for multi-camera systems

The method for selecting a primary camera in multi-camera systems addresses focus challenges by using object distance estimates and error information to dynamically switch cameras and maintain focus, improving image capture quality.

JP7579306B2Active Publication Date: 2024-11-07APPLE INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022145616
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-09-20
Filing Date
2022-09-13
Publication Date
2024-11-07
Estimated Expiration
2042-09-13

AI Technical Summary

Technical Problem

In multi-camera systems, selecting the appropriate camera for image capture can be challenging due to overlapping fields of view and varying minimum object distances, leading to potential focus issues when objects are closer than the minimum object distance of the camera.

Method used

A method for selecting a primary camera based on object distance estimates, distance error information, and minimum object distances using camera selection logic, which may include a depth sensor to determine the appropriate camera and adjust focus positions dynamically.

Benefits of technology

Ensures accurate focus on objects at varying distances by dynamically switching primary cameras and maintaining focus positions, enhancing image capture quality and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579306000001
    Figure 0007579306000001
  • Figure 0007579306000002
    Figure 0007579306000002
  • Figure 0007579306000003
    Figure 0007579306000003
Patent Text Reader

Abstract

To estimate one or more distances to a target object using a multi-camera system and select a primary camera for and / or focusing a camera of the multi-camera system using the multi-camera system.SOLUTION: Various embodiments disclosed herein include techniques for operating multiple camera systems. In some embodiments, a primary camera may be selected from a plurality of cameras using object distance estimates, distance error information and minimum object distances for some or all of the cameras. In other embodiments, a camera may be configured to use defocus information to obtain an object distance estimate to a target object closer than a minimum object distance of the camera. This object distance estimate may be used to assist in focusing another camera of the multi-camera system.SELECTED DRAWING: Figure 1A
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit under 35 U.S.C. Section 119(e) of U.S. Provisional Patent Application No. 63 / 243,611, filed September 13, 2021, the contents of which are incorporated herein by reference in their entirety.

[0002] The described embodiments relate generally to estimating one or more distances to an object of interest using a multi-camera system, selecting a primary camera for the multi-camera system using the multi-camera system, and / or focusing the cameras. [Background technology]

[0003] Cameras continue to be an important feature of consumer electronic devices such as smartphones, tablets, and computers. While these consumer electronic devices traditionally had either a single camera or two cameras facing in opposite directions, many devices now have multiple camera systems with two or more cameras with at least partially overlapping fields of view. Different cameras in these multiple camera systems can be advantageously utilized in different scenarios, and having multiple fields of view can enable the overall device to provide different levels of optical zoom without requiring a camera with a large and complex zoom lens. With multiple cameras that can be selectively used across different image modes, it can be difficult to ensure that the appropriate camera or cameras are used (and used effectively) in different cases. Summary of the Invention

[0004] The present disclosure relates to a multi-camera system. In some variations, embodiments include a multi-camera system and related methods of selecting a primary camera from the multi-camera system. Specifically, some embodiments include a method of selecting between a first camera and a second camera of a multi-camera system as a primary camera during image capture of an object of interest, the first camera having a first minimum object distance and the second camera having a second minimum object distance. These methods can include determining a first object distance estimate of the object of interest using the first camera, the first object distance estimate being associated with first distance error information; determining a second object distance estimate of the object of interest using the second camera, the second object distance estimate being associated with second distance error information; and using the first object distance estimate, the second object distance estimate, the first distance error information, the second distance error information, and the second minimum object distance to select one of the first camera and the second camera as the primary camera using a camera selection logic.

[0005] In some of these variations, the multi-camera system may include a depth sensor, and the method may further include determining a third object distance estimate for the target object using the depth sensor. In some of these embodiments, the method may further include holding a focus position of the second camera in a predetermined position when the third object distance estimate is less than the first threshold distance. In other embodiments, selecting one of the first camera and the second camera as the primary camera includes setting a first switch threshold distance using the first distance error information, the second distance error information, and the second minimum object distance. In some of these variations, selecting one of the first camera and the second camera as the primary camera includes selecting the first camera as the primary camera when either the first object distance estimate or the second object distance estimate is less than the first switch threshold distance. In others of these variations, selecting one of the first and second cameras as the primary camera includes setting a second switching threshold distance greater than the first switching threshold distance, and the method further includes selecting the second camera as the primary camera when both the first object distance estimate and the second object distance estimate are greater than the second switching threshold distance. In some variations, the second minimum object distance dynamically varies based on the calculated orientation of the second camera.

[0006] Another embodiment includes a device with a multi-camera system comprising a first camera having a first minimum object distance and a first field of view, a second camera having a second minimum object distance and a second field of view, and camera selection logic configured to select either the first camera or the second camera as a primary camera, The cameras are further configured to receive a first object distance instance from the first camera and first distance error information associated with the first object distance estimate, receive a second object distance instance from the second camera and second distance error information associated with the second object distance estimate, receive a second minimum object distance, and select the primary camera using the first object distance estimate, the second object distance estimate, the first distance error information, the second distance error information, and the second minimum object distance.

[0007] In some of these variations, the first minimum object distance is less than the second minimum object distance and the first field of view completely encompasses the second field of view. Additionally or alternatively, the multi-camera system further comprises a depth sensor configured to generate a third object distance estimate. In some of these variations, the multi-camera system is configured to hold a focus position of the second camera in place when the third object distance estimate is less than the first threshold distance. In other variations, selecting the primary camera includes setting a first switch threshold distance using the first distance error information, the second distance error information, and the second minimum object distance. In some of these variations, selecting the primary camera includes selecting the first camera as the primary camera when either the first object distance estimate or the second object distance estimate is less than the first switch threshold distance. In these variations, selecting the primary camera includes setting a second switching threshold distance greater than the first switching threshold distance, and further includes selecting the second camera as the primary camera when both the first object distance estimate and the second object distance estimate are greater than the second switching threshold distance.

[0008] In another embodiment, a method of selecting among a plurality of cameras of a multi-camera system as a primary camera during image capture of an object of interest, each of the plurality of cameras having a respective minimum object distance, may include determining at least one object distance estimate of the object of interest using at least one of the plurality of cameras, each of the at least one object distance estimate being associated with respective distance error information, and selecting a primary camera from the plurality of cameras using camera selection logic. In selecting the primary camera, the camera logic may use the at least one object distance estimate, the respective distance error information associated with each of the at least one object distance estimate, and at least one of the respective minimum object distances of the plurality of cameras. In some examples, the method may include updating a first minimum object distance of at least one of the respective minimum object distances and selecting the primary camera using the updated first minimum object distance. In other examples, the method may include updating a first distance error information of the respective distance effort information associated with each of the at least one object distance estimate, and selecting the primary camera using the updated first distance error information.

[0009] In some examples, the multi-camera system further includes a depth sensor, and selecting the primary camera using the camera selection logic further includes selecting the primary camera using a first object distance estimate from the depth sensor. In some of these variations, the method may further include holding a focus position of a first camera of the multiple cameras in a predetermined position when the first object distance estimate is less than a first threshold distance. In some cases, the method may include setting a focus position for one or more of the multiple cameras (in some embodiments, each of the multiple cameras) using the camera selection logic.

[0010] In some variations, embodiments include a multi-camera system and an associated method for focusing cameras of the multi-camera system. In some embodiments, a device may include a multi-camera system including a first camera configured with autofocus capabilities and having a first minimum object distance, and a second camera configured with autofocus capabilities and having a second minimum object distance. The device further includes focus control logic configured to cause the second camera to measure defocus information of a target object at a predetermined focus position, determine an extrapolated focus position using the defocus information, and focus the first camera on the target object based at least in part on the extrapolated focus position. In some variations, the predetermined focus position corresponds to the minimum object distance of the second camera. In other variations, the focus control logic locks the first camera to the predetermined focus position when one or more predetermined locking criteria are met. In some of these variations, the multi-camera system further includes a depth sensor, and the one or more predetermined criteria include an object distance estimate of the target object being less than a threshold locking distance.

[0011] In some variations, the first minimum object distance is less than the second minimum object distance. The device may further include a camera selection logic configured to select either the first camera or the second camera as a primary camera, and the focus control logic is further configured to cause the second camera to measure defocus information of the target object at a predetermined focus position when the first camera is selected as the primary camera. In other variations, focusing the first camera on the target object based at least in part on the extrapolated focus position includes setting a target focus position of the first camera based on a predetermined relationship between the extrapolated focus positions when confidence information associated with the extrapolated focus positions meets a first predetermined criterion.

[0012] Other embodiments described herein include a method of focusing a first camera of a multi-camera system using a second camera of the multi-camera system, the method including measuring defocus information of a target object using the first camera at a predetermined focus position, determining an extrapolated focus position using the defocus information, and focusing the first camera on the target object based at least in part on the extrapolated focus position. In some examples, the predetermined focus position corresponds to a minimum object distance of the second camera. In other cases, the method may include locking the first camera to the predetermined focus position when one or more predetermined locking criteria are met. In some of these variations, the method may include calculating an object distance estimate of the target object using a depth sensor of the multi-camera system and comparing the object distance estimate to a threshold locking distance, the predetermined locking criteria being met when the object distance estimate is less than the threshold locking distance.

[0013] In some variations, measuring defocus information of the target object at a predetermined focus position using the first camera includes measuring the defocus information while the first camera is selected as the primary camera. In other variations, focusing the first camera on the target object based at least in part on the extrapolated focus position includes setting a target focus position of the first camera based on a predetermined relationship between the extrapolated focus positions when confidence information associated with the extrapolated focus positions meets a first predetermined criterion. In some of these variations, focusing the first camera on the target object based at least in part on the extrapolated focus position includes performing a partial scan based on the extrapolated focus position when confidence information associated with the extrapolated focus position does not meet a first predetermined criterion.

[0014] Other embodiments discussed herein include methods of calculating an object distance estimate to a target object using a multi-camera system including a first camera and a second camera. The methods include determining whether the second camera is maintained at a predetermined focus position, and, in response to determining that the second camera is maintained at the predetermined focus position, measuring defocus information of the target object while the second camera is maintained at the predetermined focus position, and calculating the object distance estimate using the defocus information and the predetermined focus position. In some of these methods, determining whether the second camera is maintained at the predetermined focus position includes determining that the second camera is locked at the predetermined focus position. In other variations, the predetermined focus position is a macro focus position of the second camera, and determining whether the second camera is maintained at the predetermined focus position includes determining that the target object is closer than a minimum object distance of the second camera.

[0015] In some variations, the method may further include calculating an extrapolated focus position of the second camera using the object distance estimate. In some of these variations, the method may include focusing the first camera based at least in part on the extrapolated focus position of the second camera. In some of these variations, focusing the first camera based at least in part on the extrapolated focus position includes selecting a target focus position for the first camera based on a predetermined relationship between the focus position of the second camera and the focus position of the first camera.

[0016] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the drawings and by study of the following descriptions.

[0017] The disclosure will be readily understood from the following detailed description when taken in conjunction with the accompanying drawings, in which like reference numbers designate like structural elements and in which: [Brief description of the drawings]

[0018] [Figure 1A] 1 shows a rear view of a device with a multi-camera system. [Figure 1B] 1B illustrates exemplary components of the device of FIG. 1A. [Figure 1C] 1B shows a representation of the field of view of the cameras of the multi-camera system of FIG. 1A. [Figure 1D] 1B shows a chart illustrating the focusing ranges of the cameras of the multi-camera system of FIG. 1A. [Diagram 2] 1 illustrates a cross-sectional side view of an illustrative example of a camera that may be used with the multi-camera systems described herein. [Diagram 3] 13 illustrates variations of camera selection logic that may be used to select between a first camera and a second camera as the primary camera. [Figure 4] 1 shows a flowchart illustrating an embodiment of a method for selecting a primary camera. [Diagram 5] 1 illustrates a variation of focus control logic that may be used to set a focus position of a first camera of a multi-camera system based at least in part on information from a second camera of the multi-camera system. [Figure 6] 1 shows a flowchart illustrating an embodiment of a method for calculating a distance to a target object using defocus information and focusing a first camera on the target object based at least in part on information from a second camera.

[0019] It will be understood that the proportions and dimensions (whether relative or absolute) of the various features and elements (and collections and groups thereof), as well as the boundaries, separations and relationships presented therebetween, are provided in the accompanying figures merely to facilitate an understanding of the various embodiments described herein, and therefore may not necessarily be presented or drawn to scale, and are not intended to imply any preference or requirement for the illustrated embodiments to the exclusion of the embodiments described with reference thereto.

[0020] Directional terms, such as "top," "bottom," "upper," "lower," "front," "back," "over," "under," "above," "below," "left," "right," "vertical," "horizontal," and the like, are used with reference to certain orientations of components in certain figures described below and are not intended to be limiting. Because components in various embodiments may be arranged in many different orientations, the directional terms are used for illustrative purposes only and are not intended to be limiting in any manner. The directional terms are intended to be interpreted broadly and therefore should not be construed to exclude components that are arranged in different manners. Also, as used herein, the phrase "at least one" preceding a series of items, along with the terms "and" or "or" separating any of the items, modifies the list as a whole and not each member of the list. The phrase "at least one" does not require the selection of at least one of each listed item, but rather the phrase allows for the inclusion of at least one of any of the items, and / or at least one of any combination of the items, and / or at least one of each of the items. By way of example, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" refer, respectively, to A only, B only, or C only, any combination of A, B, and C, and / or one or more of each of A, B, and C. Similarly, it can be understood that the order of elements presented for combined or separated lists provided herein should not be construed as limiting the disclosure to only the order provided. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Description will now be made in detail of representative embodiments as illustrated in the accompanying drawings. It should be understood that the following description is not intended to limit these embodiments to one preferred embodiment. On the contrary, the following description is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the described embodiments as defined by the appended claims.

[0022] The following disclosure relates to multiple camera ("multi-camera") systems and methods of operating these multi-camera systems. In some embodiments, the multi-camera system can be configured to select a primary camera from multiple cameras of the multi-camera system, and the multi-camera system is configured to switch between a first camera and a second camera as the primary camera at or near the minimum object distance of the second camera. In other embodiments, the multi-camera system can calculate an object distance estimate to the target object using a camera of the multi-camera system in situations where the target object is located closer than the minimum object distance of the camera. In some of these embodiments, this object distance estimate can be used to help set the focus position of the different cameras of the multi-camera system. Preferably, the camera can use defocus information associated with the target object to calculate an extrapolated focus position, which can be used in focusing another camera on the target object. These and other embodiments are discussed below with reference to Figures 1A-6. However, one skilled in the art will readily appreciate that the detailed description given herein with respect to these figures is for illustrative purposes only and should not be construed as limiting.

[0023] 1A illustrates a rear view of a device 100 including a multi-camera system 102 suitable for use with various embodiments described herein. Generally, the multi-camera system 102 includes a first camera 104 and a second camera 106. The multi-camera system 102 may optionally include one or more additional cameras (e.g., a third camera 108 shown in FIG. 1A) and / or one or more depth sensors (e.g., a depth sensor 110 shown in FIG. 1A). For purposes of this application, at least the first camera 104 and the second camera 106 are configured to have autofocus capabilities, as described in more detail below.

[0024] In some embodiments, device 100 is a portable multifunction electronic device, such as a mobile phone, that also includes other functions, such as PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, Calif. Optionally, other portable electronic devices, such as laptop or tablet computers, having a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad) are also used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer that may have a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad). In some embodiments, the electronic device is a computer system in communication (e.g., via wired communication, via wireless communication) with a display generating component. The display generating component is configured to provide a visual output, such as a display via a CRT display, a display via an LED display, or a display via image projection. In some embodiments, the display generating component is integrated with the computer system. In some embodiments, the display generating component is separate from the computer system. As used herein, "displaying" content includes causing the content to be displayed by transmitting data (e.g., image data or video data) over a wired or wireless connection to an integrated or external display generation component to visually generate the content.

[0025] 1B illustrates exemplary components of device 100. In some embodiments, device 100 includes a bus 126 that operably couples an I / O section 134 to one or more computer processors 136 and memory 138. The I / O section 134 can be connected to a display 128, which can include touch-sensitive components 130 and, optionally, an intensity sensor 132 (e.g., a contact intensity sensor). In addition, the I / O section 134 can be connected to a Wi-Fi (Registered Trademark) , Bluetooth (Registered Trademark) , near field communication (NFC), cellular, and / or other wireless communication technologies to receive application and operating system data. Device 100 may include input mechanism 142 and / or 144. Input mechanism 142 is optionally a rotatable input device or a depressible and rotatable input device, for example. In some embodiments, input mechanism 148 is optionally a button. Device 100 optionally includes various sensors, such as a GPS sensor 146, an accelerometer 148, an orientation sensor 150 (e.g., a compass), a gyroscope 152, a motion sensor 154, and / or combinations thereof, all of which may be operably connected to I / O section 134.

[0026] The memory 138 of the device 100 can include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, for example, when executed by one or more computer processors 136, can cause the computer processors to perform techniques described herein (such as those performed by the camera selection logic, focus control logic, and various controllers and modules associated therewith, discussed in detail below). A computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions used by or in connection with an instruction execution system, apparatus, or device. In some embodiments, the storage medium is a transitory computer-readable storage medium. In some embodiments, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium can include, but is not limited to, a magnetic storage device, an optical storage device, and / or a semiconductor storage device. Examples of such storage devices include a magnetic disk, a CD, a DVD, or a Blu-ray. (Registered Trademark) These include optical disks based on technologies, as well as persistent solid-state memory such as flash and solid-state drives.

[0027] Processor 136 may include, for example, special purpose hardware as defined herein, a computing device as defined herein, a processor, a microprocessor, a programmable logic array (PLA), a programmable array logic (PAL), a generic array logic (GAL), a complex programmable logic device (CPLD), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other programmable logic device (PLD) configurable to execute the operating system and applications of device 100 and to facilitate capturing images as described herein. Device 100 is not limited to the components and configurations of FIG. 1B and may include other or additional components in multiple configurations.

[0028] 1A , the cameras in the multi-camera system 102 have fields of view that at least partially overlap one another. In other words, the device 100 may include additional camera(s) (not shown) that are not considered part of the multi-camera system 102 if the field of view(s) of the additional camera(s) do not at least partially overlap the field of view of at least one camera in the multi-camera system 102. For example, the device 100 may include a front-facing camera (not shown) that faces in an opposite direction from the first camera 104 and the second camera 106 (as well as any other cameras of the multi-camera system 102) and thus is not considered part of the multi-camera system 102.

[0029] Similarly, a depth sensor 110 may be considered part of the multi-camera system 102 if the depth sensor 110 is positioned and configured within the device 100 such that the depth sensor 110 can obtain depth information at one or more points within the field of view of one or more of the cameras of the multi-camera system 102. The device 100 may include one or more depth sensors (e.g., a depth sensor on the front of the device that faces away from the cameras of the multi-camera system 102 and is therefore not considered part of the multi-camera system 102). It should be understood that a device may include two or more multi-camera systems (e.g., a first multi-camera system on one side of the device and a second multi-camera system on a second side of the device), each of which may optionally be equipped with a separate depth sensor, and some or all of the multi-camera systems may be configured to utilize various methods and techniques described herein.

[0030] The cameras of the multi-camera system 102 may have different focal lengths, which may result in the cameras having different field of view sizes. For example, FIG. 1C illustrates exemplary fields of view of the cameras of the multi-camera system 102 illustrated in FIG. 1A. The first camera 104 may have a first field of view 112, the second camera may have a second field of view 114, and the third camera 108 may have a third field of view 116. As illustrated therein, the field of view 112 of the first camera 104 may have a wider (i.e., larger) field of view than both the field of view 114 of the second camera 106 and the field of view 116 of the third camera 108. Preferably, the first field of view 112 completely encompasses both the second field of view 114 and the third field of view 116 (i.e., such that there is no portion of the second field of view 114 or the third field of view 116 that is not also part of the first field of view 112), but this need not be the case. Similarly, the field of view 114 of the second camera 106 may completely encompass the field of view 116 of the third camera 108. It should be understood that while the first field of view 112, second field of view 114, and third field of view 116 shown in FIG. 1C assume a fixed focal length for each of the first camera 104, second camera 106, and third camera 108, in some cases, one or more of the cameras in the multi-camera system may have some level of optical zoom capability (which may result in a field of view that varies in size based on the zoom level). For purposes of this application, the arrangement of fields of view shown in FIG. 1C may be assumed to be the widest possible field of view for each camera.

[0031] In general, the size of a camera's field of view is inversely proportional to the camera's focal length. In other words, the longer the focal length, the narrower the camera's field of view. The choice of camera field of view can affect the situations in which a particular camera may be useful. For example, cameras with longer focal lengths (and narrower fields of view) are often used for telephoto photography, where it is desirable to increase the magnification of a subject at greater distances, and cameras with shorter focal lengths (and wider fields of view) are often used when it is desirable to capture more of a scene (e.g., landscape photography).

[0032] Also shown in FIG. 1C is a coverage area 118 for the depth sensor 110, which encompasses the area within the scene for which the depth sensor 110 can calculate depth information for the scene. It should be understood that depending on the design of the depth sensor 110 (as discussed in more detail below), the depth sensor 110 may not calculate depth values ​​for all points within the coverage area 118. Instead, the coverage area 118 is intended to reflect the widest lateral extent for which the depth sensor can provide depth information. For purposes of this application, the coverage area 118 is represented as a rectangle, and the depth sensor 110 can provide depth information for at least one point on each side of the rectangle.

[0033] As mentioned above, the coverage area 118 for the depth sensor 110 at least partially overlaps some or all of the fields of view of the cameras of the multi-camera system 102. In some variations, it may be desirable to have the coverage area 118 completely encompass the fields of view of one or more cameras. For example, in the embodiment shown in FIG. 1C, the coverage area 118 of the depth sensor 110 may completely encompass the field of view 114 of the second camera 106 (as well as the field of view 116 of the third camera 108), while not completely encompassing the field of view 112 of the first camera 104. In other variations, the coverage area 118 may completely encompass the fields of view of all cameras in the multi-camera system 102 (including the fields of view of each of the first camera 104, the second camera 106, and the third camera 108 in the variation of the device 100 shown in FIG. 1A). In yet other variations, the coverage area 118 may completely encompass the field of view of a single camera (e.g., the field of view 116 of the third camera 108 in the variation of the multi-camera system 102 shown in FIG. 1C), while not completely encompassing the field of view of the remaining cameras.

[0034] The depth sensor 110 may be any suitable system capable of calculating distances between the depth sensor 110 and various points in a scene. The depth sensor may generate a depth map that includes these calculated distances, some or all of which may be used in various techniques described below. The depth information may be calculated in any suitable manner. In one non-limiting example, the depth sensor may utilize stereo imaging, where two images are taken from different positions, and the distance (parallax) between corresponding pixels in the two images may be used to calculate the depth information. In another example, the depth sensor may utilize structured light imaging, whereby the depth sensor may image the scene while projecting a known pattern (typically using infrared illumination) towards the scene, and then calculate the depth information by looking at how the pattern is distorted by the scene. In yet another example, the depth sensor may utilize time-of-flight sensing, whereby the depth is calculated based on the time it takes light (typically infrared) emitted from the depth sensor to return from the scene. A time-of-flight depth sensor may utilize direct time-of-flight or indirect time-of-flight and may illuminate the entire coverage area 118 at once or may illuminate only a subset of the coverage area 118 at a given time (e.g., with one or more spots, stripes, or other patterns that may be either fixed or scanned across the coverage area 118).

[0035] In practice, the depth sensor 110 may have a limited operating range (i.e., the range of scene depths for which the depth sensor 110 may be able to provide depth information). In general, the operating range of the depth sensor 110 includes a minimum and a maximum distance that the depth sensor 110 can output (e.g., the depth sensor 110 does not return distance values ​​for objects closer than the minimum distance or farther than the maximum distance), although it should be understood that in some cases, the minimum distance may be zero (i.e., the depth sensor 110 may be able to identify an object in contact with a portion of the device 100 and / or the depth sensor 110). In some cases, the depth sensor 110 may be configured to provide confidence information associated with the depth information, which represents the relative accuracy of the depth information provided by the depth sensor. By providing an indication of the accuracy of the depth information, the depth sensor 110 may extend the operating range to include shorter distances where the depth calculation may be less reliable. In these cases, the confidence information may enable various systems within the device to select which portion of the operating range to use when receiving depth information. For example, device 100 may be configured to use the full operation of depth sensor 110 in some contexts, but may only use a subset of the operating range (e.g., a portion of the operating range associated with a higher confidence value) in other contexts.

[0036] In some variations, there may be a portion of the operating range of depth sensor 110 where the depth information does not include a depth value for a given point in the scene, but instead includes a depth range. In other words, if an object is located within a particular distance range, depth sensor 110 may indicate that an object is located within that distance range without specifying a particular depth within that range (e.g., in a non-limiting example, depth sensor 110 may be configured to output "less than 20 cm" for any point in a scene where an object is within 20 cm of depth sensor 110).

[0037] Similarly, each camera has a minimum object distance (also known as a minimum focus distance), which represents the shortest distance in a scene that the camera can focus on (and therefore the closest an object can be to the camera while still being in focus). FIG. 1D illustrates an example of the range of each camera of the multi-camera system 102. As illustrated, the first camera 104, the second camera 106, and the third camera 108 may have a first minimum object distance 120, a second minimum object distance 122, and a third minimum object distance 124, respectively. In general, the minimum object distance 120 of the first camera 104 is shorter than the minimum object distance 122 of the second camera 106. The minimum object distance 122 of the second camera 106 may then be shorter than the minimum object distance 124 of the third camera 108. 1D, it is assumed that each of the first camera 104, the second camera 106, and the third camera 108 has autofocus capabilities and can focus to infinity, which allows each camera to selectively focus on objects ranging from that camera's minimum object distance to infinity. In these cases, each camera may be movable over a range of focus positions, with one end of the range (referred to herein as the "macro focus position") corresponding to that camera's minimum object distance.

[0038] It should be understood that not all of the cameras of the multi-camera system 102 need have adjustable focus capabilities and may in some cases include one or more fixed focus cameras. However, as discussed in more detail below, it is preferred that at least two cameras have autofocus capabilities. When a camera of a multi-camera system is configured with autofocus capabilities, the camera may be configured in any suitable manner, as would be readily apparent to one of ordinary skill in the art. In general, each camera of the multi-camera system 102 may include an image sensor and a lens, which includes one or more lens elements configured to direct light received by the camera to the image sensor (the lens may further include a lens barrel that houses some or all of the lens elements). When a camera is configured with autofocus capabilities, the camera may include an actuator, which may be configured to cause relative movement between the image sensor and one or more lens elements of the lens along the optical axis of the lens (e.g., by movement of the image sensor and / or one or more lens elements of the lens). Additionally or alternatively, the lens 106 may include a variable focus lens element (e.g., a liquid lens), which may be actuated to adjust the optical power of the lens element. For purposes of this application, a "focus position" of a camera may refer to the relative position and / or configuration of components that can be actively moved or adjusted to change the focus of the camera. Additionally, the camera may optionally be further configured to provide optical image stabilization, in which one or more optical components of the camera (e.g., the image sensor, the lens or one or more lens elements thereof, one or more prisms or mirrors) move laterally relative to the optical axis.

[0039] 2 shows a cross-sectional side view of an illustrative example of a camera 200 that may be used with the multi-camera systems described herein. The camera 200 may comprise a lens 202, an image sensor 220, and a housing 204, and may also include an actuator configured to move the lens 202 along the optical axis of the camera to adjust the focus of the camera and move the image sensor 220 across the optical axis for optical image stabilization. To move the lens 202 (which may include one or more lens elements and a lens barrel, as discussed above), the lens 202 may be attached to the coil 212 (e.g., via a lens carrier 210 configured to hold both the lens 202 and the coil 212). The lens 202 and lens carrier 210 may be suspended relative to a stationary portion of the camera 200 (relative to the magnet holder 206) via one or more suspension elements (not shown), such as one or more flexures (e.g., leaf spring(s), suspension wire(s), flexure arm(s), etc.) and / or one or more bearings (e.g., ball bearing(s), roller bearing(s), etc.). The magnet holder 206 may hold one or more magnets 208, and the coil 212 may be positioned within the magnetic field of the magnet 208 such that when a current is driven through the coil 212, a Lorentz force is generated that may cause relative movement between the coil 212 and the magnet 208, thereby moving the lens 202 along the optical axis of the camera.

[0040] To move the image sensor 220 in one or more directions perpendicular to the optical axis, the camera 200 may include a stationary base 214 (which may be fixed relative to the housing 204) and a frame 222 that movably connects the image sensor 220 to the base 214 via a number of flexures 226. In some cases, the flexures 226 may support electrical traces 224, which may be used to communicate signals to and from the image sensor 220. Also shown is a printed circuit 218 that carries one or more coils 216 (separate from the coils 212 carried by the lens carrier 210). The coils 216 may be positioned within the magnetic field of one or more magnets (e.g., magnet 208 in the variation shown in FIG. 2), so that when a current is driven through the coils 216, a Lorentz force is generated that can cause relative movement between the coils 216 and the magnets 208, thereby moving the image sensor 220 perpendicular to the optical axis of the camera.

[0041] The camera 200 may further comprise a position sensor 228, which is configured to detect the relative position of the lens 202 within the camera 200. For example, the position sensor may include a magnetic sensor (e.g., a Hall effect sensor, a tunneling magnetoresistance (TMR) sensor, etc.) arranged to measure the magnetic field of a magnet (e.g., a probe magnet 230 as shown in FIG. 2) within the camera 200. The position sensor 228 may be fixed relative to the lens 202 (e.g., held by the lens carrier 210) and the probe magnet 230 may be fixed relative to the magnet holder 206, or vice versa. As the lens 202 moves to adjust the focus of the camera 200, the position sensor 228 moves relative to the probe magnet 230 and measures the change in the change in magnetic flux as it moves through the magnetic field of the probe magnet 230. This may then measure the relative position of the lens 202 within the camera 200. Although position sensor 228 is discussed above as detecting the position of moving lens 202, it should be understood that the cameras of the multi-camera systems described herein may include position sensors that measure the positions of any or all of the moving components of that camera (e.g., the image sensor, the lens, and / or the lens elements of the lens).

[0042] When a camera of a multi-camera system described herein is configured with an autofocus function, the camera may be able to automatically set the focus position of the camera and adjust the position and / or configuration of one or more optical components of the camera accordingly. The focus position may be determined by the camera in any suitable manner. In some cases, the camera may be configured to perform phase detect autofocus (PDAF), where at least some of the pixels of the image sensor are configured to limit the range of angles of incidence that may be collected by those pixels. These "asymmetric" pixels may be formed by splitting the camera pixels into multiple separate photodiodes or by selectively masking a portion of the photodiodes of the pixels. Each asymmetric pixel may preferentially receive light from a given direction and may group pixels associated with a common direction. One group of pixels has a different signal than another group of pixels when the image is not focused, but has a well-matched signal when the image is focused. Thus, the groups of pixels may provide information that may be used by the camera to adjust the focus of the image (e.g., using the phase difference between the groups of pixels). An example of PDAF used on a mobile device is described in U.S. Pat. No. 10,440,301, entitled “Image capture device, pixel, and method providing improved phase detection auto-focus performance,” which is incorporated by reference in its entirety.

[0043] Additionally or alternatively, the camera may be configured to select a focus position using contrast based autofocus (CBAF), where the intensity difference between nearby pixels of the image sensor is compared as a function of focus position. As the object moves into focus, the intensity difference between nearby pixels increases, so the focus position may be adjusted until a maximum (or near maximum) contrast is determined for the region of interest on the image sensor. It should be appreciated that the camera may perform multiple autofocus techniques and may select from different techniques depending on scene conditions, as discussed in more detail below.

[0044] Camera autofocus techniques can be used to set the camera focus position, but can also be used to estimate the distance of one or more objects in the scene. For example, when the camera is focused on an object in the scene, the camera focus position (e.g., the position of lens 202 in the variation of camera 200 described above with respect to FIG. 2) can be measured (e.g., via a position sensor such as position sensor 228 described above with respect to FIG. 2) and a predetermined relationship between the focus position and the scene distance can be used to estimate the object distance.

[0045] Returning to FIG. 1A, the multi-camera system 102 can be used to capture images during one or more capture modes (e.g., a photo mode in which still images can be captured, a video mode in which video can be captured, a panoramic mode in which panoramic photos can be captured, a portrait mode in which still photos with artificial blur applied can be captured, etc.). In general, during these modes, the device 100 can display a camera user interface that displays a "live preview." The live preview is a stream of images captured by the cameras of the multi-camera system, representing the field of view (which may be a subset of the field of view of one of the cameras) that is captured when the cameras initiate a media capture event. In other words, the live preview allows a user to see what portion of a scene is currently being imaged and to determine when to capture a photo or video.

[0046] When the camera initiates a media capture event (e.g., under some predefined conditions or user interaction gives a command by interacting with a shutter control on a user interface, by pressing a designated button on the device, by giving a voice command, etc.), the camera captures media depending on the current camera mode (e.g., captures a photo when in photo mode or captures a video when in video mode), which may then be stored locally on the device 100 or transmitted to a remote server for storage. It should be understood that in some cases there may be a frame buffer of image frames captured by the multi-camera system 102, and some frames collected prior to the start of the media capture event may be used in generating the captured media that is stored or transmitted.

[0047] During these capture modes, one or more cameras of the multi-camera system 102 may actively capture image frames during a given period of time, and the multi-camera system 102 may designate a "primary" camera (any other active cameras are referred to as "secondary" cameras). If only a single camera is active at a given time, that camera becomes the primary camera. However, it may be desirable for more than one camera to be active simultaneously, and typically the primary camera is prioritized in one or more ways. However, for purposes of this application, the primary camera is considered to be the camera from which the live preview is generated (i.e., frames captured by the camera are used as images for the live preview, but these frames may be cropped or otherwise modified when generating the live preview).

[0048] In some variations, the primary camera may be configured to capture image frames at a frame rate that is higher than the frame rate of the secondary camera. Additionally or alternatively, when a media capture event is initiated, the image frames captured by the primary camera are used to generate the captured media. In some cases, information from the secondary cameras (including frames captured by one or more secondary cameras) may be used to assist in the operation of the primary camera and / or to correct the image frames captured by the primary camera. As a non-limiting example, image frames from the secondary cameras may be combined with image frames from the primary camera (acting as a reference frame) as part of an image fusion operation to provide a single composite image.

[0049] Camera switching technique

[0050] In general, the multi-camera system 102 can switch between different primary cameras depending on the desired capture mode and / or scene conditions. For example, U.S. Patent No. 10,701,256, entitled "Switchover control techniques for dual-sensor camera system," which is incorporated herein by reference in its entirety, discusses techniques for switching between cameras in a multi-camera system based on a desired zoom value, an estimated scene brightness, and / or an estimated focus distance. While these techniques may be useful in many cases (and may be used in conjunction with the switching techniques described herein), it may be desirable to detach from the primary camera when an object of interest being imaged by the primary camera moves closer than the primary camera's minimum object distance, otherwise the object of interest would not be in focus in any images captured by the primary camera.

[0051] Thus, if the target object is closer than the camera's minimum object distance, the multi-camera system may be configured to change the primary camera to another camera with a shorter minimum object distance, thereby allowing the multi-camera system to continue to focus on the target object at a closer distance (e.g., via the new primary camera). For example, in the multi-camera system 102 described above with respect to FIGS. 1A-1D, the second camera 106 may be selected as the primary camera while imaging the target object under certain circumstances. If the target object moves closer than the second camera's 106 minimum object distance 122, selecting the first camera 104 as the primary camera may enable the first camera 104 to capture an image with the target object in focus (the second camera 106 can no longer focus on the target object).

[0052] To provide a smooth user experience, it may be desirable to maintain a consistent zoom level when selecting a new primary camera. Because the first camera 104 has a wider field of view 112 than the field of view 114 of the second camera 106, it may be necessary to crop the image from the first camera to match the field of view of the image captured by the second camera 106 (which may correspond to the full field of view 114 of the second camera 106 or a cropped version thereof). However, the image quality of the cropped image from the first camera 104 may be worse than an image captured by the second camera 106 with the same field of view. Therefore, it may be desirable to maintain the second camera 106 as the primary camera until the target object reaches the minimum object distance 122 of the second camera 106.

[0053] However, in practice, the multi-camera system 102 may not be able to perfectly calculate the distance to the target object (especially at scene depths where depth information from a depth sensor, such as the depth sensor 110, is unreliable or not available at all), which may result in a situation where the multi-camera system 102 erroneously calculates the target object as being farther away than the minimum object distance 122, when the target object is closer than the minimum object distance 122 of the second camera 106. In such a situation, the second camera 106 may be maintained as the primary camera (even though the first camera 104 is better suited to image the target object at that distance), which may cause the target object to be out of focus in the live preview and any media captured by the second camera 106. Thus, in one aspect of the invention, the multi-camera system 102 may be configured to switch primary cameras when imaging a target object near the camera's minimum object distance, while taking into account potential errors in calculating the distance to the target object.

[0054] Thus, the devices described herein may include a camera selection logic programmed to select a primary camera from the multiple cameras of a multi-camera system. In the case where the multi-camera system includes more than two cameras, there may be a single camera selection logic that selects a primary camera from all the cameras, or there may be multiple camera selection logics that each control switching between different pairs of cameras of the multi-camera system. For example, a first camera selection logic may select between the first camera and the second camera as the primary camera, and a second camera selection logic may select between the second camera and the third camera as the primary camera. In some cases, the first camera selection logic and the second camera selection logic may be active at different times. Additionally or alternatively, the first camera selection logic and the second camera selection logic may both be active in certain situations, such as when the second camera is the primary camera, and each may determine whether to switch the primary camera from the second camera to either the first camera or the third camera.

[0055] In general, the camera selection logic receives multiple inputs including object distance estimates from at least one camera in the multi-camera system, individual distance error information associated with each object distance estimate, and a minimum object distance of at least one camera in the multi-camera system. If the multi-camera system includes a depth sensor, the multiple camera inputs may optionally further include object distance estimates from the depth sensor (and in some of these variations may further include depth error information associated with the distance estimates from the depth sensor). The camera selection logic selects a camera of the multi-camera system as the primary camera based at least in part on these inputs.

[0056] For example, FIG. 3 illustrates a variation of camera selection logic 300 that may be used to select a primary camera from multiple cameras of a multi-camera system. By way of example, the camera selection logic 300 is described as being used to select one of the first camera 104 or the second camera 106 of the multi-camera system 102 (described above with respect to FIGS. 1A-1D), but it should be understood that the camera selection logic 300 may be configured to select between more than two cameras as the primary camera (e.g., the first camera 104, the second camera 106, and the third camera 108 of the multi-camera system 102). Furthermore, the camera selection logic 300 may be configured to switch between the first and second cameras as the primary camera when the object of interest is near the second camera's minimum object distance. It should be understood that what is considered "near" the minimum object distance depends on the accuracy of the object distance estimate used to select the primary camera, as discussed in more detail below.

[0057] As shown, the camera selection logic 300 may receive a first object distance estimate 302 to the target object calculated using the first camera 104 and a second object distance estimate 304 to the target object calculated using the second camera 106. The camera selection logic 300 may additionally receive first distance error information 306 associated with the first object distance estimate 302 and second distance error information 308 associated with the second object distance estimate 304. The camera selection logic 300 may further receive the minimum object distance 122 of the second camera 106. Optionally, the camera selection logic 300 may receive a third object distance estimate 310 to the target object from the third camera 108, as well as third distance error information 312 associated with the third object distance estimate 312. The camera selection logic 300 may also receive as input the minimum object distance 120 of the first camera 104 and / or the minimum object distance 124 of the third camera 108 (in a variation in which the camera selection logic 300 receives an object distance estimate 310 from the third camera 108). Additionally or alternatively, the depth sensor 110 may also provide an object distance estimate for the target object (e.g., the fourth object distance estimate 314 shown in FIG. 3 ) and a distance estimation error associated with the object distance estimate (e.g., the fourth distance error information 316).

[0058] Using these inputs, the camera selection logic 300 may be configured to select one of the first camera 104 and the second camera 106 as the primary camera (as shown by the first output 318 of FIG. 3). Furthermore, the camera selection logic 300 may be optionally configured to use the object distance estimate inputs (e.g., some or all of the first object distance estimate 302, the second object distance estimate 304, the third object distance estimate 310, and the fourth object distance estimate 314) and distance error information inputs associated with these distance estimate inputs to select a focus position of one or more of the cameras of the multi-camera system 102 (as shown by output 320). Additionally or alternatively, the camera selection logic may be optionally further configured to select a zoom level of the selected primary camera (as shown by output 322). For example, when switching the primary camera from the second camera 106 to the first camera 104, it may be configured to maintain the same zoom level that was set when the second camera 106 was the primary camera (and vice versa). In these cases, the camera selection logic may be configured to select a zoom level for the first camera that matches the zoom level selected for the second camera when it was the primary camera. Because the first camera 104 has a larger field of view than the second camera 106, this may involve cropping the image captured by the first camera 104 to achieve the selected zoom level, and the device 100 may display a cropped version of the image captured by the first camera 104 during live preview when the first camera 104 becomes the primary camera. It should be understood that the zoom level may also be adjusted in response to other camera operations (e.g., after receiving a user input that changes the zoom level or after changing the capture mode), and that the output 322 of the camera selection logic 300 may be adjusted accordingly.

[0059] The various inputs to the camera selection logic 300 and how the camera selection logic 300 may use these inputs are discussed in more detail below.

[0060] Target Object

[0061] As described above, the camera selection logic 300 may receive one or more object distance estimates to a target object. In particular, the target object may correspond to a portion of a scene selected by the device as a region of interest (ROI). One or more images (and optionally depth information) captured by one or more cameras of the multi-camera system may be analyzed to identify an ROI that acts as the target object. In some cases, an ROI is selected from an image (or images) captured by one camera of the multi-camera system, and a corresponding ROI is selected for the remaining cameras (and any depth sensors) of the multi-camera system. In these cases, the corresponding ROI may be adjusted to account for differences in field curvature and / or relative placement of the cameras and depth sensors within the device. For purposes of this application, the individual ROIs selected for each of the cameras and depth sensors associated with the target object are collectively referred to as the ROI of the target object.

[0062] In some cases, an object detection algorithm may be used to identify one or more objects and / or faces in a scene, and the multi-camera system may select an ROI associated with one or more of the detected objects and / or faces as the target object. If multiple objects are detected, the multi-camera system may select an ROI encompassing two or more objects (it should be understood that the ROI may include two sections that need not be adjacent), or may select an ROI corresponding to a single object (which may be selected from multiple objects based on one or more predetermined criteria). In some cases, the target object's ROI may be selected at least in part based on user input (e.g., a user may tap a portion of the screen to designate an object and / or a portion of the scene as the target object). In the absence of detected objects / faces and / or user input, the multi-camera system may set a default ROI as the target object. It should be understood that the target object may change over time as the scene changes and / or new user input is received.

[0063] Object Distance Estimation

[0064] Once a target object is selected, some or all of the cameras of the multi-camera system may be used to estimate the distance from the multi-camera system to the target object. For example, autofocus information from each camera may be used to calculate an individual object distance estimate for the target object. In particular, the cameras may be focused on the target object using one or more autofocus techniques (as discussed in more detail above), and a focus position at which the camera is focused on the target object may be measured (e.g., via a position sensor, discussed above). By utilizing a predetermined relationship between focus position and object distance (which may be unique to each camera), the measured focus position may be used to calculate an object distance estimate for the target object. It should be understood that when the camera selection logic is described as receiving an object distance estimate from a camera, the camera selection logic may receive as the distance object estimate a distance value or a focus position value (which may then be converted to a distance value, if desired, using a predetermined relationship between focus position and object distance for that camera).

[0065] If the target object becomes closer than the minimum object distance of the camera, the camera may no longer be able to focus on the target object. In some cases, the camera selection logic may not actively receive an object distance estimate from the camera, or may receive and indicate that the target object is less than the minimum object distance of the camera. In other cases, the camera selection logic may receive an extrapolated object distance estimate from the camera. This object distance estimate may be calculated using defocus information measured by the camera at a predetermined focus position where the target object may not be in focus. Since the camera may not be able to achieve a focus position that can focus the camera on the target object, the object distance estimate may be associated with an extrapolated focus position that represents a virtual focus position where the camera may focus on the target object. Since there may be a predetermined relationship between the extrapolated focus position and the object distance estimate (similar to the discussion above), the camera selection logic may receive either a distance value or a focus position value as the distance object estimate. Extrapolated focus positions and their corresponding object distance estimates are described in more detail below with respect to Figures 5 and 6.

[0066] If the multi-camera system includes a depth sensor, the depth sensor of the multi-camera system can be used to estimate the distance from the multi-camera system to the target object. In these cases, the depth sensor can calculate depth information for a portion of the field of coverage that corresponds to the ROI of the target object. It should be understood that if depth information is available for multiple points within the ROI, i) the depth information for these multiple points can be provided to the camera selection logic, or ii) this depth information can be down-selected, averaged, or otherwise processed to generate a reduced number of distance values ​​(e.g., a single distance value representing the object distance estimate from the depth sensor).

[0067] Distance error information

[0068] As mentioned above, object distance estimates from the cameras and depth sensors of a multi-camera system may not be perfectly accurate, and the accuracy of these object distance estimates may vary from scene to scene, unit to unit, and given time unit. For example, when position sensor measurements are used to calculate the focus position of the camera as discussed above, there may be inaccuracies associated with the position sensor measurements, so the measured focus position may not reflect the actual focus position of the camera (and the corresponding object distance estimate may not reflect the actual distance to the target object). Therefore, the device may be configured to maintain distance error information for each camera, which may be used by the camera selection logic when selecting a primary camera.

[0069] The distance error information may be any suitable metric or metrics that provide an indication of the accuracy of the object distance estimation of that camera. This distance error information typically varies with the object distance estimate of a given camera. For example, the predetermined relationship between the focus position and a given object distance estimate may be non-linear, so that the same inaccuracy at two different focus positions will have different effects on the corresponding object distance estimate dependence. In general, this predetermined relationship will result in more accurate object distance estimates from a camera as the focus position approaches macro focus, which is located in relation to the minimum object distance of the camera.

[0070] In some cases, the distance error information of the camera may include a focus position error distribution that represents the probability of different measurement errors occurring for a given focus position measurement. This error distribution may be initially generated during factory calibration (e.g., by taking multiple focus position measurements when autofocusing on a target object at a known distance). In some variations, the focus position error distribution (and thus the distance error information) may be considered static, in that the position error distribution is not updated during operation of the device (although new focus position error distributions may be generated during subsequent calibration events). In other variations, the focus position error distribution (and thus the distance error information) may be dynamically updated during operation of the device. For example, if a multi-camera system includes a depth sensor, measurements from the depth sensor that meet a predefined accuracy / confidence threshold may be treated as a ground truth distance of the target object, which may be compared to a calculated object distance estimate of the target object using the focus position measured or extrapolated by the camera, and the results of this comparison may be used to update the focus position error distribution.

[0071] While the above examples of distance error information include a focus position error distribution for a given focus position measurement, it should be understood that this can be converted to an object distance error distribution for a given object distance estimate (e.g., using a predetermined relationship between the focus position and object distance estimates for a given camera). It should be understood that the camera selection logic can receive as input either a focus position error distribution (which can then be converted to an object distance error distribution, if desired, using a predetermined relationship between the focus position and object distance estimates for that camera), or an object distance error distribution. Additionally or alternatively, the device can maintain an error distribution (e.g., a focus position error distribution or an object distance error distribution), but the camera selection logic can receive as input one or more metrics derived from that error distribution (e.g., the mean and standard deviation of the distribution).

[0072] Similarly, when a depth sensor is used to provide an object distance estimate for a target object, the object distance estimate may be associated with corresponding distance error information, which may represent the accuracy of the object distance estimate provided by the depth sensor. The distance error information may include confidence information, such as one or more confidence values, as described in more detail above.

[0073] Minimum Object Distance

[0074] The minimum object distance of a camera may vary from unit to unit and in some cases may change during operation of the camera. The minimum object distance of a given camera may be initially measured during factory calibration and may be used by the camera selection logic as an input. It should be understood that although the value of the minimum object distance of a camera received by the camera selection logic is assumed to be the minimum object distance of the camera, in practice this value may differ from the actual minimum object distance (e.g., if a drop event or other event causes a permanent change in the relative position between the camera's image sensor and lens after the minimum object distance is measured).

[0075] In some variations, the minimum object distance of the camera may be considered static in that the minimum object distance is not updated during operation of the device (although a new minimum object distance may be measured during a subsequent calibration event). In other variations, the minimum object distance may change dynamically during operation of the device. For example, in some variations where the camera is configured to move the image sensor across the optical axis of the camera, the minimum object distance of the camera may change due to unintended movement of the image sensor along the optical axis during device operation. For example, in the variation of camera 200 described above with respect to FIG. 2, gravity may move image sensor 220 along the optical axis (e.g., via sagging of flexure 226 of frame 222) depending on the camera's orientation, which may change the minimum object distance of camera 200. In these cases, multi-camera system 102 may monitor changes in the orientation of camera 200 (e.g., as measured by the device's gyroscope, etc.) and update the minimum object distance during device operation based on these changes in camera orientation. In other cases, the camera may change to a different operating mode that uses a reduced range of focus positions during autofocus operations. In these cases, the change in the range of available focus positions may also change the minimum object distance for that camera, which may be reflected in the input to the camera selection logic. Optionally, the device may be configured to determine accuracy information associated with the camera's minimum object distance, which represents the likelihood that the minimum object distance value maintained by the device reflects the camera's actual minimum object distance. This accuracy information may be used as an input (not shown) to the camera selection logic 300 to assist in the selection of the primary camera.

[0076] It should be understood that some or all of the above inputs to the camera selection logic may change over time, and the method of selecting a primary camera using the camera selection logic may be repeated as the inputs to the camera selection logic change. For example, the camera selection logic may be repeated in multiple iterations as one or more object distance estimate inputs change (which may be updated when there is relative movement between the device and the scene), as distance error information inputs for a given object distance estimate change, and / or as a minimum object distance input for a given camera changes.

[0077] In general, the camera selection logic may be configured to change the primary camera closer to the minimum object distance of the camera while keeping below a threshold the probability that a camera is selected as the primary camera when the target object is closer than the minimum object distance of the camera. As a non-limiting example, if the camera selection logic 300 discussed above is used to select a primary camera from between the first camera 104 and the second camera 106, the camera selection logic 300 may be configured such that there is less than a 0.05% (or any other suitable predetermined probability) chance that the second camera 106 will be selected as the primary camera when the target object is closer than the minimum object distance 122 of the second camera 106. In this way, the distance error information of the object distance values ​​from the cameras of the multi-camera system may take into account the distance at which the camera selection logic allows the target object to reach the minimum object distance before switching. In other words, if the object distance estimate is very accurate, the camera selection logic may switch to a primary camera closer to the minimum object distance (because inaccuracies in the object distance estimate are less likely to cause an inaccurate determination that the target object is located farther than the minimum object distance). This can allow for different switching thresholds between different units or within a time unit, which can be particularly advantageous when the depth sensor of the multi-camera system cannot provide reliable depth information of the distance at which the target object is located.

[0078] The camera selection logic 300 may use any suitable technique for selecting a primary camera from the inputs discussed above. For example, in some variations, the camera selection logic 300 may include a neural network trained to obtain the minimum object distance(s), the object distance estimate(s), and associated distance error information, and output the primary camera selection.

[0079] As another example, Figure 4 shows a flowchart depicting an example method 400 that the camera selection logic 300 of Figure 3 may use to select a primary camera from multiple cameras of a multi-camera system. Although shown selecting one of a first camera and a second camera as the primary camera (wherein the second camera has a longer minimum object distance than the first camera), it should be understood that the description of method 400 may be extended to select a primary camera from among three or more cameras.

[0080] As shown in FIG. 4, the camera selection logic 300 may be configured to make an initial primary camera determination at step 402. This may occur when the multi-camera system is first initialized to begin image capture operations or when the multi-camera system switches between different capture modes (such as those described in more detail above). In some cases, there may be a default camera (e.g., a second camera) that is always selected (or a separate default camera for each capture mode). In other examples, the initial selection may depend on the scene content. For example, the multi-camera system may perform a start-up operation in which one or more cameras capture one or more preliminary images before starting live preview (e.g., to identify objects of interest in the scene, to set initial autofocus, color balancing, and / or exposure settings, etc.), and an initial primary camera selection may be made during this start-up (e.g., by looking at one or more object distance estimates to the object of interest).

[0081] At step 404, the camera selection logic 300 may determine one or more switching thresholds based on one or more of the inputs to the camera selection logic 300, which may be used in selecting a primary camera. The camera selection logic 300 determines the switching threshold (e.g., using a look-up table, a predefined relationship between the inputs and the switching threshold(s), etc.) using at least i) the minimum object distance of at least one camera (e.g., the minimum object distance of the second camera 106) and ii) distance error information associated with at least one of the object distance estimates. While the camera selection logic 300 may determine a given switching threshold using distance error information associated with a single object distance estimate, in some cases it may be preferable to use distance accuracy information associated with two or more object distance estimates when calculating a given switching threshold. In some cases, the second threshold may be set to the first threshold using a predefined correction, but for purposes of this application, the second threshold will be considered to have been determined using the same inputs used to determine the first threshold. By utilizing the camera's distance accuracy information and minimum object distance, the selection of the switching threshold(s) may be tailored to achieve switching near the minimum object distance while keeping the probability that the camera will be selected as the primary camera below the threshold when the target object is closer than the camera's minimum object distance, as discussed above.

[0082] In some cases, the camera selection logic 300 may be configured to calculate multiple sets of switching thresholds, with each set of switching thresholds used to control a different switching between two cameras of the multi-camera system 102 as the primary camera. For example, the switching thresholds calculated in step 404 may include a first set of switching thresholds that may be used to determine whether to change the primary camera from the second camera to the first camera, and a second set of switching thresholds that may be used to determine whether to change the primary camera from the first camera to the second camera. In step 406, the camera selection logic 300 may determine which camera of the multi-camera system is currently the primary camera. If the second camera is the primary camera, the method may determine whether to maintain the second camera as the primary camera or switch to the first camera as the primary camera. If the first camera is the primary camera, the method may determine whether to maintain the first camera as the primary camera or switch to the second camera as the primary camera.

[0083] Specifically, if the second camera is the primary camera, the camera selection logic 300 may compare one or more of the subject distance estimates to a first set of switching thresholds at step 408. If this comparison satisfies a first switching criterion (shown in step 410), the camera selection logic 300 may switch the primary camera to a different camera (e.g., select the first camera as the primary camera, as shown in step 412). If the comparison fails to meet the first switching criterion, the camera selection logic 300 may maintain the second camera as the primary camera (as shown in step 414). After the camera selection logic 300 selects the first camera as the primary or maintains the second camera as the selected primary camera, the camera selection logic 300 may maintain that camera as the selected primary camera until one or more of the inputs to the camera selection logic 300 are updated (at step 416), at which point a new iteration of the method 400 may begin again at step 404.

[0084] In one preferred embodiment, the first set of switching thresholds includes a single switching threshold determined using the minimum object distance 122 of the second camera 106, first distance accuracy information 306 associated with a first object distance estimate 302 of the target object calculated using the first camera 104, and second distance accuracy information 308 associated with a second object distance estimate 304 of the target object calculated using the second camera 106. In these variations, each of the first object distance estimate 302 and the second object distance estimate 304 may be compared (e.g., at step 408) to this switching threshold, and if at least one of the first object distance estimate 302 and the second object distance estimate 304 is less than the switching threshold, the camera selection logic 300 may determine that the first switching criterion is met (thereby switching the primary camera to the first camera 104).

[0085] In other embodiments, the first set of switching thresholds may include multiple thresholds. In these embodiments, different object distance estimates may be compared to different switching thresholds of the first set of switching thresholds. For example, the first object distance estimate 302 may be compared to a first switching threshold and the second object distance estimate 304 may be compared to a second switching threshold. The camera selection logic 300 may determine that the first switching criterion is met (thereby switching the primary camera to the first camera) if either the first object distance estimate 302 is less than the first switching threshold or the second object distance estimate 304 is less than the second switching threshold (or either of them are less than their corresponding switching thresholds). It should be understood that if the third camera 108 and / or the depth sensor 110 are used to generate object distance estimates, these object distance estimates may also be compared to one or more of the first set of switching thresholds in step 408, which may be further used to determine whether the first switching criterion is met in step 410.

[0086] Similarly, if the first camera 104 is the primary camera at step 406, the camera selection logic 300 may compare one or more of the subject distance estimates to a second set of switching thresholds at step 420. If this comparison satisfies a second switching criterion (shown in step 422), the camera selection logic 300 may switch the primary camera to a different camera (e.g., select the second camera 106 as the primary camera, as shown in step 424). If the comparison fails to meet the first switching criterion, the camera selection logic 300 may maintain the first camera 104 as the primary camera (as shown in step 414). After the camera selection logic 300 selects the second camera 106 as the primary or maintains the first camera 104 as the selected primary camera, the camera selection logic 300 may maintain that camera as the selected primary camera until one or more of the inputs to the camera selection logic 300 are updated (at step 416), at which point a new iteration of the method 400 may begin again at step 404.

[0087] The second set of switching thresholds may include a single switching threshold or multiple switching thresholds, as discussed in more detail above. For example, step 420 may include comparing the first object distance estimate 302 and the second object distance estimate 304 to a first switching threshold of the second set of switching thresholds. In some of these variations, the second switching criterion is met if both the first object distance estimate 302 and the second object distance estimate 304 exceed the first switching threshold (or alternatively, the second switching criterion is met if at least one of the first object distance estimate 302 and the second object distance estimate 304 exceeds the first switching threshold). In other variations, the second set of switching thresholds may include multiple thresholds, with the first object distance estimate 302 being compared to a first switching threshold of the second set and the second object distance estimate 304 being compared to a second switching threshold of the second set.

[0088] One or more switching thresholds in the first set may be the same as one or more switching thresholds in the second set, but it may be desirable to select different switching thresholds between the first set and the second set to add hysteresis to the switching behavior. For example, a first switching threshold belonging to the first set (which may be the only switching threshold in the first set) and a second switching threshold belonging to the second set (which may be the only switching threshold in the second set) may be selected such that they are separated by a predetermined separation amount. In some of these cases, the first switching threshold may be determined as discussed above (e.g., using distance error information from at least one minimum object distance and at least one object distance estimate), and the second switching threshold may be determined by adding the predetermined separation amount to the first switching threshold. It should be understood that the predetermined separation amount may be static or may change over time. For example, the predetermined separation amount may be adjusted based on device movement (e.g., the amount of device movement that may be measured in a predetermined period of time prior to calculating the switching threshold). In this way, the switching behavior can be configured to have more hysteresis when device movement is greater (e.g., via hand shake) compared to when device movement is smaller (e.g., when the device is placed on a tripod).

[0089] It should be understood that the camera selection logic may be configured to select and execute different methods of selecting a primary camera from the cameras of the multi-camera system under different circumstances. For example, the camera selection logic may be configured to select a first method under some circumstances that uses object distance estimates from multiple cameras of the multi-camera system as inputs (such as discussed above with respect to FIG. 4). If one of the cameras is unable to provide a corresponding object distance estimate, e.g., if that camera's position sensor is not operating as intended (e.g., if it is experiencing a certain amount of external magnetic interference), the camera selection logic may be configured to select a second method that does not require an object distance estimate from that camera.

[0090] Returning to FIG. 4, if a camera other than the second camera 106 is the primary camera, the second camera 106 may optionally be locked to a predetermined focus position. This may be desirable when the object of interest is at or below the second camera's 106 minimum object distance 122, so that the second camera 106 may be maintained at a focus position near the macro focus position. This may allow the second camera 106 to be in a preferred position when the object of interest moves beyond the second camera's 106 minimum object distance 122, and the camera selection logic 300 again selects the second camera 106 as the primary camera. In these cases, the second camera 106 may be prevented from moving away from the predetermined focus position even if the object of interest remains below the second camera's 106 minimum object distance 122, for example, if the autofocus function of the second camera 106 erroneously calculates the focus position of the second camera 106 to move away from the macro focus position.

[0091] Although shown as step 418 in FIG. 4 as occurring after step 406, it should be understood that this locking may occur at any time after a different camera (e.g., the first camera 102) is selected as the primary camera (e.g., immediately after step 412). When the second camera 106 is locked to the predetermined focus position, this predetermined focus position is preferably a macro focus position of the second camera 106 (i.e., a focus position associated with the second camera's minimum object distance), although other predetermined focus positions may be used. The second camera may be automatically locked to the predetermined focus position when the second camera 106 ceases to be the primary camera (e.g., when the first camera 104 is selected as the primary camera), or may be locked only if one or more locking criteria are met in addition to the second camera 106 ceasing to be the primary camera. For example, in some variations, one or more locking criteria may be met (and the second camera 106 may be locked to the predetermined focus position) when the object distance estimate 314 from the depth sensor 110 is less than a threshold locking distance. In some of these variations, the second camera 106 may not be released from the predetermined focus position until it is again selected as the primary camera (and thus the focus position may be adjusted to focus on the object of interest) and / or one or more release criteria are met. For example, the second camera 106 may be released from the predetermined focus position (even while not a primary camera) if the depth object estimate 314 from the depth sensor 110 exceeds a threshold unlock distance. In some cases, there may be a predetermined separation between the threshold lock distance and the threshold unlock distance, which may add hysteresis to the locking behavior of the second camera.

[0092] It should be understood that when the camera selection logic 300 selects a new camera as the primary camera, in some cases there may not be an instantaneous switch between the primary cameras. Specifically, either step 412 or 424 (or both steps 412 and 424) of the method 400 discussed above may include a transition sub-step to transition from the current primary camera to the newly selected primary camera. This may include maintaining the current primary camera as the primary camera until the newly selected primary camera meets one or more transition criteria (at the time the primary camera is switched to the newly selected primary camera). In some variations, the primary camera is directed to focus on the target object during the transition step, and the transition criteria may include determining whether the newly selected primary camera is focused on the target object (or is more closely focused on the target object than the current primary camera).

[0093] Using step 412 as an example, at the moment the camera selection logic 300 determines that the first camera 104 is the primary camera, the first camera 104 may not yet be focused on (or even near) the object of interest. If the camera selection logic 300 were to instantly switch to the first camera 104 as the primary camera, the object of interest in the live preview may be blurrier than if the live preview continued to show image frames from the second camera 106, which may be an undesirable user experience. To address this, the selection logic 300 may initiate a transition step in which the second camera 106 is maintained as the primary camera until the first camera 104 meets a transition criterion, at which point the primary camera switches back to the first camera 104. In some variations, this transition criterion may be whether the first camera 104 has achieved a focus position that corresponds to the current focus position of the second camera 106 (with the intent that the target object is at least in focus in the image from the first camera 104 as it is in the image captured by the second camera 106). In other variations, the transition criterion may be whether the first camera 104 has achieved a focus position that corresponds to the target object (i.e., whether the first camera 104 is focused on the target object).

[0094] It should be appreciated that during this transition step, the newly selected primary camera may focus on the subject on the target object in any suitable manner. In some cases, the newly selected primary camera may focus on the target object using its camera's autofocus functionality (e.g., using the autofocus functionality of the first camera 104 discussed in step 412 above). In other cases, the newly selected primary camera may focus the newly selected primary camera using one or more of the autofocus assistance techniques discussed below with respect to Figures 5 and 6. In particular, as discussed in more detail below, calculating an extrapolated focus position using depth-from-defocus (which may be used in conjunction with a focus relationship module) may have particular utility at short subject distances where other autofocus techniques may not be available. For example, in some cases, these autofocus assistance techniques may be used to set the focus position of the newly selected camera or to prompt the camera to perform a partial scan, where parameters of the partial scan are based at least in part from the autofocus assistance techniques (discussed in more detail below).

[0095] It should also be appreciated that during the transition step, the current primary camera may be locked at a predefined focus position. For example, when transitioning from the second camera 106 to the first camera 104 as the primary camera, the second camera 106 may be locked at a predefined focus position (which may preferably be a macro focus position for the second camera 106) until the first camera 104 meets the transition criteria discussed above. When the transition step is completed (and the second camera 106 is no longer the primary camera), the second camera 106 may be released and the second camera 106 may be locked at the predefined focus position discussed above with respect to step 418. The predefined focus position at which the second camera 106 is locked during the transition step may be the same as or different from the predefined focus position at which the second camera 106 is locked after the first camera 104 becomes the primary camera.

[0096] Focus related Autofocus

[0097] Also described herein are techniques for calculating an object distance estimate to a target object using the cameras of a multi-camera system when the target object is located closer than the minimum object distance of the camera. The distance object estimate may be calculated by measuring defocus information associated with the target object while the camera is held at a predefined focus position. In some cases, this distance object estimate may be used as an input to a camera selection logic for selecting a primary camera of the multi-camera system, as described in more detail above. Additionally or alternatively, the distance object estimate may be used to determine an extrapolated focus position (as described above), which may then be used to help set focus positions of different cameras of the multi-camera system.

[0098] For example, when capturing images with a primary camera (e.g., to generate a live preview and / or capture media as discussed above), the autofocus function of the primary camera may have limited applicability in certain situations. For example, contrast-based autofocus techniques typically require the camera to scan through a range of focus positions, which may disrupt the user experience and may be relatively slow when compared to other autofocus techniques such as PDAF. On the other hand, PDAF may encounter difficulties in low light conditions (which may make it difficult to measure the phase difference between different asymmetric pixels) or when the target object has a flat texture. Thus, in these cases, it may be desirable to obtain additional information that may assist in setting the focus position of the primary camera.

[0099] For example, in some cases, depth information from a depth sensor (e.g., depth sensor 110 of multi-camera system 102 described above with respect to FIGS. 1A-1D) may be used to provide an object depth estimate for the target object (as discussed in more detail above), which may then be used in setting the focus position of the primary camera. If the primary camera images a target object that is less than or near the minimum distance of the depth sensor's operating range, the depth information may not be available or reliable enough to be used in setting the focus position of the primary camera.

[0100] In other cases, it may be desirable to use focus position information from a first camera to assist in setting the focus position of a second camera. For example, U.S. Patent No. US10,429,608, entitled "Primary-subordinate camera focus based on lens position sensing," incorporated herein by reference in its entirety, discusses using a first camera to focus on an object at a first focus position, and then selecting a second focus position for the second camera based at least in part on the first focus position (such as using a "focus relationship" between the first and second cameras). While this technique may be beneficial in many cases, it may not be available when the object is less than the first camera's minimum object distance (and therefore the first camera cannot focus on the object). Thus, the techniques described herein may have particular utility in cases where these other autofocus assistance techniques are not available.

[0101] 5 illustrates an embodiment of focus control logic 500 that can cause a first camera to focus on a target object based at least in part on defocus information measured by a second camera. For illustrative purposes, this focus control logic 500 is discussed herein as being used with a first camera 104 and a second camera 106 of a multi-camera system 102. As shown, the second camera 106 can include a PDAF processing module 502 that can be configured to perform PDAF and generate focus information associated with a plurality of points within a field of view 122 of the second camera 106. This focus information can include a focus value that represents how focused / defocused the incident light is at a given point in the scene.

[0102] In some cases, the focus information from the PDAF processing module 502 can be used to set a focus position of the second camera 106 using the focus position controller 504. Specifically, the focus position controller 504 can control the focus position of the second camera 106 (e.g., by controlling an actuator to move an image sensor and / or a lens of the second camera 106) and can also measure a current focus position of the second camera 106. The focus position controller 504 can be configured to perform PDAF to adjust the focus position of the second camera 106 using the focus information from the PDAF processing module 502 to focus the second camera 106 on a target object.

[0103] It should be appreciated that the focus position controller 504 can provide the measured focus position (or object distance derived therefrom) as an input to the camera selection logic, which can be, for example, the second object distance estimate 304 received by the camera selection logic 300 described above with respect to FIG. 3 (also partially reproduced in FIG. 5) and used to select the primary camera (at output 318). The camera selection logic 300 can then provide (at output 320) a target focus position to the focus position controller 504, which can set the second camera to the target focus position. This can be completed, for example, when the camera selection logic is configured to lock the second camera to a predetermined focus position, as discussed above with respect to FIG. 4.

[0104] The second camera 106 may further include a depth-from-defocus module 506, which may receive focus information from the PDAF processing module 502 and a current focus position from the focus position controller 504, and may generate distance information, such as a depth map, based on the amount of defocus in the focus information received from the PDAF processing module. Specifically, for each focus position of the second camera 106, there may be a predefined relationship between defocus and scene distance. Such a predefined relationship, which may be determined experimentally during device calibration, may be used to convert the defocus information of a point in a scene to the distance of that point in the scene.

[0105] In some cases, the depth defocus module 506 may be used to generate distance information associated with the target object, which may be provided to the camera selection logic 300 as an object distance estimate 304 from the second camera 106, which may be used by the camera selection logic in selecting the primary camera. Similarly, the depth defocus module 506 may be configured to output distance error information associated with the depth information generated by the depth defocus module 506 (as discussed in detail above with respect to FIG. 3). This distance error information may be depth error information 308 associated with the object distance estimate 304 from the second camera 106 that is received by the camera selection logic 300 as an input (not shown in FIG. 5A).

[0106] In other cases, the defocus module 506 may output distance information to the focus control logic 500. As shown, the focus control logic 500 may include multiple modules that may assist in focusing one or more cameras of a multi-camera system. For example, in the variation of the focus control logic 500 shown in FIG. 5, the focus control logic 500 may provide a target focus position and / or one or more focus instructions to the focus position controller 512 of the first camera 104. Although two modules (depth sensor assistance module 508 and focus relationship module 510) are shown in FIG. 5, it should be understood that the focus control logic 500 may include more or fewer modules. The focus control logic 500 may utilize different modules under different circumstances and may prioritize or combine the outputs of the various modules in providing a target focus position and / or one or more focus instructions to the focus position controller 512 of the first camera 104. Additionally, although the focus control logic 500 is shown in FIG. 5 only to assist in focusing the first camera 104, it should be understood that the focus control logic 500 (or different focus control logic) may be used to assist in focusing other cameras in the multi-camera system 102.

[0107] In a variation in which the focus control logic 500 includes a depth sensor assistance module 508, the depth sensor assistance module 508 can receive distance information (e.g., an object distance estimate to the target object) from the depth sensor 110, and the depth sensor assistance module 508 can select a target focus position of the first camera 104 that corresponds to the distance information received from the depth sensor 110.

[0108] In variations in which the focus control logic 500 comprises a focus relationship module 508, the second camera 106 may focus on the target object (e.g., using autofocus techniques as described in more detail above) and measure the focus position using the focus position controller 504 when the target object is in focus. The thus measured focus position of the second camera 106 may be provided to the focus relationship module 508, which can select a target focus position for the first camera 104 to focus the first camera 104 on the target object. The focus relationship module may use a predetermined relationship (such as the focus relationship discussed in U.S. Pat. No. 10,429,608, previously incorporated by reference in its entirety) that maps the focus positions of the second camera to respective target focus positions on the first camera.

[0109] As mentioned above, if the target object is less than the minimum object distance 122 of the second camera 106, the second camera 106 cannot focus on the target object, so any focus position measured by the focus position controller 504 will not be accurate enough to generate a target focus position for the first camera 104 using the focus relationship module 510. To address this, the target object's distance information generated by the depth defocus module 506 (generated from the focus information from the PDAF processing module 502 and the measured focus position from the focus position controller 504, as discussed above) can be used to generate an extrapolated focus position (which represents a virtual focus position at which the camera would be able to focus on the target object, as discussed above). This extrapolated focus position (which may exceed the macro focus position and therefore may not be a focus position achievable by the second camera 106) can then be used by the focus relationship module 510 instead of the measured focus position from the focus position controller 504 to generate a target focus position for the first camera 104. The focus relationship module 510 may also receive distance accuracy information from the depth defocus module 506, which may affect how the focus relationship module 510 assists in focusing the first camera, as described below with respect to FIG. 6.

[0110] 6 shows a flowchart illustrating an embodiment of a method 600 for calculating an object distance estimate to an object of interest using defocus information from a camera of a multi-camera system. The method 600 may further use the object distance estimate from a camera to assist in focusing another camera of the multi-camera system. For purposes of explanation, the method 600 is described in the context of the first camera 104 and the second camera 106 of the multi-camera system 102 discussed above with respect to FIGS. 1A-1D.

[0111] The method 600 may begin by determining whether the focus position of the second camera 106 is maintained at a predetermined focus position at step 602. In some cases, this may include determining that the second camera 106 is locked at a focus position (i.e., such that the focus position does not change as a function of changes in the scene), such as when the camera selection logic 300 locks the focus position of the second camera 106 when the second camera 106 is not the primary camera (as discussed in more detail above). In other cases, the predetermined focus position is a macro focus distance of the second camera 106, and determining whether the second camera 106 is maintained at the predetermined focus position includes determining whether the target object is closer than the minimum object distance 122 of the second camera 106. For example, the autofocus operation of the second camera 106 can set the second camera 106 to a macro focus position when attempting to focus on a target object that is closer than the minimum object distance 122 of the second camera 106, and the second camera 106 can remain in the macro focus position while the target object is closer than the minimum object distance 122 of the second camera 106.

[0112] If the second camera 106 is not maintained at the predetermined focus position, the second camera 106 may be focused on the object of interest using autofocus techniques at step 604. If the second camera 106 is focused on the object of interest, a focus position for the second camera 106 may be determined at step 606. This focus position may be used to assist in focusing the first camera 104, such as in step 608, where the first camera is focused based at least in part on the focus position determined at step 606 (e.g., using focus relationship module 508 of camera focus logic 500 described above with respect to FIG. 5).

[0113] Conversely, if the second camera 106 is maintained at the predetermined focus position, the second camera 106 may measure defocus information of the target object (which in some cases may be located closer than the minimum object distance 122 of the second camera 106) in step 610. This may be done, for example, using the PDAF processing module 502 discussed above. In step 612, an object distance estimate to the target object may be calculated using the defocus information and the predetermined focus position (e.g., the depth defocus module 508 described above with respect to FIG. 5 may perform this calculation).

[0114] In addition, the object distance estimate may be used to calculate an extrapolated focus position of the second camera in step 614, as discussed in more detail above. Such extrapolated focus position may be used to assist in focusing of the first camera 104, such as in step 616, where the first camera is focused based at least in part on the extrapolated focus position calculated in step 614 (e.g., using focus relationship module 508 of camera focus logic 500, described above with respect to FIG. 5).

[0115] In some variations, step 616 may further include comparing the distance error information associated with the object distance estimate to one or more focus selection criteria at 618. If the distance error information satisfies the focus selection criteria, the first camera 102 may be set to a target focus position at step 620 (which may then be calculated based on the extrapolated focus position discussed above). If the distance error information does not satisfy the focus selection criteria, the first camera 102 may be configured to perform a partial scan (e.g., a partial CBAF scan), and the boundaries of the partial scan may be determined based on the extrapolated focus position. In this manner, the extrapolated focus position of the second camera may be used to directly set the focus position of the first camera in some cases (e.g., when there is more confidence in the accuracy of the object distance estimate), while in other cases the extrapolated focus position may guide the selection of the focus position of the first camera (e.g., when there is less confidence in the accuracy of the object distance estimate).

[0116] The foregoing description, for convenience of explanation, uses specific terminology to provide a thorough understanding of the described embodiments. However, it will be apparent to one skilled in the art upon reading this description that specific details are not required to practice the described embodiments. Thus, the foregoing descriptions of the specific embodiments described herein have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. It will be apparent to one skilled in the art upon reading this description that many modifications and variations are possible in light of the above teachings.

[0117] Object depth estimation and camera focusing techniques for multi-camera systems

[0118] [Claim 1] a first camera configured with an autofocus function and having a first minimum object distance; a second camera configured with autofocus capability and having a second minimum object distance; and focus control logic configured to cause the second camera to measure defocus information of a target object at a given focus position, to determine an extrapolated focus position using the defocus information, and to focus the first camera on the target object based at least in part on the extrapolated focus position; 16. A device comprising: [Claim 2] The device of claim 1 , wherein the predetermined focus position corresponds to the minimum object distance of the second camera. [Claim 3] The device of claim 1 , wherein the focus control logic locks the first camera at the predetermined focus position when one or more predetermined locking criteria are met. [Claim 4] 4. The device of claim 3, wherein the multi-camera system further includes a depth sensor, and the one or more predetermined criteria includes an object distance estimate of the target object being less than a threshold lock distance. [Claim 5] The device of claim 1 , wherein the first minimum object distance is less than the second minimum object distance. [Claim 6] 2. The device of claim 1, further comprising: camera selection logic configured to select either the first camera or the second camera as a primary camera, and the focus control logic is further configured to cause the second camera to measure the defocus information of the target object at the predetermined focus position when the first camera is selected as the primary camera. [Claim 7] 2. The device of claim 1, wherein focusing the first camera on the target object based at least in part on the extrapolated focus positions includes setting a target focus position of the first camera based on a predetermined relationship between the extrapolated focus positions when confidence information associated with the extrapolated focus positions meets a first predetermined criterion. [Claim 8] 1. A method for focusing a first camera of a multi-camera system using a second camera of the multi-camera system, comprising: Measuring defocus information of a target object using the first camera at a predetermined focus position; determining an extrapolated focus position using the defocus information; and focusing the first camera on the target object based at least in part on the extrapolated focus position; A method comprising: [Claim 9] The method of claim 8 , wherein the predetermined focus position corresponds to a minimum object distance of the second camera. [Claim 10] The method of claim 8 , further comprising locking the first camera to the predetermined focus position when one or more predetermined locking criteria are met. [Claim 11] 11. The method of claim 10, further comprising: calculating an object distance estimate for the target object using a depth sensor of the multi-camera system; and comparing the object distance estimate to a threshold lock distance, wherein the predetermined lock criterion is met when the object distance estimate is less than the threshold lock distance. [Claim 12] 9. The method of claim 8, wherein measuring defocus information of the target object using the first camera at the predetermined focus position includes measuring defocus information while the first camera is selected as a primary camera. [Claim 13] 9. The method of claim 8, wherein focusing the first camera on the target object based at least in part on the extrapolated focus positions comprises setting a target focus position of the first camera based on a predetermined relationship between the extrapolated focus positions when confidence information associated with the extrapolated focus positions meets a first predetermined criterion. [Claim 14] 14. The method of claim 13, wherein focusing the first camera on the target object based at least in part on the extrapolated focus position includes performing a partial scan based on the extrapolated focus position when confidence information associated with the extrapolated focus position does not meet a first predetermined criterion. [Claim 15] 1. A method for calculating an object distance estimate to a target object using a multi-camera system including a first camera and a second camera, comprising: determining whether the second camera is maintained at a predetermined focus position; in response to determining that the second camera is maintained at the predetermined focus position, measuring defocus information of the target object while the second camera is maintained at the predetermined focus position; calculating the object distance estimate using the defocus information and the predetermined focus position; A method comprising: [Claim 16] 16. The method of claim 15, wherein determining whether the second camera is maintained at a predetermined focus position comprises determining that the second camera is locked at the predetermined focus position. [Claim 17] 16. The method of claim 15, wherein the predetermined focus position is a macro focus position of the second camera, and determining whether the second camera is maintained at the predetermined focus position includes determining that the target object is closer than a minimum object distance of the second camera. [Claim 18] The method of claim 15 , further comprising: calculating an extrapolated focus position of the second camera using the object distance estimate. [Claim 19] 20. The method of claim 18, further comprising focusing the first camera based at least in part on the extrapolated focus position of the second camera. [Claim 20] 20. The method of claim 19, wherein focusing the first camera based at least in part on the extrapolated focus position comprises selecting a target focus position for the first camera based on a predetermined relationship between a focus position of the second camera and a focus position of the first camera.

Claims

1. 1. A method for selecting between a first camera of a multi-camera system having a first minimum object distance and a second camera having a second minimum object distance greater than the first minimum object distance as a primary camera during image capture of an object of interest, the method comprising: determining a first object distance estimate for the target object using the first camera, the first object distance estimate being associated with first distance error information; determining a second object distance estimate for the target object using the second camera, the second object distance estimate being associated with second distance error information; selecting one of the first camera and the second camera as the primary camera using the first object distance estimate, the second object distance estimate, the first distance error information, the second distance error information, and the second minimum object distance using camera selection logic; A method comprising:

2. The method of claim 1 , wherein the multi-camera system includes a depth sensor, and further comprising determining a third object distance estimate for the target object using the depth sensor.

3. The method of claim 2 , further comprising: holding a focus position of the second camera in a predetermined position when the third object distance estimate is less than a first threshold distance.

4. 2. The method of claim 1 , wherein selecting one of the first camera and the second camera as the primary camera includes setting a first switch threshold distance using the first distance error information, the second distance error information, and the second minimum object distance.

5. 5. The method of claim 4, wherein selecting one of the first camera and the second camera as the primary camera comprises selecting the first camera as the primary camera when either the first object distance estimate or the second object distance estimate is less than the first switch threshold distance.

6. 5. The method of claim 4, wherein selecting one of the first camera and the second camera as the primary camera includes setting a second switching threshold distance that is greater than the first switching threshold distance, and further includes selecting the second camera as the primary camera when both the first object distance estimate and the second object distance estimate are greater than the second switching threshold distance.

7. The method of claim 1 , wherein the second minimum object distance changes dynamically based on changes in orientation of the second camera.

8. a multi-camera system including a first camera having a first minimum object distance and a first field of view, and a second camera having a second minimum object distance and a second field of view that is greater than the first minimum object distance; a camera selection logic unit configured to select either the first camera or the second camera as a primary camera; The camera selection logic unit includes: receiving a first object distance estimate from the first camera and first distance error information associated with the first object distance estimate; receiving a second object distance estimate from the second camera and second distance error information associated with the second object distance estimate; receiving the second minimum object distance; the first object distance estimate, the second object distance estimate, the first distance error information, the second distance error information, and the second minimum object distance.

9. The device of claim 8 , wherein the first minimum object distance is less than the second minimum object distance and the first field of view completely encompasses the second field of view.

10. The device of claim 8 , wherein the multi-camera system further comprises a depth sensor configured to generate a third object distance estimate.

11. 11. The device of claim 10, wherein the multi-camera system is configured to hold a focus position of the second camera in a predetermined position when the third object distance estimate is less than a first threshold distance.

12. 9. The device of claim 8, wherein selecting the primary camera includes setting a first switch threshold distance using the first distance error information, the second distance error information, and the second minimum object distance.

13. 13. The device of claim 12, wherein selecting the primary camera comprises selecting the first camera as the primary camera when either the first object distance estimate or the second object distance estimate is less than the first switch threshold distance.

14. 13. The device of claim 12, wherein selecting the primary camera includes setting a second switching threshold distance greater than the first switching threshold distance, and further includes selecting the second camera as the primary camera when both the first object distance estimate and the second object distance estimate are greater than the second switching threshold distance.

15. 1. A method for selecting, during image capture of an object of interest, one of a plurality of cameras of a multi-camera system as a primary camera, each of the cameras having a respective minimum object distance, the method comprising: determining at least one object distance estimate for the target object using at least one of the plurality of cameras, each of the at least one object distance estimates being associated with respective distance error information; Use the camera selection logic to the at least one object distance estimate; the individual distance error information associated with each of the at least one object distance estimate; and selecting the primary camera from the plurality of cameras using at least one of the respective minimum object distances of the plurality of cameras; A method, wherein a second minimum object distance of a second camera of the multiple cameras is less than a first minimum object distance of a first camera of the multiple cameras.

16. The method of claim 15 , further comprising: updating the first minimum object distance; and selecting the primary camera using the updated first minimum object distance.

17. 16. The method of claim 15, further comprising: updating a first distance error information of the individual distance error information associated with each of the at least one object distance estimates; and selecting the primary camera using the updated first distance error information.

18. 16. The method of claim 15, wherein the multi-camera system further includes a depth sensor, and selecting the primary camera using the camera selection logic further includes selecting the primary camera using a first object distance estimate from the depth sensor.

19. 20. The method of claim 18, further comprising: holding a focus position of the first camera of the multiple cameras at a predetermined position when the first object distance estimate is less than a first threshold distance.

20. The method of claim 15 , further comprising using the camera selection logic to set a focus position for each of the plurality of cameras.

Citation Information

Patent Citations

  • Camera module, camera, camera control method and control program

    JP2014106274A

  • Imaging apparatus, control method of imaging apparatus and program

    JP2019133099A

  • Electronic apparatus, control unit of electronic apparatus, control program, and control method

    JP2019203929A

  • Intelligent photographing method and apparatus, and intelligent terminal

    US20190349535A1