Systems and methods for obtaining super macro images
By employing a multi-camera system on mobile devices, combining wide-angle and telephoto cameras, and optimizing image capture strategies using lens actuators and application processors, the problem of poor image quality in macro photography on mobile devices is solved, and high-quality macro image generation is achieved.
Patent Information
- Application Number
- CN202311778591.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-04-21
- Filing Date
- 2021-05-15
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-05-15
AI Technical Summary
Existing mobile devices struggle to capture high-quality images in macro photography, especially due to the shallow depth of field and large variations in object depth of telephoto cameras, making it difficult to achieve full-focus imaging, particularly for popular macro subjects such as insects and flowers.
A multi-camera system is employed, combining a wide-angle camera and a telephoto camera. The telephoto lens module is moved by a lens actuator to focus at a distance of 3.0 cm to 35 cm. The application processor analyzes the image data to define the capture strategy, generates new macro images, and optimizes the image quality through Gaussian Laplacian operator analysis and phase detection autofocus technology.
It achieves high-quality macro image capture on mobile devices, automatically detects and focuses on objects of interest, and generates clear macro images, overcoming the challenges of shallow depth of field and large changes in object depth that are common with telephoto cameras.
Smart Images

Figure CN117572549B_ABST
Abstract
Description
[0001] This application is a divisional application of application number 202180002905.0 (PCT application number PCT / IB2021 / 054186), filed on May 15, 2021, entitled "System and method for obtaining super macro images".
[0002] Cross-referencing of related applications
[0003] This application claims priority to the following U.S. provisional patent applications: No. 63 / 032,576, filed May 30, 2020; No. 63 / 070,501, filed August 26, 2020; No. 63 / 110,057, filed November 5, 2020; No. 63 / 119,853, filed December 1, 2020; No. 63 / 119,853, filed December 1, 2020; No. 63 / 164,187, filed March 22, 2021; No. 63 / 173,446, filed April 11, 2021; and No. 63 / 177,427, filed April 21, 2021, all of which are expressly incorporated herein by reference. Technical Field
[0004] The topics disclosed in this article generally relate to macro images, and in particular to methods for obtaining such images using a moving telephoto (“Tele” or “T”) camera. Background Technology
[0005] Multi-camera setups (with "dual-camera" setups having two cameras being one example) are now standard on portable electronic mobile devices ("mobile devices," such as smartphones, tablets, etc.). Multi-camera setups typically include a wide field of view (or "angle") of FOV. W A camera ("wide-angle" or "W" camera) and at least one additional camera, having (more than FOV) W Shallower FOV (FOV with telephoto camera) T ), or ultra-wide field of view (FOV) UW (Compared to FOV) W Wider angle (UW camera). Figure 10 A known dual-camera setup, including a W camera and a folding T camera, is shown.
[0006] "Macro photography" is becoming a popular differentiator. Macro photography refers to capturing objects so close to the camera that the image recorded on the image sensor is almost as large as the actual captured object. The ratio of object size to image size is the magnification of the object in the image. For system cameras such as digital single-lens reflex (DSLR) cameras, a macro image is defined as having an object magnification of approximately 1:1 or greater, such as 1:1.1. In the context of mobile devices, this definition is more relaxed, so images with an object magnification of approximately 10:1 or even 15:1 are also referred to as "macro images." Known mobile devices offer macro photography capabilities, typically by using ultra-wide-angle (UW) cameras to achieve very close focusing, with relatively short effective focal lengths (EFLs), such as EFL = 2.5mm.
[0007] Ultra-wide (UW) cameras can focus at close distances (e.g., 1.5 cm to 15 cm) required for macro photography, but their spatial resolution is poor. For example, when an UW camera with an EFL of 2.5mm focuses on an object at 5 cm (lens-object distance), the object-to-image magnification is approximately 19:1. This is based on the thin lens equation:
[0008]
[0009] With an EFL of 2.5mm, an object-lens distance v of 2.6mm, and an object-lens distance u of 50mm, even when focusing as close as 1.5cm, the object-to-image magnification ratio of an UW camera will be approximately 5:1. These short object-lens distances (e.g., u = 5cm or less) are very challenging for users—for example, they can make framing images extremely difficult, prevent the capture of popular macro objects such as live subjects (e.g., insects), and introduce shadows and blur the lighting in a scene.
[0010] A dedicated macro camera can be achieved using the telephoto camera of a smartphone. Telephoto cameras that focus on close-up objects have a very shallow depth of field (DOF). Therefore, capturing macro images in macro photography mode is very challenging. Popular macro objects, such as flowers or insects, have significant depth variations and cannot be imaged in full-in-focus in a single shot. It is beneficial to incorporate multiple cameras into a mobile device, which (i) captures macro images from a greater lens-to-object distance (e.g., 3.0–35 cm) and (ii) at a greater object-to-image magnification ratio (e.g., 1:5–25:1). Summary of the Invention
[0011] In the following text, and for the sake of simplicity, the terms "UW image" and "W image", "UW camera" and "W camera", and "UWFOV" (or FOV) will be used interchangeably. UW ) and "W FOV" (or FOV) W (These terms can be used interchangeably.) A W camera may have a larger FOV than a telephoto camera or a telephoto camera with macro capabilities, while a UW camera may have a larger FOV than a W camera. Generally, but not limited to, FOV... T It can be 15-40 degrees, FOV W It can be 60-90 degrees and FOV UW The FOV can be 90-130 degrees. A W or UW camera may be able to focus on the distance between the object and the lens relevant to macro photography, and may be within, for example, 2.5-15 cm. In some cases (e.g., between W and UW), the FOV ranges given above may overlap to some extent.
[0012] In various embodiments, a system is provided comprising: a wide-angle camera for providing at least one wide-angle image; a telephoto camera including a telephoto lens module; a lens actuator for moving the telephoto lens module to focus on any distance or set of distances between 3.0 cm and 35 cm, having an object magnification ratio between 1:5 and 25:1; and an application processor (AP) configured to analyze image data from the wide-angle camera to: define a capture strategy for capturing a sequence of macro images using the telephoto camera, wherein the sequence of macro images has a focus plane for transferring from one captured macro image to another; and generate new macro images from the sequence. The focus plane and depth of field of the new macro images can be continuously controlled. In some embodiments, continuous control can be post-capture.
[0013] In some embodiments, the telephoto camera may be a folding telephoto camera including an optical path folding element (OPFE). In some embodiments, the telephoto camera may be a dual-folding telephoto camera including two OPFEs. In some embodiments, the telephoto camera may be a pop-up telephoto camera including a pop-up lens.
[0014] In some embodiments, the focus may be an object-lens distance of 3.0 to 25 cm, 3.0 to 15 cm, or 10 to 35 cm.
[0015] In some embodiments, the telephoto camera may have an EFL of 7 to 10 mm, 10 to 20 mm, or 20 to 40 mm.
[0016] In some embodiments, a telephoto capture strategy can be adjusted during the capture of a sequence of macro images based on information from the captured macro images.
[0017] In some embodiments, information from the captured macro image is processed by Laplacian of Gaussian analysis.
[0018] In some embodiments, the image data from the UW camera is phase detection autofocus (PDAF) data.
[0019] In some embodiments, a UW image can be used as a reference image for the generation of a new macro image.
[0020] In some embodiments, the generation of a new macro image can use a video stream of a UW image as a reference image.
[0021] In some embodiments, the AP can be configured to automatically detect the object of interest (OOI) in a sequence of captured macro images and generate a new macro image when the OOI is fully in focus.
[0022] In some embodiments, the AP can be configured to automatically detect the Out-of-focus area (OOI) in the UW image data and generate a new macro image when the OOI is fully in focus.
[0023] In some embodiments, the AP can be configured to automatically detect OOI in a sequence of input macro images and generate new macro images when a specific image segment of the OOI has a specific amount of forward defocus blur and a specific amount of backward defocus blur.
[0024] In some embodiments, the AP can be configured to: automatically detect OOI in UW image data, and generate a new wide-angle image when a specific image segment of the OOI has a specific amount of forward defocus blur and a specific amount of backward defocus blur.
[0025] In some embodiments, the AP can be configured to compute a depth map from a sequence of captured macro images and use the depth map to generate new macro images.
[0026] In some embodiments, the AP can be configured to provide new macro images with real artificial lightning scenes.
[0027] In some embodiments, the AP can be configured to analyze image data from a wide-angle camera to automatically select objects and define a capture strategy for capturing objects with a telephoto camera. In some embodiments, a focus peaking map can be displayed to the user to select objects to be captured with the telephoto camera.
[0028] In some embodiments, the AP can be configured to calculate a depth map based on PDAF data and use the depth map to generate a new macro image.
[0029] In some embodiments, the telephoto lens module may include one or more D-cut lenses.
[0030] In some embodiments, the system may further include a liquid lens for focusing to an object-lens distance of 4-15 cm. In some embodiments, the refractive power of the liquid lens can be continuously varied within a range of 0-30 diopters. In some embodiments, the liquid lens may be located on the top side of the OPFE of the folding telephoto camera. In some embodiments, the liquid lens may be located between the OPFE of the folding telephoto camera and the telephoto lens module. Attached Figure Description
[0031] The following description, with reference to the accompanying drawings listed after this paragraph, illustrates non-limiting examples of embodiments disclosed herein. The drawings and description are intended to illustrate and explain the embodiments disclosed herein and should not be construed as limiting in any way. Identical elements in different drawings may be represented by the same numerals. Elements in the drawings are not necessarily drawn to scale.
[0032] Figure 1A A perspective view of an embodiment of a folded telephoto lens and sensor module in telephoto lens mode is shown, with focus at infinity.
[0033] Figure 1B It shows Figure 1A The telephoto lens and sensor module focus on a stereoscopic image of the near object in macro lens mode;
[0034] Figure 1C A cross-section of another continuously zoom telephoto lens and sensor module disclosed herein in its minimum zoom state is shown;
[0035] Figure 1D It shows Figure 1C The module is in the middle zoom state;
[0036] Figure 1E It shows Figure 1C The module is in maximum zoom mode;
[0037] Figure 1F A cross-sectional view of the telephoto lens and sensor module of another continuous zoom disclosed herein is shown in its minimum zoom state.
[0038] Figure 1G It shows Figure 1F The module is in the middle zoom state;
[0039] Figure 1H It shows Figure 1F The module is in maximum zoom mode;
[0040] Figure 1I An embodiment of the folding telephoto camera disclosed herein is shown;
[0041] Figure 1J The pop-up camera is shown in either operational or "pop-out" mode.
[0042] Figure 1K It shows Figure 1J The pop-up camera is in an inactive or "collapsed" state;
[0043] Figure 1L A cross-sectional view shows the exemplary telephoto camera lens system disclosed herein in a folded state;
[0044] Figure 1M It shows Figure 1L The lens system is in the first telephoto state, with a first EFL and a first zoom factor;
[0045] Figure 1N It shows Figure 1L The lens system is in a second telephoto state with a second EFL and a second zoom factor;
[0046] Figure 10 It shows Figure 1L The lens system is in long focal macro mode, with a third EFL and a third zoom factor;
[0047] Figure 1PAnother exemplary telephoto camera lens system disclosed herein is schematically illustrated in a cross-sectional view in a pop-up state;
[0048] Figure 1Q It shows Figure 1P The lens system is in the first folded state;
[0049] Figure 1R It shows Figure 1P The lens system is in the second folded state;
[0050] Figure 1S A cross-sectional view in a pop-up state schematically illustrates yet another exemplary telephoto camera lens system disclosed herein;
[0051] Figure 1T It shows Figure 1S The lens system is in a folded state;
[0052] Figure 1U The dual-camera output image size and ratio between the ultra-wide-angle FOV and macro FOV are schematically shown;
[0053] Figure 2A An embodiment of the folding telephoto digital camera with macro capabilities disclosed herein is shown;
[0054] Figure 2B Another embodiment of the folding telephoto digital camera with macro capabilities disclosed herein is shown;
[0055] Figure 2C A cross-sectional view of the telephoto lens and sensor module in the first zoom state, as disclosed herein, is shown.
[0056] Figure 2D It shows Figure 2C The module is in the second zoom state;
[0057] Figure 2E It shows Figure 2C The module is in the third zoom state;
[0058] Figure 3A The diagram shows a focused point object where microlenses project light from the object onto the center of two sub-pixels, resulting in zero parallax.
[0059] Figure 3B Showing out of focus Figure 3A The light rays from a point object;
[0060] Figure 4A A method for capturing the macro focus stack disclosed herein is shown;
[0061] Figure 4BAnother method for generating the focused stack disclosed in this paper is shown;
[0062] Figure 5A An exemplary macro object and settings for capturing macro objects are shown;
[0063] Figure 5B It shows Figure 5A Output image of macro settings;
[0064] Figure 5C Another exemplary macro object and settings for capturing macro objects are shown;
[0065] Figure 5D It shows Figure 5C Output image of macro settings;
[0066] Figure 6 A method for generating a single macro image from multiple images in a focus stack is shown;
[0067] Figure 7 A graphical user interface (GUI) is shown, which allows users to send commands to modify the appearance of the output image;
[0068] Figure 8A Demonstrating symmetric fuzzy functions;
[0069] Figure 8B It shows having Figure 8A The asymmetric fuzzy function described in the text;
[0070] Figure 9 A system for performing the methods disclosed herein is shown; and
[0071] Figure 10 An exemplary dual-camera setup is shown. Detailed Implementation
[0072] Telephoto cameras with macro photography modes can switch to macro mode by moving within the telephoto lens, thereby changing the lens properties. Cameras with this capability are described, for example, in jointly owned international patent applications PCT / IB2020 / 051405 and PCT / IB2020 / 058697. For instance, Figures 19A and 19B in PCT / IB2020 / 051405 show two folded telephoto camera states: one with the telephoto lens in a first "tele lens" state, and the other with the telephoto lens in a second "macro lens" state. Because telephoto cameras have a larger EFL (Exposure Flux) and the image sensor's image area is smaller in macro mode than in telephoto mode, the "macro lens" state may have a small macro FOV, such as FOV 198 below.
[0073] In the following text, if an image meets both of these criteria, it is called a "macro image":
[0074] The magnification ratio of objects to images in the range of 1:5 to 25:1; and
[0075] Capture objects at lens distances ranging from 30mm to 350mm using a camera with an EFL range of 7mm-40mm.
[0076] Figure 1A and 1B An embodiment of the folding telephoto lens and sensor module disclosed herein and numbered 100 is illustrated schematically. Figure 1A The module 100 is shown in a top perspective view in a telephoto lens state focused at infinity, and... Figure 1B This shows the magnification (M) of the image with the maximum object pair. max The module 100 is in macro lens mode, where it focuses on a (near) object about 4 cm from the camera from the same top perspective view.
[0077] Module 100 also includes a first lens group (G1) 104, a second lens group (G2) 106, and a third lens group (G3) 108, a module housing 102, and an image sensor 110. In this embodiment, lens groups 104, 106, and 108 are fixedly coupled, meaning the distance between the lens groups does not change. Lens groups 104, 106, and 108 can together form a lens with an EFL of 13 mm. Lens groups 104, 106, and 108 share a lens optical axis 112. For focusing, lens groups 104, 106, and 108 are actuated together along the lens optical axis 112 by a VCM mechanism (not shown). The VCM mechanism (not shown) can also be used to change between lens focusing states.
[0078] Reference Figure 1B The optical design detailed in Example 6 of Table 25 of PCT / IB2020 / 051405 can achieve M max = 2.3:1 (for an object at 4.2cm). This is an approximation based on a thin lens with EFL = 13mm, lens-image distance v = 19mm, and object-lens distance u = 42mm. M max It can be done as follows Figure 1B The lens configuration shown is used to achieve this, in which lens group G1+G2+G3 moves together as far away from the object as possible (i.e., away from sensor 110).
[0079] By capturing objects from a greater distance, smaller objects can be continuously selected for image magnification M. Using... Figure 1A The lens configuration and lens group G1+G2+G3 are moved as close as possible to the image sensor 110 to achieve zero magnification (for objects at infinity). For zero and M... max The magnification ratios between these limits are such that lens groups G1+G2+G3 move together within the aforementioned constraints. For example, a magnification ratio M = 4.3:1 might be required. To obtain the magnification from M... max When the state is switched to M=4.3:1, lenses G1+G2+G3 must move together about 3mm toward the image sensor.
[0080] In another embodiment, the macro camera can have an EFL of 25mm and can be compared to the UW camera described above with an EFL of 2.5mm. Both cameras can include the same image sensor, for example, with an active image sensor width of 4mm. When focused at 5cm, the macro camera with an EFL of 25mm will have a 1:1 object-to-image magnification ratio and will capture an object width of 4mm (the same as the sensor width). In contrast, the UW camera with an object-to-image magnification ratio of approximately 19:1 will capture an object width of 76mm.
[0081] Telephoto cameras with an EFL of 7-40mm can be advantageous for macro photography because they offer high image magnification. However, focusing a telephoto camera onto a short object—the distance between the lens and the object—is not easy and requires a large lens travel. These lenses must support optical specifications, such as limiting the eccentricity deviation between the lens and the image sensor (relative to a plane perpendicular to the optical path) to 25 micrometers or less, for example, to 5 micrometers. For example, focusing a macro camera with an EFL of 25mm to 10cm (relative to infinity) requires approximately 6.3mm of lens travel. For upright (non-folding) telephoto cameras, a lens travel of 2mm or more is incompatible with the height limitations of mobile devices (and cameras). However, in folding camera designs (in... Figure 1A-1B and Figure 2A-2B(as described in the text) or "pop-up" camera design (in...) Figure 1J-1K As described, for example in jointly owned international patent application PCT / IB2020 / 058697, the height of a smartphone does not limit the travel of such lenses.
[0082] In other embodiments, the folded or unfolded telephoto camera used to capture macro images may have an EFL of 7-40mm, such as 18mm. For macro functionality, the folded or unfolded telephoto camera may be able to continuously focus on an object-lens distance of 30-350mm.
[0083] Figure 1C-1E An embodiment of a continuous zoom telephoto lens and sensor module disclosed herein and numbered 120 is shown in different zoom states. Figure 1C The module 120 is shown in its minimum zoom state, with an EFL of 15mm; Figure 1D The module 120 in intermediate zoom mode is shown, with an EFL of 22.5mm; and Figure 1E The module 120 is shown in its maximum zoom state, with an EFL of 30mm.
[0084] Module 120 includes a lens 122 with eight single lens elements L1-L8, an image sensor 124, and an optional optical window 126. The optical axis is indicated by 128. Module 120 is included in a folding telephoto camera, such as camera 1000. Module 120 has a continuous zoom range, which can be continuously switched between a minimum zoom state and a maximum zoom state. Maximum zoom state EFL MAX EFL and minimum zoom state EFL MIN The EFL satisfies the EFL MAX =2x EFL MIN Lens 122 is divided into three lens groups: group 1 (“G1”) closest to the object, group 2 (“G2”), and group 3 (“G3”) closest to sensor 124. To change the zoom state, G1 and G3 move together as a group (“G13”) relative to G2 and sensor 124. To focus, G1+G2+G3 move together as a group relative to sensor 124.
[0085] Figure 1F-1H Another embodiment of a telephoto lens and sensor module with continuous zoom, disclosed herein and numbered 130, is shown in different zoom states. Figure 1F The module 130 is shown in its minimum zoom state, with an EFL of 10mm; Figure 1G The module 130 in intermediate zoom mode is shown, with an EFL of 20mm; and Figure 1HThe module 130 is shown in its maximum zoom state, with an EFL of 30mm.
[0086] Module 130 includes a lens 132 with ten single lens elements L1-L10, an image sensor 134, and an optional optical window 136. Module 130 is included in a folding telephoto camera such as camera 1000. Module 130 has a continuous zoom range, capable of continuously switching between a minimum zoom state and a maximum zoom state. Maximum zoom state EFL MAX EFL and minimum zoom state EFL MIN The EFL satisfies: EFL MAX =3x EFL MIN Lens 132 is divided into four lens groups: group 1 (“G1”), group 2 (“G2”), group 3 (“G3”) closest to the object, and group 4 (“G4”) closest to sensor 134. To change the zoom state, G1 and G3 move together as a group (“G13”) relative to G2, G4, and sensor 134. To focus, G13+G2+G4 move together as a group relative to sensor 134.
[0087] Figure 1I An embodiment of a folding telephoto camera disclosed herein and numbered 140 is shown. Typically, a folding telephoto camera is based on an optical path folding element (OPFE). Such a scanning folding telephoto camera is described, for example, in jointly owned international patent application PCT / IB2016 / 057366. Camera 140 is based on two OPFEs and can therefore be referred to as a “double-folded” telephoto camera. Module 140 includes a first “object OPFE” 142, an object OPFE actuator 144, an “image OPFE” 146, and an image OPFE actuator 148. A lens (not shown) is included in a lens barrel 150. Camera 140 also includes an image sensor 151 and a focus actuator 153.
[0088] Module 140 is a scanning folding telephoto camera. Through the rotational motion of the object OPFE 142 and the image OPFE 146, the native (diagonal) FOV (FOVN) of camera 140 can be manipulated to scan the scene. The FOVN can be 10-40 degrees, and the scanning range of the FOVN can be ±5 degrees to ±35 degrees. For example, a scanning folding telephoto camera with a 20-degree FOVN and ±20 FOVN scanning covers a 60-degree telephoto FOV.
[0089] Figure 1J-1K An exemplary pop-up telephoto camera 160 is shown, for example, in a jointly owned international patent application, PCT / IB2020 / 058697. Figure 1Jshows the pop-up camera 160 in an operative or "pop-up" state. The pop-up camera 150 includes an aperture 152, a lens barrel 154 including a lens (not shown), a pop-up mechanism 156, and an image sensor 158. Figure 1K shows the pop-up camera 160 in a non-operative or "folded" state. The camera 150 is switched from the pop-up state to the folded state by the pop-up mechanism 156. In some dual-camera embodiments, both the W camera and the T camera may be pop-up cameras. In other embodiments, only one of the W or T cameras may be a pop-up camera, while the other (non-pop-up) camera may be a folded or non-folded (upright) camera.
[0090] Figure 1L-1O schematically illustrates, in a cross-sectional view, an exemplary pop-up telephoto macro camera lens system 170 as disclosed herein. The lens system 170 may be included in the pop-up camera as Figures 1J to 1K described in. Figure 1L shows the lens system 170 in a folded state. Figure 1M shows the lens system 170 in a first telephoto state, which has a first EFL (EFL1) and a first zoom factor (ZF1). Figure 1N shows the lens system 170 in a second telephoto state, which has a second EFL (EFL2) and a second ZF2, where EFL1 < EFL2 and ZF1 < ZF2. Figure 10 shows the lens system 170 in a telephoto macro state, which has a third EFL3 and a third ZF3. In the telephoto macro state, the camera including the lens system 170 may focus on an object at an object-to-lens distance close to <350mm to capture a macro image.
[0091] Figure 1P-1R schematically illustrates, in a cross-sectional view, another exemplary pop-up telephoto macro camera lens system 180 as disclosed herein. The lens system 180 includes a lens 182 and an image sensor 184. The lens system 180 may be included in a pop-up camera, as Figure 1J-1K described in. Figure 1P shows the lens system 180 in a pop-up state. In the pop-up state, the camera including the lens system 180 may focus on an object at an object-to-lens distance close to <350mm to capture a macro image. Figure 1Q shows the lens system 180 in a first folded state. Figure 1R shows the lens system 180 in a second folded state.
[0092] Figure 1S-1TA cross-sectional view schematically illustrates another exemplary pop-up telephoto macro camera lens system 190 as disclosed herein. Lens system 190 includes a lens 192 and an image sensor 194. Lens system 190 may be included in, for example... Figure 1J-1K The pop-up camera described in the text. Figure 1S The lens system 190 in its pop-up state is shown. In the pop-up state, a camera including the lens system 190 can focus on an object less than 350mm away from the lens to capture a macro image. Figure 1T The lens system 190 is shown in a folded state.
[0093] Cameras including or comprising modules 100, 120, 130, 140, 150, 170, 180, 190, and 220 may be able to / use to capture macro images using macro camera modules (e.g., macro camera module 910).
[0094] Figure 1U Example 195 illustrates the exemplary three-camera output image sizes and ratios between ultra-wide (UW) FOV 196, wide (W) FOV 197, and macro FOV 198. Regarding telephoto cameras used to capture objects at lens-object distances, such as 1m or more, a larger image is formed on the image sensor plane in telephoto-based macro mode. Therefore, the area the image may cover is larger than the active area of the image sensor, so only a cropped FOV of the telephoto camera's FOV is available for capturing macro images. For example, consider a macro camera with an EFL of 30mm and an image sensor with an effective image sensor width of 4mm. When focusing on an object at 5cm (lens-object distance), a lens-image distance of v = 77mm is required for focusing, and an object-to-image magnification ratio of approximately 1:1.5 can be achieved. Approximately 43% of the macro FOV of a practical telephoto FOV is available for capturing macro images.
[0095] The following description refers to W cameras, but it is assumed that UW cameras can be used instead.
[0096] Figure 2AAn embodiment of the folding telephoto camera with macro capability disclosed herein, designated 200, is shown. Camera 200 includes an image sensor 202, a lens 204 having an optical axis 212, and an OPFE 206, such as a prism. Camera 200 also includes a liquid lens (LL) 208 mounted on the top side (object-facing surface, not shown) of the prism 206 in a direction 214 perpendicular to the optical axis 212. The liquid lens has optical characteristics adjustable by a voltage supplied by an LL actuator 210. In this embodiment, LL 208 can continuously provide a diopter range of 0 to 35 diopters. In macro photography mode, the entire lens system including LL 208 and lens 204 can have an EFL of 7-40 mm. The DOF can be as shallow as 0.01-2 mm. In this embodiment and the following embodiments, the liquid lens has a mechanical height H. LL And the optical height (clear height) CH. CH defines the corresponding height of the clear aperture (CA), where CA defines the lens surface area that meets the optical specifications. That is, CA is the effective optical area, and CH is the effective height of the lens, see, for example, jointly owned international patent application No. PCT / IB2018 / 050988.
[0097] For conventional lenses with fixed optical properties (compared to LL lenses with adaptive optical properties), the ratio between the net height and the lens mechanical height H (CH / H) is typically 0.9 or greater. For liquid lenses, the CH / H ratio is typically 0.9 or less, for example, 0.8 or 0.75. Therefore, in order to utilize the CH and H of an optical system comprising prisms and lenses... LL It can be designed to be 15% or 20% larger than the smallest side of the prism's top surface. In embodiment 200, the LL actuator 210 is positioned along the optical axis 212 of the lens, i.e., in the -X direction of the XYZ coordinate system shown. The lens 204 can be a D-cut lens having a lens width W greater than the lens height H. In one example, the width / height W / H ratio of the D-cut lens can be 1.2.
[0098] Figure 2BAnother embodiment of a folding telephoto camera with the macro capabilities disclosed herein is shown, designated 200'. Camera 200' includes the same elements as camera 200, except that in camera 200', LL 208 is located between prism 206 and lens 204. As in camera 200, lens 204 may be a D-shaped cut lens having a lens width W greater than the lens height H. In one example, the width / height W / H ratio of the D-shaped cut lens may be 1.2. As in camera 200, in macro photography mode, the entire lens system including LL 208 and lens 204 may have an EFL of 7mm-40mm and a DOF as shallow as 0.01-7.5mm.
[0099] Figure 2C-2E Another embodiment of a continuous zoom telephoto lens and sensor module disclosed herein and numbered 220 is schematically illustrated in different zoom states. Module 220 is included in a folding telephoto camera, such as camera 1000. Module 220 includes a lens 222, (optional) optical elements 224, and an image sensor 226. Figure 2C-2E Three fields are displayed, each with three rays: the upper edge ray, the lower edge ray, and the principal ray. Lens 222 comprises six single lens elements L1-L6. The optical axis is indicated by 228.
[0100] Figure 2C The module 220, which focuses to infinity, is shown. Figure 2D Module 220 with a focus of 100mm is shown. Figure 2E The module 220 is shown with the focus set to 50mm.
[0101] Lens 220 is divided into two lens groups G1 (including lens elements L1 and L2) and G2 (including L3, L4, L5, and L6), which move relative to each other and together act as a lens to focus relative to the image sensor. Because these cameras have very shallow degrees of freedom, capturing the focus stack and constructing a good image from it is not easy. However, the method described below allows this to be done.
[0102] Some multi-camera systems equipped with W cameras and telephoto cameras with macro capabilities (or just one of the cameras) feature phase-detection autofocus (PDAF) sensors, such as 2PD sensors, where each sensor pixel is divided into two or more sub-pixels and supports depth estimation by calculating parallax. PDAF sensors utilize multiple microlenses (“MLs”) or partially covered MLs to detect pixels in and out of focus. The multiple MLs are calibrated so that the focused object is projected onto the sensor plane at the same position relative to the lenses; see [link to documentation]. Figure 3A .
[0103] Figure 3A A focused point object 302 is shown, where ML projects light from the object onto the center of two sub-pixels, resulting in zero disparity. Figure 3B The image shows light rays from a defocused point object 304. The terms "main lens," "ML," and "sub-pixel pair" are shown in... Figure 3A and Figure 3B It is shown in the same way in the middle. Figure 3B In the diagram, the left-hand ML projects light from object 304 onto the center of the left sub-pixel. The right-hand ML projects the same object onto the right sub-pixel, resulting in a parallax value of 2. Objects before / after the focal plane (not shown) are projected to different positions relative to each lens, creating positive / negative differences between projections. PDAF parallax information can be used to create a "PDAF depth map." Note that this PDAF depth map is both coarse (due to a very small baseline) and relative to the focal plane. That is, zero parallax is detected for objects in focus, not objects at infinity. In other embodiments, the depth map may be based on image data from a stereo camera, time-of-flight (ToF) images, or by methods known in the art for monocular depth, such as depth of motion.
[0104] Figure 4A A method for capturing a macro focus stack (or “defining a telephoto capture strategy”) as disclosed herein is illustrated. The term “focus stack” refers to multiple images captured under the same imaging conditions (i.e., the camera and object do not move during the capture process of the focus stack, but the lens focus moves in defined steps between successive image captures). An application controller (AP), such as… Figure 9 The AP 940 shown can be configured to perform the steps of this method. In step 402, the object is focused. In some embodiments, and to focus an object or region, a focus peaking map known in the art can be displayed to the user. If a scanning telephoto camera such as camera 140 is used, the object can be focused by detecting the object in the W camera's FOV and automatically turning the scanning telephoto camera's FOV toward this object. The object in the W camera's FOV can be automatically selected for focusing by an algorithm or manually selected by a human user. For example, a saliency algorithm known in the art that provides a saliency map can be used for automatic object selection by algorithm. The user gives a capture command in step 404. A first image is captured in step 406. In step 408, according to the following description and... Figure 5A and Figure 5BThe method shown analyzes the image. In some embodiments, only a segment of the image (instead of the entire image) may be analyzed. The segment being analyzed may be defined by an object detection algorithm running on image data from a macro camera or an W camera. Alternatively, the image segment being analyzed (i.e., the OOI) may be manually labeled by the user. Based on the result of this analysis, in step 410, the lens is moved by a defined step to focus forward (i.e., focus moves one step away from the camera), or in step 412, the lens is moved by a defined step to focus backward (i.e., focus moves one step towards the camera). Forward or backward focusing may depend on the command generated in step 408. For example, when a detection is detected... Figure 5B (or Figure 5D When the platform (plateau) A (A') in the ) is detected, a back focus command can be triggered. For example, when the platform (plateau) A (A') is detected, a back focus command can be triggered. Figure 5B (or Figure 5D If platform A (A') is not present in the sequence, a forward focus command can be triggered. An additional image is captured in step 414. These steps are repeated until the analysis output in step 408 provides a command to reverse backward focus or an abort command to stop the focus stack capture. For example, when a forward focus command is detected... Figure 5B (or Figure 5D When platform A (A') or E (E') is in the focus stack capture, an abort command can be triggered. The abort command ends the focus stack capture in step 416. In another embodiment, step 410 can be replaced by step 412 and step 412 can be replaced by step 410, that is, backward focusing can be performed first and then forward focusing.
[0105] If a scanning telephoto camera (e.g., camera 140) is used to capture a macro focus stack and a telephoto capture strategy is defined, an object (“object FOV”) covering a segment of FOV larger than the original telephoto FOV can be captured by multiple focus stacks, each covering a different segment of the object FOV. For example, W camera image data can be used to divide the object FOV into multiple smaller FOVs (than the telephoto FOVN), where the FOVs are captured sequentially using the focus stack capture process described above, and then stitched together after capturing multiple FOVs.
[0106] If a continuous zoom telephoto camera (e.g., camera 120 or camera 130) is used to capture a macro focus stack and a telephoto capture strategy is defined, such as based on the size, content, or color of the object's field of view (FOV), a specific zoom factor can be selected. For example, W camera image data can be used to analyze macro objects. Based on this analysis, an appropriate zoom factor can be selected for the continuous zoom telephoto camera. One selection criterion could be that the FOV of the continuous zoom telephoto camera completely covers the macro object. Another selection criterion could be that the FOV of the continuous zoom telephoto camera not only completely covers the macro object but also additionally covers a certain amount of the background FOV, for example, for aesthetic reasons. Another selection criterion could be to select an FOV such that the image captured by the continuous zoom telephoto camera can have a specific DOF. As a first example, a larger DOF might be advantageous for capturing objects with a focus stack that includes a smaller number of individual images. As a second example, a specific DOF might be beneficial for, for example, the aesthetic appearance of macro images.
[0107] Figure 4B This illustrates another method for capturing focus stacks (or defining telephoto capture strategies). AP (e.g.) Figure 9The AP 940 shown can be configured to perform the steps of this method. In step 452, a PDAF image is captured using the W camera. In step 454, a depth map is calculated from the PDAF image, as is known in the art. In step 456, focus stack parameters such as focus step size and focus stack brackets are derived from the depth map. Focus stack brackets are the upper and lower limits of the focus stack, i.e., they comprise two planes: a first in-focus plane with the largest object-lens distance in the focus stack, and a second in-focus plane with the smallest object-lens distance in the focus stack. Multiple images with offset focus are captured between these two limits. The focus step size defines the distance between two consecutive in-focus planes captured in the focus stack. The focus planes can have a specific depth defined by the DOF (center-focus plane). The parameters defined in step 456 can be used to control the camera. For example, the parameters can be input into standard burst mode functions for focus stack capture, such as those available on Android smartphones. In step 458, the focus stack is captured according to the parameters. In other embodiments, the PDAF image in step 452 may be captured by a telephoto camera with macro capabilities instead of a W camera. A telephoto camera's PDAF image may exhibit higher spatial resolution, which may be desirable, and stronger blurring of the out-of-focus areas, which may or may not be desirable. For objects with shallow depth, stronger blurring of the out-of-focus areas may be necessary, for example, at a depth <1 mm. For objects with greater depth, stronger blurring of the out-of-focus areas may be undesirable, for example, at a depth >2.5 mm. Strong blurring may make the depth calculation performed in step 454 impossible.
[0108] In some embodiments, in step 452, PDAF image data may be captured only from a specific scene segment, for example, only for the ROI. In other embodiments, in step 452, PDAF image data may be captured from the entire scene, but the depth map calculation in step 454 may be performed only for the segment. Specific scene segments can be identified by image analysis performed on image data from a UW or W or telephoto camera. In step 452, PDAF maps may be captured not only from a single image but also from a video stream.
[0109] In some embodiments, instead of calculating the depth map in step 454, the depth map or image data used to calculate the depth map may be provided by an additional camera.
[0110] In some embodiments, different analysis methods can be applied to analyze the entire macro scene at only one (or only a few) focus locations. From this analysis, preferred focus stack steps and focus stack ranges can be derived. These values are then input into standard burst mode functionality for focus stack capture.
[0111] In some embodiments, for focus stack capture in step 458, imaging settings (e.g., values for white balance and exposure time) can remain constant for all images captured in the focus stack.
[0112] Capturing a focus stack containing a macro image with a shallow DOF may require actuating the camera lens with high precision, as the DOF defines the minimum precision limit for the focusing process. The required actuation precision can be derived from the DOF of the image. For example, it may be necessary to allow actuation precision that allows control of the position of the focus plane with a precision 2–15 times greater than the DOF. For instance, consider a focus stack containing a macro image with a DOF of 50 μm, where scene fragments less than 25 μm from the focus plane are in focus. Therefore, the minimum focusing precision is 25 μm to 3 μm.
[0113] Optical image stabilization (OIS), known in the art, can be used during focus stack capture. OIS can be based on the OPFE of the activation lens or image sensor or camera 910. In some embodiments, depth data from a macro scene can be used for OIS.
[0114] Figure 5A An illustrative macro object (a flower in this case) and a camera used to capture the macro object (not to scale) are shown. The flower was captured from a top position (labeled "camera").
[0115] Figure 5B It shows Figure 5A An exemplary output image of the macro settings, wherein Figure 5A use Figure 4A The method described in the figure is used to obtain the results. The points in the figure represent the analysis results of a specific image in the focus stack, i.e., each image in the focus stack is analyzed during the focus stack capture as described above, where the analysis provides the number of pixels in each image (the sum of pixels within focus). These numbers can be plotted as shown here. The analysis can use functions known in the art, such as the Laplace-Gaussian operator or Brenner's focus measure. An overview of suitable functions can be found in: Santos et al., “Evaluation of autofocusing capabilities in molecular cytogenetic analysis,” Journal of Microscopy, Vol. 188, No. 3, December 1997, pp. 264-272.
[0116] The analysis output is a measurement of the number of pixels in focus in each image. The higher the output number for a particular image, the higher the total number of pixels in focus in the image. The assumption of focus stack analysis is that the major parts of a macro object exhibit an analysis curve characterized by common specific features. This curve is characterized (starting from the left side of the image, i.e., from a camera scene setting where the focus is farther than the macro object) at plateau A (the focus is farther than the object, so almost no pixels are in focus, and the output number is low). This is followed by a positive gradient region B (first the farthest part of the macro object is in focus, then a larger portion of the macro object is in focus). This is followed by a plateau C (e.g., the center of the macro object and most of the object are in focus). Then comes a negative gradient D (the focus shifts away from the center of the macro object). Finally, there is a plateau E. Figure 4A The abort command shown is triggered by detecting platform A or platform E. Depending on the starting focus position of the focus stack capture, the focus stack capture will be aborted or the direction of focus movement will be switched (from towards the camera to away from the camera or vice versa). Typically, focus stack capture can begin from a focus position where a part or point of the macro object is in focus. The analysis will output a high number for the first image. The focus then moves away from the camera, meaning the analysis output moves on platform C (towards the left in the diagram) until it reaches gradient region B in the diagram, and finally reaches platform region A. If the number of the analysis output does not increase further, the focus will move back to the first position (on platform C) and the focus will be moved towards the camera. The same steps as above are performed until the final platform E is reached. At this point, the focus stack capture process ends.
[0117] Figure 5C Another exemplary macro object (here, a bee) and another camera used to capture the macro object (not scaled) are shown. Figure 5D It shows Figure 5C Another exemplary output image of the macro setting, where the method described in Figure 4 is used for processing. Figure 5C Although the details differ due to variations in object depth distribution, the features A'-E' here are similar to... Figure 5B The features in AE are similar.
[0118] According to, for example, in Figure 4A , Figure 4B and Figures 5A to 5D The telephoto image captured by the described method is an input macro image that can be further processed, for example, by... Figure 6 The method described in [the document / document].
[0119] Figure 6A method for generating a single macro image from multiple images in a focus stack is illustrated. An AP, such as the AP 940, can be configured to perform the steps of this method. In step 602, suitable images from the focus stack are selected using analytical methods known in the art. Criteria that may disqualify an image as a "suitable" image might include: significant motion blur in the image (e.g., from a handshake), redundancy in the capture data, or poor focus. Only the selected suitable images are further used in this process. In step 604, the suitable images are aligned using methods known in the art. In step 606, a suitable image region is selected from the aligned images. The selection criteria for a "suitable" region may include the degree of focus of the region, such as whether a region is sharply focused or has some degree of defocus blur. The selection of selection criteria depends on the input of the user or the program. The user may want the output image to have a macro object in full focus (i.e., an image with a depth of field greater than the depth of the macro object), meaning that all parts of the macro object are in focus simultaneously. However, a view in full focus is not usually the most pleasing image for a human observer (because human perception also produces some degree of blur due to depth), so an image with a certain focus plane and a certain amount of image blur may be more attractive. The "focus plane" is the plane formed by all the focus of the unprocessed image. An image from the focused stack generated as shown in the figure. Figures 4A-4B The selection of a suitable image in step 606 allows for the continuous selection of any focus plane and any amount of blur in the output image 612. The amount of blur in an image segment that is out of focus may depend on its position in the scene. For an image segment of an object that is a certain distance d away from the camera relative to the focus plane, the amount of blur may differ from that of an image segment that is a certain distance d closer to the camera than the focus plane. Continuous control of the focus plane position and depth of field of the new macro image can be performed after the capture focus stack (“post-capture”). In some embodiments, continuous control of the focus plane position and depth of field of the new macro image can also be performed before the capture focus stack (“pre-capture”), for example, by enabling this by displaying a preview video stream to the user. In step 608, the selected images are fused into a single image using methods known in the art. In some embodiments and optionally, the fusion in step 608 may use depth map information, such as estimated depth map information. Depth from focus or depth from defocus methods known in the art are used. In other embodiments, information from PDAF (see [link to PDAF]) may be used. Figure 3A-B) Depth map information. PDAF information can be provided from the image sensor of a UW camera, a W camera, or a telephoto camera with macro capabilities. In some embodiments, the telephoto camera can capture PDAF data simultaneously with capturing a telephoto focus stack image, i.e., capturing a PDAF image stack under the same focus conditions as the focus stack image. A depth map can be calculated from this PDAF image stack. For example, only in-focus image segments from a single PDAF image can be used, as they can be assigned to a specific depth with high precision. By fusing depth estimation data from all in-focus image segments of the PDAF image stack, a high-quality depth map can be generated.
[0120] In some embodiments, in step 608, telephoto image data and wide-angle image data can be fused into a single image.
[0121] In other embodiments, only a subset of the images selected in step 602 may be merged into a single image in step 608 and output in step 612. For example, subsets of only 1, only 2, or only 3, or only 4, or only 5 images may be merged into a single image in step 608 and output in step 612. In yet another embodiment, only one of the images selected in step 602 may be output in step 612. The single output image is fine-tuned in step 610 to finalize the result, for example, by reducing noise. Fine-tuning may include filters such as smoothing image seams, enhancement, radial blurring, and chroma attenuation. The image is output in step 612.
[0122] In other embodiments, the selection of a suitable image region in step 606 can be based on image analysis performed on the image from the W camera. Since the W camera has a wider FOV and a larger DOF (relative to a telephoto camera with macro capabilities), it may be beneficial to additionally use W image data to generate a single macro image, for example, for object recognition and segmentation. For example, the FOV can be adjusted before or during focus stacking capture using a telephoto camera with macro capabilities. W This method detects regions of interest (ROIs) or objects of interest (OOIs) at a macro scale. ROIs or OOIs can be segmented using methods known in the art. Segmentation means identifying the coordinates of the field of view (FOV) segments containing the ROI or OOI. (The FOV is then used to...) W and FOV T The calibration was performed, and these coordinates were converted to FOV. T Coordinates. The coordinates of the ROI or OOI can be used to select a suitable image region in step 606. In some embodiments, segmentation analysis can be performed on a single image. In other embodiments, segmentation analysis can be performed on a video stream, i.e., a sequence of single images.
[0123] In some embodiments, image information from the W camera can be used for further tasks. One or more W images can be used as ground truth "anchors" or reference images in the macro image generation process. Ground truth here refers to W image information about a scene fragment, which is much more complete than telephoto image information about the same scene fragment. A single W image provides more information about the macro object compared to a single telephoto image. As an example, one might think of a ROI or OOI that is mostly in focus and fully visible in a single W image, but only partially visible in a single telephoto image, for example, because the DOF is significantly lighter at a distance. The W ground truth or reference image can be used as... Figure 6 The ground-based anchor points in the following steps of the method described in the document:
[0124] In step 602, the W image can be used to select a suitable image. Ground reality may, for example, allow the identification of telephoto images that exceed a certain focus blur or motion blur threshold.
[0125] In step 604, the W image can be used as a reference image for image alignment. In one example, the telephoto images of the focus stack can all be aligned with reference to the W reference image. In another example, the telephoto images of the focus stack can be aligned first with reference to the W reference image, and for more detailed alignment, other telephoto images in the focus stack can be referenced for alignment.
[0126] In step 606, the W image can be used to define a suitable image region as described above.
[0127] In step 608, the W image can be used to correct fusion artifacts. Fusion artifacts are defined as visual features that do not exist in the actual scene, but are unwanted byproducts of the image fusion process.
[0128] In step 610, the W image can be used to identify image segments in the fused image that exhibit unwanted features and can be corrected. These unwanted features may be defocusing or motion caused by factors such as image misalignment, unnatural chromatic aberration, or blurring. Defocus blur may be caused, for example, by estimation errors in the depth map used in image fusion step 608.
[0129] In yet another embodiment, the above method may not involve any image processing such as that described in steps 608-612, but can be used to select a single image from the focus stack. This selection can be performed automatically (e.g., by using...). Figures 5A-5D The method described in the text analyzes the focus stack of the clearest, sharpest, and best-composed images (or is performed manually by a human user). Figure 7 The diagram illustrates a graphical user interface (GUI) that allows users to send commands to modify the appearance of the output image. For example, users can send commands such as "forward blur" and "backward blur" to obtain a more blurred image or an image where more of the image is in focus. "Background blur" and "forward blur" refer to... Figure 8A and 8B The blurring options described herein. In one embodiment, if the user command is to modify the appearance of the image, this method will re-execute from step 606, but with a different set of selection criteria. In another embodiment, if the user command is to modify the appearance of the image, a blurring algorithm (artificial blurring) can be applied to the output image to form another output image. The focus plane can be changed by marking a new image segment, which should be in focus by touching the device screen. The blur can be changed according to the user's wishes. The user may want to modify the DOF of the displayed image, for example, from an image in full focus (i.e., infinite DOF) to a shallower degree of freedom. The user may want to modify the focus plane of an image that is not in full focus. The user can modify the image, and a preview image generated by an estimate of the projected output image can be displayed. If the user performs a click on "apply", the full algorithm can be applied, such as... Figure 6 As described in [the text].
[0130] Figure 8A This demonstrates the symmetrical blur function. By moving the sliders in the graph (forward / backward blur), as shown... Figure 8A As shown, users can move linearly along the X-axis and apply blur to the image as indicated on the Y-axis. Figure 8B It shows having Figure 8A The asymmetric blur function described in the text. The application of blur functions allows users to blur the foreground and background in different ways. For example, in some cases, from an artistic point of view, forward blur may not be necessary at all. Asymmetric blur makes this possibility possible.
[0131] In some embodiments, further image features may include, for example, artificial lightning. Artificial lightning refers to lightning in a scene that can be altered by the user or a program, for example, by artificially moving light sources in the scene. For artificial lightning, the presence of a depth map may be beneficial.
[0132] Figure 9A system 900 for performing the methods described above is shown. System 900 includes a first telephoto camera module (or simply "telephoto camera") 910. The telephoto camera 910 may be a folding telephoto camera with macro capabilities, a double-folding telephoto camera, a pop-up telephoto camera, a scanning folding telephoto camera, or a vertical (non-folding) telephoto camera. If the camera 910 is a folding camera, it includes an optical path folding element (OPFE) 912 for folding the optical path by 90 degrees, a lens module 914, and an image sensor 916. A lens actuator 918 performs movement of the lens module 914 to bring the lens to different lens states for focusing and optionally for OIS. System 910 may include an additional second camera module 930 and an application processor (AP) 940. The second camera module 930 may be a W camera or a UW camera. In some embodiments, both a W camera and a UW camera may be included. AP 940 includes an image generator 942 for generating images and an image analyzer 946 for analyzing the images as described above, as well as an object detector 944. A human-machine interface (HMI) 950, such as a smartphone screen, allows the user to transmit commands to the AP. A memory element 970 can be used to store image data. Calibration data used for calibration between the camera 910 and the second camera module 930 can be stored in the memory element 970 and / or an additional storage element (not shown). The additional storage element can be integrated into the camera 910 and / or the second camera module 930. The additional storage element can be an EEPROM (Electrically Erasable Programmable Read-Only Memory). The memory element 970 can be, for example, an NVM (Non-Volatile Memory).
[0133] Figure 10 The illustration depicts a dual-camera system known in the art and designated 1000 (which may be part of a multi-camera system having more than two cameras), see, for example, co-owned international patent application PCT / IB2015 / 056004. The dual-camera 1000 includes a folding telephoto camera 1002 and a wide-angle camera 1004. The telephoto camera 1002 includes an OPFE 1006 and a lens 1008, which may include multiple lens elements (not visible in this illustration, but, for example, an optical axis 1010 and an image sensor 1012) having an optical axis 1010. Figure 1C-1H (See image). The wide-angle camera 1004 includes a lens 1014 having an optical axis 1016 and an image sensor 1018. The OPFE 1006 folds the optical path from a first optical path 1020, which is substantially parallel to the optical axis 1016, to a second optical path, which is substantially parallel to the optical axis 1010.
[0134] Although this disclosure has been described with reference to certain embodiments and generally associated methods, changes and arrangements of embodiments and methods will be apparent to those skilled in the art.
[0135] This disclosure should be understood to be limited not only to the specific embodiments described herein, but also to the scope of the appended claims.
[0136] Furthermore, for clarity, the term "substantially" is used herein to imply the possibility that a value may vary within an acceptable range. According to one example, the term "substantially" as used herein should be interpreted as implying a possible variation of up to 5% above or below any specified value. According to another example, the term "substantially" as used herein should be interpreted as implying a possible variation of 2.5% above or below any specified value. According to a further example, the term "substantially" as used herein should be interpreted as implying a possible variation of up to 1% above or below any specified value.
[0137] All references mentioned in this specification are incorporated herein by reference in their entirety to the same extent that each individual reference is specifically and individually indicated as being incorporated herein by reference. Furthermore, any reference or designation of any reference in this application should not be construed as an admission that such reference is prior art to this application.
Claims
1. A system, characterized in that: The system includes: Long-zoom camera with an effective focal length of 7mm to 40mm; An actuator operable to move components of the telephoto camera to focus the telephoto camera to a distance or a set of distances from 5cm to 35cm, and having an object-to-image magnification ratio of 1:5 to 25:1; and Application processor, configured as follows: Analyze the image data captured by the telephoto camera to define a capture strategy for capturing an object; A sequence of macro images of the object captured by the telephoto camera, wherein the sequence of macro images has a focal plane that moves from one captured macro image to another captured macro image, and New macro images are generated from a sequence of captured macro images, wherein the system is included in a mobile electronic device.
2. The system as described in claim 1, characterized in that: The telephoto camera is focused to a distance or a set of distances from 7.5cm to 35cm.
3. The system as described in claim 1, characterized in that: The telephoto camera is focused to a distance or a set of distances from 10cm to 35cm.
4. The system as described in claim 1, characterized in that: The new macro image includes image data of a single macro image from the sequence of macro images.
5. The system as described in claim 4, characterized in that: The selection of a single macro image from the sequence of macro images is performed manually by the user.
6. The system as described in claim 4, characterized in that: The application processor is configured to automatically select the individual macro image from the sequence of macro images.
7. The system as described in claim 6, characterized in that: The single macro image is the sharpest image automatically selected by analyzing the focus stack.
8. The system as described in claim 6, characterized in that: The single macro image is the optimal composition image automatically selected by analyzing the focus stack.
9. The system as described in claim 1, characterized in that: The new macro image comprises image data of two to five macro images from the sequence of macro images.
10. The system as described in claim 1, characterized in that: The focal plane and depth of field of the new macro image can be continuously controlled after capture.
11. The system as described in claim 1, characterized in that: The application processor is configured to generate the new macro image when the object is in full focus.
12. The system as described in claim 1, characterized in that: The application processor is configured to generate the new macro image such that a particular new macro image fragment has a specific amount of forward defocus blur and a specific amount of backward defocus blur.
13. The system as described in claim 1, characterized in that: The application processor is configured to provide new macro images with real artificial lightning scenes.
14. The system as described in claim 1, characterized in that: The system also includes a wide-angle camera, and the application processor is configured to analyze image data from the wide-angle camera to capture the object using the telephoto camera.
15. The system as described in claim 1, characterized in that: The system also includes an ultra-wide-angle camera, and the application processor is configured to analyze image data from the ultra-wide-angle camera to capture the object using the telephoto camera.
16. The system as claimed in claim 1, characterized in that: The system also includes a time-of-flight camera, and the application processor is configured to analyze image data from the time-of-flight camera to capture the object using the telephoto camera.
17. The system as claimed in claim 1, characterized in that: A focus peak map is displayed to the user for selecting the object captured by the telephoto camera.
18. The system as claimed in claim 1, characterized in that: The telephoto camera is a folding telephoto camera that includes an optical path folding element.
19. The system as claimed in claim 1, characterized in that: The telephoto camera is a pop-up telephoto camera that includes a pop-up lens.
20. The system as claimed in claim 1, characterized in that: The telephoto camera includes a liquid lens for focusing.
21. The system as described in claim 1, characterized in that: The application processor is configured to calculate a depth map based on image data or phase detection autofocus image data, and use the depth map to define the capture strategy or generate the new macro image.
22. The system as described in claim 1, characterized in that: The telephoto camera has an effective focal length of 10 to 30 mm.
23. The system as described in claim 1, characterized in that: The telephoto camera has an effective focal length of 10 to 20 mm.
24. The system as claimed in claim 1, characterized in that: The telephoto camera is capable of switching between a minimum zoom state and a maximum zoom state, and the application processor is configured to analyze image data to switch the telephoto camera to a specific zoom state to capture macro images.
25. The system as described in claim 1, characterized in that: The telephoto camera is a scanning telephoto camera that uses a corresponding telephoto field of view to scan the scene.
26. The system as described in claim 1, characterized in that: The system also includes a wide-angle camera, and the generation of the new macro image uses the wide-angle image as a reference image.
27. The system as claimed in claim 1, characterized in that: During the sequence of capturing the macro images, the telephoto capture strategy is adjusted based on information from the captured macro images.
28. The system as claimed in claim 1, characterized in that: The system also includes a wide-angle camera, and the generation of the new macro image includes fusing wide-angle image data and telephoto image data.
29. The system as described in any one of claims 1-28, characterized in that: The mobile electronic device is a smartphone.
30. The system as described in any one of claims 1-28, characterized in that: The mobile electronic device is a tablet computer.
31. A method, characterized in that: The method includes: in a mobile electronic device comprising a telephoto camera and an application processor: The telephoto camera is focused at a distance or a set of distances from 5cm to 35cm, and the magnification ratio of the object to the image is between 1:5 and 25:
1. Analyze the image data captured by the telephoto camera to define a capture strategy for capturing an object; The application processor is configured to capture a sequence of macro images of the object using the telephoto camera, the sequence of macro images having a focal plane that moves from one captured macro image to another; and New macro images are generated from the sequence of captured macro images.
32. The method as described in claim 31, characterized in that: The mobile electronic device is a smartphone.
33. The method as described in claim 31, characterized in that: The mobile electronic device is a tablet computer.
Citation Information
Patent Citations
Device and method for realizing focusing in micro-distance mode
CN106488136A
Zoom dual-aperture camera with folded lens
CN106576138A
System and method for obtaining ultra-macro images
CN114080565B
System for obtaining ultra-micro-distance images
CN117518313A
Systems and methods for tracking a region using an image sensor
US20190215440A1
