Systems and cameras for tilting a focal plane of a super-macro image

By using optical path folding elements to tilt the focal plane, the challenge of shallow depth of field in mobile device macro image capture is addressed, ensuring all segments of the object are in focus, enhancing macro image quality.

KR103046946B1Active Publication Date: 2026-11-11COREPHOTONICS
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
KR1020247004989
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-12-10
Filing Date
2022-06-07
Publication Date
2026-11-11
Estimated Expiration
2042-06-07

AI Technical Summary

Technical Problem

Existing mobile devices with telephoto cameras face challenges in capturing macro images due to the shallow depth of field, which results in partial focus issues, and there is no satisfactory solution to tilt the focal plane like in standalone cameras.

Method used

Incorporating an optical path folding element (O-OPFE) and an image-side optical path folding element (I-OPFE) to enable tilting of the focal plane, allowing the focal plane to be oriented at various angles relative to the camera, thereby adjusting the focus to capture macro images effectively.

Benefits of technology

The solution enables all segments of the object to be captured in focus, even at shallow depth of field conditions, by tilting the focal plane to align with the object's orientation, improving macro image capture quality on mobile devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112024016601745-PAT00008_ABST
    Figure 112024016601745-PAT00008_ABST
Patent Text Reader

Abstract

The system comprises a wide / ultra-wide camera, a folded tele camera including an optical path folding element and a tele lens module, a lens actuator that moves the tele lens module to focus at an object-lens distance between 3.0 cm and 35 cm while having an object-to-image magnification of 1:5 to 25:1, and an application processor (AP), wherein the AP is configured to analyze image data from the UW camera to define a tele-capture strategy for a sequence of macro images having a focal plane slightly shifted from one captured macro image to another captured macro image, and to generate a new macro image from said sequence, and the focal plane and depth of field of said new macro image can be continuously controlled.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 208,186 filed June 8, 2021, U.S. Provisional Application No. 63,274,700 filed November 2, 2021, and U.S. Provisional Application No. 63 / 288,047 filed December 10, 2021, the entire contents of which are incorporated herein by reference.

[0003] The subject matter disclosed in this specification generally relates to macro images, in particular to a method for acquiring macro images with a mobile telephoto ("tele") camera. Background Technology

[0004] Multi-cameras (an example being a "dual camera" with two cameras) are currently widespread in portable electronic mobile devices (or simply "mobile devices," such as smartphones, tablets, etc.). Multi-cameras generally feature a wide field of view (or "wide-angle") FOV. W A camera ("wide" or "W" camera), and at least one additional camera, for example (FOV W Includes a "tele" (or "T") camera having a narrow FOV, i.e., FOVT.

[0005] A triple camera including a W camera, an ultra-wide (UW) camera, and a double-folded tele camera is illustrated in FIG. 5a and FIG. 5b.

[0006] "Macro shooting" mode is becoming a popular differentiating factor. "Macro shooting" refers to capturing an object close to the camera so that the image recorded on the image sensor is approximately the same size as the actual object being photographed. The ratio of object size to image size is the object-to-image magnification ("M").

[0007] A macro camera providing a large M can be implemented as a telecamer on a mobile device. Because M is large, the camera may be referred to as a "super-macro camera" or "SM camera," and the image may be referred to as a "super-macro image" or "SM image." For example, the jointly owned international patent application PCT / IB2021 / 054186 describes a telecamer and a method for mobile super-macro shooting.

[0008] Because the effective focal length (EFL) of a telephoto camera is large, exceeding 7mm, the depth of field (DOF) of a telephoto camera focused on a nearby object is very shallow (see Table 1). DOF is the distance between the nearest and farthest objects in an image that is in acceptable sharp focus. Consequently, it is very difficult to capture macro images in macro-shooting mode using a telephoto camera. For example, to obtain more beneficial lighting or a more convenient capture experience, one may want to capture an object oriented at a certain angle (θ) relative to the rear of the mobile device and the focal plane of the macro camera (see Fig. 1a). This can result in an image that is only partially in focus. In standalone cameras such as digital single-lens reflex (DSLR) cameras, this problem is solved by "tilt-shift lenses" that allow the focal plane to be tilted. However, there is no satisfactory solution to this problem in the type of compact cameras included in smartphones.

[0009] It would be advantageous if a super-macro camera were equipped on a mobile device capable of tilting the focal plane. means of solving the problem

[0010] In various embodiments, an object-side optical path folding element (O-OPFE) for folding a first optical path (OP1) into a second optical path (OP2); a lens; an image-side optical path folding element (I-OPFE) for folding the second optical path (OP2) into a third optical path (OP3), wherein OP1, OP2, and OP3 are perpendicular to each other; and an I-OPFE actuator; The image sensor comprises an image plane oriented parallel to the XY plane in an exemplary orthogonal XYZ coordinate system, wherein the camera is a super-macro (SM) camera and has a focal plane, wherein at a zero-tilt focal plane position the focal plane is parallel to the XZ plane, and at a zero-tilt I-OPFE position the I-OPFE forms an angle of 45 degrees with both the X-axis and the Z-axis and is parallel to the Y-axis, wherein the I-OPFE actuator operates to tilt the focal plane around a first focal plane rotation axis parallel to the Z-axis by rotational operation of the I-OPFE around a first I-OPFE rotation axis parallel to the Y-axis, and / or tilt the focal plane around a second focal plane rotation axis parallel to the X-axis by rotational operation of the I-OPFE around a second I-OPFE rotation axis parallel to the zero-tilt I-OPFE position and perpendicular to the Y-axis.

[0011] In some embodiments, OP1 and OP3 are oriented parallel to the Y-axis, OP2 is oriented parallel to the X-axis, the image sensor is oriented parallel to the XZ plane, and the I-OPFE actuator is operated to tilt the focal plane around the first focal plane rotation axis by rotational operation of the I-OPFE around a third I-OPFE rotation axis parallel to the Z-axis, and / or to tilt the focal plane around the second focal plane rotation axis by rotational operation of the I-OPFE around a fourth I-OPFE rotation axis parallel to the zero-tilt I-OPFE position and perpendicular to the Z-axis.

[0012] In some embodiments, the SM camera is focused at an object-lens distance of 3.0 to 15 cm.

[0013] In some embodiments, the SM camera is focused at an object-lens distance of 15 to 50 cm.

[0014] In some embodiments, the focal plane is tilted 0 to 20 degrees around the first focal plane rotation axis and / or the second focal plane rotation axis.

[0015] In some embodiments, the I-OPFE is tilted 0 to 40 degrees around the first I-OPFE rotation axis or the third I-OPFE rotation axis to tilt the focal plane 0 to 20 degrees around the first focal plane rotation axis, and / or the I-OPFE is tilted 0 to 40 degrees around the second I-OPFE rotation axis or the fourth I-OPFE rotation axis to tilt the focal plane 0 to 20 degrees around the second focal plane rotation axis.

[0016] In some embodiments, the I-OPFE is tilted 0 to 60 degrees around the first I-OPFE rotation axis or the third I-OPFE rotation axis to tilt the focal plane 0 to 40 degrees around the first focal plane rotation axis, and / or the I-OPFE is tilted 0 to 60 degrees around the second I-OPFE rotation axis or the fourth I-OPFE rotation axis to tilt the focal plane 0 to 40 degrees around the second focal plane rotation axis.

[0017] In some embodiments, the focal plane is tilted from 0 to 40 degrees around the first focal plane rotation axis and / or the second focal plane rotation axis. In some embodiments, the focal plane is tilted from 0 to 60 degrees around the first focal plane rotation axis and / or the second focal plane rotation axis. In some embodiments, the focal plane is tilted from 0 to 80 degrees around the first focal plane rotation axis and / or the second focal plane rotation axis.

[0018] In some embodiments, calibration data is used to calculate a control signal for tilting the focal plane around the first focal plane rotation axis and the second focal plane rotation axis, and the calibration data depends on the distance between the SM camera and the object to be captured.

[0019] In some embodiments, when the focal plane is located at a first distance from the SM camera, in order to tilt the focal plane around the first focal plane rotation axis or the second focal plane rotation axis by a specific focal plane tilt stroke, a first rotation stroke around the first, second, third, or fourth I-OPFE rotation axis is required, and when the focal plane is located at a second distance from the SM camera, in order to tilt the focal plane around the first focal plane rotation axis or the second focal plane rotation axis by the same specific focal plane tilt stroke, a second rotation stroke around the first, second, third, or fourth I-OPFE rotation axis is required, wherein the second distance is greater than the first distance and the first rotation stroke is smaller than the second rotation stroke.

[0020] In some embodiments, the SM camera has an effective focal length (EFL) of 7 to 10 mm. In some embodiments, the SM camera has an EFL of 10 to 20 mm. In some embodiments, the SM camera has an EFL of 20 to 40 mm.

[0021] In some embodiments, the I-OPFE actuator is a voice coil motor.

[0022] In some embodiments, the I-OPFE is a mirror.

[0023] In some embodiments, the O-OPFE is a prism.

[0024] In some embodiments, the lens is divided into two lens groups, the first lens group is located on the object side of the O-OPFE, and the second lens group is located between the image side of the O-OPFE and the object side of the I-OPFE.

[0025] In some embodiments, the SM camera is a scanning camera that scans a scene having each SM field of view.

[0026] In some embodiments, the SM camera may switch between two or more discrete zoom states.

[0027] In some embodiments, the SM camera can be continuously switched between a minimum zoom state and a maximum zoom state.

[0028] In some embodiments, the zoom factor of the maximum zoom state is twice the zoom factor of the minimum zoom state.

[0029] In some embodiments, the zoom factor in the maximum zoom state is at least three times greater than the zoom factor in the minimum zoom state.

[0030] In various embodiments, a mobile device is provided comprising the above or below camera and an application processor (AP) configured to control the tilting of the focal plane.

[0031] In some embodiments, the mobile device is a smartphone.

[0032] In some embodiments, the AP is configured to use calibration data including the rotation angle of the first and / or second and / or third and / or fourth I-OPFE rotation axes and the object-lens distance to control the tilting of the focal plane around the first and / or second focal plane rotation axis.

[0033] In some embodiments, the AP is configured to control the tilting of the focal plane according to user input.

[0034] In some embodiments, the AP is configured to execute an algorithm for automatically tilting the focal plane.

[0035] In some embodiments, the AP is additionally configured to automatically capture an image or image stream after the focal plane is tilted.

[0036] In some embodiments, the focal plane is tilted so that the object of interest captured by the SM camera is entirely in focus.

[0037] In some embodiments, the focal plane is tilted so that the object of interest captured by the SM camera has a specific amount of out-of-focus deblur.

[0038] In some embodiments, the mobile device further includes a wide camera and / or an ultra-wide camera, and the AP is further configured to analyze image data from any one of the SM camera, the wide camera, and the ultra-wide camera in order to automatically tilt the focal plane.

[0039] In some embodiments, the image data is photodiode autofocus image data. In some embodiments, the analysis of the image data uses a protrusion map and / or object detection algorithm.

[0040] In some embodiments, the AP is configured to analyze image data from the wide camera or the ultra-wide camera to steer the field of view (FOV) of the scanning camera toward a specific segment within the FOV of the wide camera.

[0041] In some embodiments, the AP is configured to analyze image data from the wide camera or the ultra-wide camera and switch the SM camera to a specific zoom state for capturing a macro image having a specific magnification and a specific field of view.

[0042] In various embodiments, each wide field of view (FOV) W Wide camera for capturing wide images having ), FOV WA method is provided comprising: providing an electronic mobile device including an SM camera configured to tilt its focal plane as a super-macro (SM) camera for capturing SM images having a smaller SM field of view, and a processor; and configuring the processor to autonomously tilt the focal plane of the SM camera according to an analysis using wide image data and / or SM image data, and to autonomously capture an SM image at a given focal plane tilt.

[0043] In some embodiments, the tilting of the focal plane is performed so that the captured SM image is entirely in-focus.

[0044] In some embodiments, the tilting of the focal plane is performed so that the captured SM image has a specific amount of out-of-focus deblur. Brief explanation of the drawing

[0045] Non-limiting examples of the embodiments disclosed herein are described below with reference to the drawings attached to this specification listed after this paragraph. The drawings and descriptions are intended to illuminate and clarify the embodiments disclosed herein and should not be construed as limiting in any way. Identical elements in different drawings may be indicated by the same number. The elements in the drawings are not necessarily drawn to scale. FIG. 1a illustrates an exemplary SM capture scenario known in the art. Figure 1b shows an SM image captured in portrait mode according to the scenario illustrated in Figure 1a. FIG. 2a illustrates an exemplary SM image capture scenario as disclosed in the present specification. Figure 2b shows an SM image captured in portrait mode according to the scenario illustrated in Figure 2a. FIG. 3a illustrates a cross-sectional view of an example of a double-folded camera of known technology. FIG. 3b illustrates a cross-sectional view of an example of a double-folded camera having a tiltable focal plane disclosed in this specification. FIG. 4a illustrates, by way of example, a camera disclosed herein in a zero-tilt focal plane position. FIG. 4b shows the camera of FIG. 4a in a non-zero tilt focal plane position. FIG. 4c illustrates an example of a method for automatic control of focal plane tilting disclosed in this specification. FIG. 5a illustrates an example of a mobile device including a double-folded SM camera having a tiltable focal plane as disclosed in this specification. FIG. 5b shows a perspective view of a double-folded telecam of known technology. FIG. 6a illustrates an I-OPFE module disclosed in this specification in a plan view. FIG. 6b is a perspective view of the I-OPFE module of FIG. 6a without an I-OPFE and an OPFE holder. FIG. 6c is the same perspective view as FIG. 6b, but shows an I-OPFE module with an I-OPFE and an OPFE holder shown in a perspective view. FIG. 7 schematically illustrates an embodiment of a mobile device capable of tilting the focal plane disclosed in this specification. Specific details for implementing the invention

[0046] The cameras and images below are recognized as "SM" if they support two criteria.

[0047] - M 1:5 to 25:1.

[0048] - Captures objects at an object-lens distance of 30mm to 750mm with an EFL in the range of 7mm to 50mm.

[0049] The 1:5 and 25:1 scales refer to ratios where the physical size of the captured object is 1 and 25, respectively, and the image size of the captured object is 5 and 1, respectively.

[0050] It should be noted that the above definition of SM refers to both camera specifications (e.g., an EFL in the range of 7mm to 50mm) and image (or scene) specifications (e.g., an object-lens distance in the range of 30mm to 750mm). In particular, this implies that the same camera can be used for image capture in both an SM scene (e.g., a scene with an exemplary object-lens distance of 100mm) and another scene (e.g., a scene with an exemplary object-lens distance of 2m). When capturing an SM scene, the camera is referred to as an SM camera. When capturing another scene, the camera is referred to as a telecamera.

[0051] Table 1 plots the depth of field (DOF) (mm) of an SM camera as a function of EFL (mm) and the distance from the camera to the object ("object-lens distance" or "u") (cm). For the DOF calculation, an f-number of f / 3 and a circle of confusion of 1.5 μm were assumed. The DOF is shown for the range of EFL = 8 mm to 40 mm (columns) and u = 2 cm to 150 cm (rows).

[0052] [Table 1]

[0053]

[0054] FIG. 1a illustrates an exemplary SM capture scenario known in the art. A mobile device (100) (e.g., a smartphone) includes an SM camera (102) having a focal plane (104). The focal plane (104) is oriented parallel to the rear of the mobile device (100) and has a specific DOF. The DOF extends to both sides of the focal plane (104). The SM camera (102) captures an object (106). The mobile device (100) is oriented at an angle (α) with respect to the direction of the object (106). Due to the shallow DOF (see Table 1), only a portion of the object (106) can be placed within the DOF area and can be in-focus (the point area of ​​the object (106)). Other parts of the object (106) may be placed outside the DOF area and may be out of focus (white area of ​​the object (106)). The focal plane (104) forms an angle (β) = 90 degrees with the normal (108) of the rear of the mobile device (100). The mobile device (100) is illustrated in a "portrait mode" orientation known in the art, which means that the length dimension of the camera image is parallel to the length dimension of the object. In another embodiment, the mobile device (100) may be in a "landscape mode" orientation as known in the art.

[0055] FIG. 1b illustrates an SM image (110) captured in portrait mode according to the scenario illustrated in FIG. 1a. Only segments of the object (106) that were within the DOF are in focus, and segments of the object (106) that were outside the DOF are out focus. In this scenario, the DOF of the SM camera prevents all-in-focus object capture in a direction parallel to the Y-axis of the image. In another scenario, the DOF of the SM camera can prevent all-in-focus object capture in a vertical direction parallel to the X-axis of the image.

[0056] FIG. 2a illustrates an exemplary SM image capture scenario as disclosed herein. A mobile device (200) (e.g., a smartphone) includes an SM camera (202) having a tiltable focal plane (204). "Tiltable" refers to the fact that the focal plane (204) can be tilted to have a direction not parallel to the mobile device (200). The focal plane (204) forms an angle (β) < 90 degrees with the normal (208) of the rear surface of the mobile device (200).

[0057] The SM camera (202) captures an object (106). As in the scenario of FIG. 1a, the mobile device (200) is oriented at an angle (α) with respect to the direction of the object (106). However, in contrast, the focal plane (204) of the SM camera (202) is tilted around a rotation axis parallel to the Z-axis so that the focal plane (204) is oriented parallel to the object (106). Despite the shallow DOF, all segments of the object (106) are placed within the DOF area and remain in focus. As with the mobile device (100) in FIG. 1a, the mobile device (200) is illustrated in portrait mode.

[0058] FIG. 2b illustrates an SM image (210) captured in portrait mode according to the scenario illustrated in FIG. 2a. All segments of the object (106) are in focus.

[0059] FIG. 3a illustrates a cross-sectional view of a first example of a double-folded camera (300) of known technology. The camera (300) comprises a first object-side optical path folding element ("object OPFE" or "O-OPFE", e.g., a prism or a mirror) (302), a lens (304), a second image-side OPFE ("image OPFE" or "I-OPFE", e.g., a mirror or a prism) (306), and an image sensor (308). The image sensor (308) is oriented in a plane parallel to the XY plane. The camera (300) has a focal plane (310) oriented parallel to the XZ plane, that is, a focal plane (310) perpendicular to the image sensor (308) and parallel to the rear surface of a mobile device (not shown) hosting the camera (300). The optical path ("OP") of the camera (300) is folded twice, from a first OP ("OP1") (312) perpendicular to the focal plane (310) and parallel to the Y-axis, to a second OP ("OP2") (314) parallel to the focal plane (310) and parallel to the X-axis, and then to a third OP ("OP3") (316) perpendicular to the focal plane (310) and parallel to the Z-axis. The I-OPFE (306) is oriented at an angle of 45 degrees with respect to the image sensor, and with respect to both OP2 (314) and OP3 (316). Hereinafter, this orientation of the I-OPFE (306) is referred to as the "zero-position" or "zero-tilt position" of the I-OPFE. In a second embodiment of the double-folded camera (not shown) of the known technology, OP3 (316) is oriented parallel to OP1 (312) (i.e. parallel to the Y-axis), and the image sensor (308) can be oriented in a plane parallel to the XZ plane.

[0060] FIG. 3b illustrates a cross-sectional view of a first example of a double-folded camera (320) having a tiltable focal plane (330) disclosed herein. Similar to the camera (300), the camera (320) includes an O-OPFE (302'), a lens (304'), and an image sensor (308'), which may or may not be identical to each element of the camera (300). Additionally, the camera (320) includes a tiltable I-OPFE (322) (e.g., a mirror or prism). In another embodiment, the lens (304') may be divided into two or more lens groups (not shown), wherein a first lens group containing one or more lens elements may be located on the object side of the O-OPFE (302'), and a second lens group containing one or more lens elements may be located on the image side of the O-OPFE (302') and on the object side of the tiltable I-OPFE (322). In yet another embodiment (not shown), the lens (304') may be located on the object side of the O-OPFE (302'). That is, the lens element may not be located on the image side of the O-OPFE (302'). The OP of the camera (320) is folded twice into three equally divided OPs, as shown in FIG. 3a. In particular, this means that the image sensor (308') is oriented parallel to the XY plane, and the tiltable focal plane (330) (at the zero-position) is oriented parallel to the XZ plane. The tiltable I-OPFE (322) is shown in a zero-tilt position (such as 306 in FIG. 3a) and in a position tilted at a tilt angle φ relative to the zero-tilt position. At the zero-tilt position, the tiltable I-OPFE (322) is oriented parallel to the Y-axis and oriented at an angle of 45 degrees relative to the image sensor (308'), OP2 (314) (which is parallel to the X-axis), and OP3 (316) (which is parallel to the Z-axis).

[0061] Here and below, the first I-OPFE tilt angle "φ" is defined as the rotation angle of the I-OPFE (322) centered on a first I-OPFE rotation axis (324) that is parallel to the Y-axis and can intersect with the geometric center of the reflective surface of the I-OPFE (322). By tilting the I-OPFE (322) around the first I-OPFE rotation axis (324), the tiltable focal plane (330) is tilted around a first focal plane tilt axis parallel to the Z-axis.

[0062] The second I-OPFE tilt angle is defined as the rotation angle of the I-OPFE (322) around a second I-OPFE rotation axis (not shown) parallel to the zero-tilt position of the I-OPFE (322), that is, the second I-OPFE rotation axis forms an angle of 45 degrees in both the X-axis and the Z-axis and is perpendicular to the Y-axis. By tilting the I-OPFE (322) around the second I-OPFE rotation axis, the tiltable focal plane (330) is tilted around a second focal plane tilt axis parallel to the X-axis.

[0063] Depending on the orientation of the object to be captured, such as object (106), it may be desirable to tilt the focal plane (330) around the first focal plane tilt axis, around the second focal plane tilt axis, or around a combination of both. Given the orientation of a specific object to be captured, such as object (106), it may be desirable to tilt the focal plane (330) around the first focal plane tilt axis when the mobile device (e.g., mobile device (700)) including the camera (320) is maintained in a portrait mode (or orientation) known in the art for image capture, and it may be desirable to tilt the focal plane (330) around the second focal plane tilt axis when the mobile device including the camera (320) is maintained in a landscape mode known in the art for image capture. Alternatively, the opposite may be possible. This means that the decision on whether to tilt the focal plane around the first or second tilt axis depends on both the orientation of the object to be captured and the orientation of the camera hosting the mobile device.

[0064] To tilt the focal plane of the double-folded camera of the second example (i.e., OP3 is parallel to the Y-axis and the image sensor is oriented in a plane parallel to the XZ plane) around a first focal plane tilt axis parallel to the Z-axis, I-OPFE can be tilted around an I-OPFE tilt axis parallel to the Z-axis. To tilt the focal plane of the double-folded camera of the second example around a second focal plane tilt axis parallel to the X-axis, I-OPFE can not only be tilted around an I-OPFE tilt axis parallel to I-OPFE, but is also oriented perpendicular to the Z-axis to form an axis of 45 degrees with respect to the X-axis and Y-axis.

[0065] The tiltable focal plane (330) of the camera (320) is a tilted focal plane that is not perpendicular to the OP (312) but forms an angle (θ) with a plane (334) oriented perpendicular to the OP (312). Referring to FIGS. 1a and 2a, the camera (320) can be used to capture, for example, an object (106) that is entirely in-focus when focal plane tilting is performed so that θ = α is approximately satisfied (α is the angle between the mobile device (200) and the object (106), see FIG. 2a). Here, "approximately satisfied" means that an object such as the object (106) can be captured entirely in-focus even at an angle (θ) = α ± Δ due to the finite thickness of the DOF (as shown in FIGS. 1a and 2a). The magnitude of Δ depends on the angle (α) and the object-lens distance ("u").

[0066] In the case of a tiltable focal plane such as 330, the object-lens distance (u) for an in-focused object is measured along the OP (312) from the center of the FOV to the camera lens. As can be seen in FIG. 3b, tilting of the focal plane does not change u. In a double-folded camera (320), consecutive OPs (e.g., OP1 and OP2, OP2 and OP3, etc.) are oriented perpendicularly to each other. In other examples, consecutive OPs may be oriented at different angles. For example, they may differ from 90 degrees to 10 degrees or more than 20 degrees. In particular, the O-OPFE and / or I-OPFE in the zero-tilt position here may be oriented to fold the optical path at a different angle.

[0067] FIGS. 4a and 4b illustrate the relationship between the tilt angle φ and θ (as defined in FIG. 3b). For simplicity, the optical system of the double-folded camera (320) is represented as a simpler general or non-folded camera (400). For example, for an angle (φ) that is not too large up to φ=20 degrees, it can be assumed that tilting the image sensor of a non-folded camera, such as the image sensor (404) (similar to 308'), by an angle (φ) with respect to the zero-tilt position at 90 degrees for the OP (412) (similar to 316) of the camera, results in a tilt angle (θ) similar to that obtained by tilting an I-OPFE, such as the I-OPFE (322), by the same tilt angle (φ) with respect to the zero-tilt position (at 45 degrees for the OP (314) and OP (316) of the double-folded camera). This means that the focal plane tilting described in this specification can be performed by a double-folded camera as well as by a standard or non-folded camera. In a double-folded camera such as 320, an I-OPFE such as 322 is tilted for focal plane tilting, whereas in a standard camera, the camera lens is tilted relative to the camera's image sensor and the mobile device hosting the standard camera, or the camera's image sensor is tilted relative to the camera lens and the mobile device hosting the standard camera.

[0068] FIG. 4a illustrates an exemplary camera (400) in a zero-tilt focal plane position. The camera (400) includes a lens (402) and a tiltable image sensor (404) and is focused on a focal plane (410) perpendicular to the Y-axis for some object-lens distance (u) covering a specific horizontal field of view (FOV) of the scene. Tiltable image sensors are known in the art. In FIG. 4a, the image sensor (404) is shown in a zero-tilt position. In the zero-tilt position, the image sensor (404) is oriented perpendicular to the Y-axis. As is known in the art, the optical system is designed so that the focal plane (410) is oriented parallel to the image sensor (404), parallel to the rear of a mobile device (not shown) hosting the camera (400), and perpendicular to the OP (412) of the camera (400). Three exemplary field (or "object") points (having ray numbers 1 to 3) are shown, which are a first field point ("FP1") located to the right of the center of the FOV, a second field point ("FP2") located in the center of the FOV, and a third field point ("FP3") located to the left of the center of the FOV. Designing the optical system such that the focal plane (410) of the camera (400) is oriented parallel to the image sensor (404) means that the fields (1 to 3) are in focus at the same object-lens distance (u). Since the field points at u are in focus when all rays emanating from these field points propagate the same optical path length ("OPL"), all rays of FP1, FP2, and FP3 propagate a specific OPL (OPL1, OPL2, and OPL3, respectively). OPL1, OPL2, and OPL3 are placed on the focal plane (410). The width of the sensor (404) is indicated as "S", and the width of the FOV is indicated as "X".

[0069] FIG. 4b illustrates a camera (400) in a non-zero tilt focal plane position. The image sensor (404) is now tilted and oriented in a plane that is not perpendicular to the Y-axis, but forms an angle (φ) with the X-axis. When the image sensor (404) is tilted, OPL1, OPL2, and OPL3 are not placed on a focal plane (410) perpendicular to the Y-axis, but rather on a focal plane (410) that forms an angle ≠ 90 degrees with respect to the Y-axis and an angle (θ) with respect to the X-axis. The object-lens distance (u) and lens-image distance (v) are not constant with respect to x, as shown in FIG. 4a, but depend on x. That is, u = u(x) and v = v(x). In the case of FP1, u1 = u center +Δu1 and v1=v center It is +Δv1. In the case of FP3, u3=u center +Δu3 and v3=v center It is +Δv3. "u center " and "v center " represents the object-lens distance at the FOV center position and the lens-image distance at the image sensor center position, respectively. For FP2, there are no changes, and u2=u center and v2=v center ... Since FP1 and FP3 are symmetrically positioned with respect to FP2, we assume Δu1=-Δu3 and Δv1=-Δv3. Below, only FP1 will be considered. Unless otherwise explicitly stated, the object-lens distance (u) here and below is u center It represents.

[0070] The thin lens equation now also depends on x (Equation 1).

[0071]

[0072] Tables 2 to 4 illustrate the relationship between the focal plane tilt stroke (θ) and the combination of SM camera examples (characterized by EFL (mm)) and object-lens distance (u) (mm) for three image sensor tilt angles φ1 = 5 degrees (Table 2), φ2 = 20 degrees (Table 3), and φ3 = 40 degrees (Table 4).

[0073] [Table 2] (φ1 = 5 degrees)

[0074]

[0075] [Table 3] (φ2 = 20 degrees)

[0076]

[0077] [Table 4] (φ3 = 40 degrees)

[0078]

[0079] From Tables 2 to 4, the specific focal plane tilt stroke (θ P A specific I-OPFE rotation stroke (φ) to tilt the focal plane around the first focal plane rotation axis or the second focal plane rotation axis by )(particularly "P") P ) is required, and here φ P It is evident that is smaller for closer object-lens distances (u). This applies to two different object-lens distances (u1, u2) [here, u1 <u2]에서 동일한 초점면 틸트 스트로크(θ P To perform ), a smaller amount of rotation of I-OPFE in u1 than in u2 (i.e., a smaller rotation stroke (φ) P This means that )) is required. That is, for a specific amount of focal plane tilting, the following is satisfied: the smaller the object-lens distance, the more the I-OPFE rotation stroke (φ) required to achieve a specific amount of focal plane tilting is P ) also becomes smaller.

[0080] The tilting of the focal plane can be controlled by a human user or automatically controlled by an algorithm.

[0081] FIG. 4c illustrates an example of a method for automatic control of focal plane tilting disclosed herein. In the first step (422), an object of interest ("OOI") detector, such as an OOI detector (742) (Fig. 7), can detect the OOI and its location within the FOV of the SM camera. The OOI can be selected by the user and / or suggested by an algorithm, for example, by using a protrusion map or an object detection algorithm. In some embodiments, the OOI detector (742) can use wide (W) or ultrawide (UW) image data to detect the OOI. In the step (424), a tilt estimator, such as a tilt estimator (744) (Fig. 7), can calculate the relative tilt between the focal plane of the SM camera and the OOI. For tilt estimation, image data from the SM, W, or UW camera, for example, RGB or photodiode autofocus (PDAF) image data, can be used. In another embodiment, depth map data from an additional camera, for example, a Time-of-Flight (TOF) camera, may be used. Based on the tilt estimation, in step (426), the I-OPFE control unit (746) (Fig. 7) may use calibration data such as that presented in Tables 2 through 4 as a Look-Up-Table (LUT) and thereby calculate an I-OPFE control signal transmitted to an I-OPFE actuator (e.g., I-OPFE actuator (724)) that tilts the focal plane to align the focal plane of the SM camera with the direction of OOI.

[0082] Calibration data may include object-lens distances and tilt angles of four I-OPFE rotation axes to control the tilting of the focal plane around two focal plane rotation axes. Calibration data may include object-lens distances and tilt angles of two I-OPFE rotation axes to control the tilting of the focal plane around one focal plane rotation axis. In another embodiment, the focal plane may not be aligned with the OOI direction, but the focal plane and the OOI direction may be tilted relative to each other in a defined manner, so that an artistic (or aesthetic) combination of in-focus and out-focus segments is formed in the SM image. For example, the artistic combination of in-focus and out-focus segments in the SM image may be a bokeh effect that can be controlled through focal plane tilting. In some examples, the artistic combination may be used to highlight a specific object or a specific segment of the scene.

[0083] In other examples referring to a standard or non-folded camera, in step (426), the lens tilt controller or image sensor tilt controller may use calibration data to calculate a lens control signal or image sensor control signal that tilts the focal plane by tilting the lens of the camera and / or the image sensor of the camera to align the orientation of the focal plane and the OOI.

[0084] In another example referring to a camera having an optical path that folds two or more times, for example, three or four or more times, in step (426), the OPFE tilt controller may use calibration data to calculate an OPFE control signal that tilts the focal plane by tilting one of the camera's OPFEs by an OPFE actuator to align the orientation of the focal plane and the OOI. Each camera having an optical path that folds three or four times includes two or more OPFEs (e.g., prisms and / or mirrors) and has four optical path segments (OP1-OP4) or five optical path segments (OP1-OP5), respectively, where consecutive OPs (e.g., OP1 and OP2, etc.) may be oriented at a 90-degree angle relative to each other or at other angles.

[0085] In step (428), an SM image is captured. In some embodiments, multiple SM images are captured to form a video stream together. In some examples, different artistic combinations of in-focus segments and out-focus segments in specific SM images of the video stream can be used to highlight specific objects or specific segments of the scene.

[0086] In some embodiments, and in the case of a scanning SM camera, the W image data is the native FOV of the scanning SM camera N It can be used to steer into OOI.

[0087] FIG. 5a illustrates an example of a mobile device (500) comprising a double-folded SM camera (502) having a tiltable focal plane as disclosed herein. In addition to a telecamera acting as the SM camera (502), the mobile device (500) has a wide FOV WIt includes a multi-camera comprising a wide (or "W") camera (504) having a wide FOV and an ultra-wide (or "UW") camera (506) having an ultra-wide FOVUW. The multi-camera is integrated into the rear (or world-facing) surface (508) of the mobile device (500). The mobile device (500) may include some or all of the components of the mobile device (700). The mobile device (500) may be, for example, a smartphone.

[0088] FIG. 5b illustrates a perspective view of a double-folded telecamera (520) of known technology. The camera (520) includes an object OPFE (522) (e.g., a prism), a lens (524), an I-OPFE (526) (e.g., a mirror), and an image sensor (528). The camera (520) has three OPs [OP1 (532), OP2 (534), and OP3 (536)]. The double-folded telecamera (520) is implemented according to the first example as described above. That is, OP1 (532) is parallel to the Y-axis, OP2 (534) is parallel to the X-axis, and OP3 (536) is parallel to the Z-axis. The image sensor (528) is oriented in a plane parallel to the XY plane.

[0089] FIGS. 6a through 6c illustrate a tiltable I-OPFE module (600) that enables the OPFE to be tilted within a range of approximately ±30 degrees. FIG. 6a illustrates the I-OPFE module (600) in a top view. The I-OPFE module (600) includes an I-OPFE (602) (e.g., a mirror or prism) within an OPFE holder (604) and an OPFE tilt mechanism (610). The OPFE tilt mechanism (610) rotates the I-OPFE (602) around a single rotation axis (606) located on a pivot ball (612) and parallel to the Z-axis. Support balls (614, 616) mediate the rotation. The OPFE tilt mechanism (610) also includes a voice coil motor ("VCM") actuator (620) equipped with a VCM magnet (622) and a VCM coil (624) for rotational operation. VCM operation is detected by a sensing and compression magnet (632) and a Hall sensor (634).

[0090] FIG. 6b is a perspective view of an OPFE module (600) without an I-OPFE (602) and an OPFE holder (604). A yoke (636) can be seen. FIG. 6c is the same perspective view as FIG. 6b, but shows an OPFE module (600) with an I-OPFE (602) and an OPFE holder (604) shown. A rotation axis (606) is indicated.

[0091] By using two separate magnets (622 and 632), the OPFE module (600) enables separation of detection and operation. The Hall sensor (634) is decoupled from the magnetic field of the coil (624). The rotation axis (606) is located at a relatively long distance from the VCM actuator (620), providing a large lever for rotational operation. The rotation axis (606) is located at a relatively short distance from the Hall sensor (634), so that detection of large rotational OPFE operation can be performed within a small stroke.

[0092] FIG. 7 schematically illustrates an embodiment of a mobile device (e.g., a smartphone) capable of tilting the focal plane, disclosed herein and numbered 700. The mobile device (700) comprises an SM camera (710), which is a double-folded camera comprising a lens (712), an image sensor (714), an O-OPFE (716) and an O-OPFE actuator (718), an I-OPFE (722) and an I-OPFE actuator (724) for rotational operation of the O-OPFE and I-OPFE, respectively. Rotational operation of the I-OPFE is performed according to the method disclosed herein. Calibration data for rotational operation of the I-OPFE according to a control signal from the I-OPFE control unit (744) is stored in the first memory (726) (e.g., EEPROM or "electrically eraseable programmable read-only memory") or the third memory (750) (e.g., NVM or "non-volatile memory"). The SM lens (712) may have a fixed EFL providing a fixed zoom factor (ZF), or an adaptive EFL providing an adaptive ZF. The adaptation of the focal length may be discrete or continuous. That is, it may be a discrete number of variable focal lengths to provide two or more discrete zoom states with specific ZFs, or the adaptation of the ZF may be continuous. The O-OPFE actuator (718) and the I-OPFE actuator (724) can actuate the O-OPFE (716) and I-OPFE (722), respectively, for focal plane tilting, OIS, and for FOV scanning when the SM camera (710) is a scanning folded camera. The I-OPFE actuator (724) can actuate the I-OPFE (722) around two vertical axes according to the desired focal plane tilt direction.

[0093] The mobile device (700) further includes a W camera module (730) including a W lens (732), an image sensor (734), and a second memory (736).

[0094] The mobile device (700) may further include an application processor (AP) (740). The AP (730) includes an OOI detector (742) and a tilt estimator (744) for estimating the relative tilt (i.e., tilt) between the mobile device (700) and the OOI to be captured. The tilt estimator of 744 is used by the I-OPFE controller (746) to calculate a control signal for an I-OPFE actuator (724) that rotates the I-OPFE (722) according to a control signal.

[0095] The SM camera (710) may have, for example, an EFL of 7 mm to 50 mm or more, a diagonal FOV of 5 to 40 degrees, and an f number of f / # = 1.5 to 7.5. The W camera (730) may have, for example, an EFL of 2.5 mm to 15 mm, a diagonal FOV of 50 to 130 degrees, and an f / # = 1.0 to 3.5.

[0096] The SM camera (710) may be a scanning camera. By the rotational operation of the O-OPFE (716) and I-OPFE (722), the native (diagonal) FOV (FOV) of the SM camera (710) N ) can be steered to scan the scene. FOV N It can be 10 to 40 degrees, and FOV N The scanning range of can be ±5 degrees to ±35 degrees. For example, a 20-degree FOV N and ±20 FOV N A scanning SM camera with scanning capabilities covers a tele-FOV of 60 degrees.

[0097] Although the present disclosure has been described in terms of specific embodiments and generally associated methods, modifications and substitutions of the embodiments and methods will be apparent to those skilled in the art. The present disclosure is not limited by the specific embodiments described herein, but should be limited only by the appended claims.

[0098] All references mentioned in this specification are incorporated herein in their entirety to the same extent that each individual reference is specifically and individually incorporated herein as a reference. Furthermore, any citation or identification of any reference in this application shall not be construed as an acknowledgment that such reference is available as prior art of this application.

Claims

Claim 1 Each wide field of view (FOV) W Wide camera for capturing wide images having ), FOV W Each smaller SM field of view (FOV) SM A method comprising: providing an electronic mobile device including a super-macro (SM) camera for capturing an SM image having a focal plane; and a processor, wherein the SM camera has a focal plane; and configuring the processor to autonomously tilt the focal plane of the SM camera according to an analysis using wide image data and / or SM image data, and to autonomously capture an SM image at a given focal plane tilt. Claim 2 A method according to claim 1, wherein the tilting of the focal plane is performed so that the captured SM image is entirely in focus. Claim 3 A method according to claim 1, wherein the tilting of the focal plane is performed such that the captured SM image has a specific amount of out-of-focus deblur. Claim 4 In claim 1, the method wherein the SM camera has an effective focal length of 7 to 10 mm. Claim 5 In claim 1, the method wherein the SM camera has an effective focal length of 10 to 20 mm. Claim 6 In claim 1, the method wherein the SM camera has an effective focal length of 20 to 40 mm. Claim 7 A method according to claim 1, further comprising the step of configuring the processor so that the SM camera is focused at an object-lens distance of 3.0 to 16 cm. Claim 8 A method according to claim 1, further comprising the step of configuring the processor so that the SM camera is focused at an object-lens distance of 15 to 50 cm. Claim 9 In claim 1, the method wherein the focal plane is tilted from 0 to 20 degrees. Claim 10 A method according to claim 1, wherein the SM camera comprises an optical path folding element (OPFE). Claim 11 In claim 10, the tilting of the focal plane of the SM camera is obtained by rotating the OPFE. Claim 12 In claim 11, the tilting of the focal plane of the SM camera follows two axes, and the method is obtained by rotating the OPFE along the two axes. Claim 13 A method in which, in any one of claims 1 to 12, the electronic mobile device is a smartphone. Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete

Citation Information

Patent Citations

  • Optical equipment

    JP4973219B2

  • Camera module

    KR1020120080443A

  • Folded camera with optical image stabilization

    KR1020200063240A

  • Optical zoom device with focus-adjustable lens core

    KR1020200096243A

  • Optical Imaging System and Portable Terminal

    KR1020210027185A