Apparatus and method for optical image stabilization

Abstract

The present disclosure provides an apparatus and method for optical image stabilization. The apparatus or method is for compensating an undesired rotational movement of a handheld electronic device comprising a tele-zoom folded camera, wherein the compensation depends on the undesired rotational movement and on a viewpoint of the tele-zoom folded camera.

Description

[0001] Related applications

[0002] This application is a divisional application of application number 202180004818.9 (PCT application number PCT / IB2021 / 056617), filed on July 22, 2021, entitled "Optical image stabilization in a scanning folding camera". Technical Field

[0003] The examples disclosed in this article generally relate to digital cameras, and in particular to the correction of images obtained using folding digital cameras. Background Technology

[0004] Compact digital cameras with folded optics are known, also referred to as "folding cameras," see, for example, co-owned international patent application PCT / IB2016 / 057366. In handheld electronic devices such as smartphones and tablets (also referred to herein as "handheld devices"), a folding telephoto camera is often part of a multi-camera system, accompanied by one or more additional cameras, such as an ultra-wide-angle camera and a wide-angle camera. The field of view (FOV) of an ultra-wide-angle camera... UW Larger than a wide-angle camera, which has a larger field of view (FOV) than a wide-angle camera. T A telescope with a larger FOV W (Assuming that the image sensor sizes are similar).

[0005] Figure 1A A folding telephoto camera, designated 100, is schematically shown in a perspective view. Camera 100 includes a lens 102 having a lens optical axis 110, an optical path folding element (OPFE) 104, and an image sensor 106. The OPFE 104 folds a first optical path from an object, scene, or panoramic view portion 114 along an axis 108 substantially parallel to the X-axis into a second optical path along an axis 110 substantially parallel to the Z-axis. Camera 100 is designed such that the OPFE 104 rotates relative to the image sensor about axis 110 (X-axis), i.e., in the YZ plane, as indicated by arrow 112. In other words, the folding telephoto camera 100 is a "scanning telephoto camera" (STC). Figure 1B The optical path folding element 104 is shown after being rotated 30 degrees from the zero position.

[0006] Figure 1CA top view shows a handheld device 120 comprising an STC 100 with a lens 102, an optical path folding element 104, and an image sensor 106. A device normal (“N”) is orthogonal to a screen 116 of the device 120 and faces the observer. The optical axis of the camera is parallel to the X-axis. In other examples, the STC 100 may be contained within 120, thus the optical axis of the camera is parallel to the Y-axis.

[0007] The image is acquired from a camera's point of view (POV). The point of view is defined by a vector that originates at the camera's aperture position and terminates at an object point at the center of the FOV (see...). Figure 3A The two viewpoint vectors 324 and 328 correspond to two fields of view (FOV). T (326 and 332). Besides the viewpoint vector, one can also refer to the FOV center direction vector (FOV center direction, FOVCD). As an example, in spherical coordinates (r, θ, φ) defined according to ISO conventions, the viewpoint of a camera at r = 0 is defined by (1, θ, φ), with the polar angle θ and azimuth angle φ defining the position of the object point at the center of the telescope's FOV. The length of the viewpoint vector may be 1 (a unit vector), or it may have a constant length (e.g., EFL), or it may have a variable length, for example, to lie on a specific plane.

[0008] For example, as described and referenced in the jointly owned PCT patent application number PCT / IB2016 / 057366. Figure 1A-1B The rotation of the optical path folding element can be performed around the X-axis and around the Y-axis, so as to... Figure 1A-1B In two-dimensional (2D) space, a "scan" is performed using the field of view (FOV).

[0009] Modern cameras included in handheld devices typically include optical image stabilization (OIS), which mitigates unwanted camera movement caused by a user's hand movements (often referred to as "shake"). With OIS, optical components are moved to reduce movement of the imaged object on the camera's image sensor. The lens module and / or the image sensor and / or the optical path folding element and / or the entire camera can be moved. An inertial measurement unit (IMU) included in the handheld device provides motion data along six degrees of freedom, i.e., and references... Figure 1CLinear motion in XYZ, roll (tilt) around the Z-axis, yaw (tilt) around the Y-axis, and pitch (tilt) around the X-axis. Typically, OIS is used only for pitch and yaw rotation compensation, not roll rotation, because pitch and yaw rotation account for the majority of image degradation caused by hand shake. The coordinate systems of the IMU, the ordinary (i.e., non-scanning) camera, and the handheld device can be aligned and do not change over time. This is invalid for an STC. When a FOV scan is performed, the relationship between the coordinate system of the handheld device and the coordinate system of the STC does change. Therefore, OIS as known in the art cannot be used for hand motion compensation in an STC.

[0010] OIS is needed for scanning telescopes, which would be advantageous. Summary of the Invention

[0011] For simplicity, the terms “electronic device,” “electronic handheld device,” “handheld device,” or simply “device” are used interchangeably. For simplicity, the term “smartphone” can be used to refer to all electronic handheld devices with a scanning folding camera and the methods for implementing OIS in such cameras described herein.

[0012] In various embodiments, a variety of telescopic folding cameras are provided for compensating for an undesirable rotational movement of a handheld electronic device including such a camera, wherein the compensation depends on the undesired rotational movement and on a point of view (POV) of the telescopic folding camera.

[0013] In various embodiments, a one-handed electronic device includes:

[0014] A telephoto folding camera, comprising: an optical pathfolding element (OPFE) for folding light from a first optical path toward a second optical path, the first optical path forming an angle of less than 90 degrees with a normal of the device, the second optical path being substantially orthogonal to the normal of the device; a lens having a lens optical axis along the second optical path; and an image sensor, wherein the device is a handheld electronic device; and an optical pathfolding element actuator for tilting the optical pathfolding element in one or more directions to point a point of view (POV) of the telephoto folding camera toward a portion of a scene;

[0015] A motion sensor for sensing an undesired rotational movement of the device; and

[0016] At least one actuator is provided for moving at least one component of the telephoto folding camera to compensate for the undesired rotational movement of the device, wherein the compensation depends on the undesired rotational movement of the device and on the viewpoint of the telephoto folding camera.

[0017] In some embodiments, the undesired rotational motion occurs around the normal to the device.

[0018] In some embodiments, a device further includes a wide-angle camera, the wide-angle camera having a field of view (FOV). W (field of view FOV) W () greater than the field of view (FOV) of the telescope camera T .

[0019] In some embodiments, sensing the rotational motion includes measuring the rotational motion in three directions.

[0020] In some embodiments, the actuator for moving the components of the telephoto folding camera to compensate for undesirable rotational movements of the device is the optical path folding element actuator, which is used to tilt the optical path folding element in one or more directions to point a viewpoint of the telephoto folding camera at a portion of a scene.

[0021] In some embodiments, moving the components of the telephoto folding camera to compensate for undesirable rotational movements of the device includes moving the lens.

[0022] In some embodiments, moving the components of the telephoto folding camera to compensate for undesirable rotational movements of the device includes moving the image sensor.

[0023] In some embodiments, a device further includes a processing unit configured to perform a coordinate transformation to align the coordinates of the telephoto camera with the coordinates of the handheld device, and vice versa.

[0024] In some embodiments, the apparatus further includes a processing unit configured to perform a coordinate transformation that aligns the coordinates of a reference coordinate system with the coordinates of the handheld device and the telescope.

[0025] In some embodiments, the coordinate transformation is performed using the Rodrigues' rotation formula.

[0026] In some embodiments, the motion sensor includes an inertial measurement unit (IMU).

[0027] In some embodiments, a device further includes a microcontroller unit (MCU) configured to read out the motion sensor and to provide control signals to the rotational motion compensation actuator. In some embodiments, the microcontroller (MCU) is included in an application processor (AP).

[0028] In some embodiments, an apparatus further includes an application processor configured to provide a viewpoint control signal to the optical path folding element actuator to tilt the optical path folding element.

[0029] In various embodiments, a variety of methods are provided, including:

[0030] A handheld device is provided including a telescope folding camera, the telescope folding camera comprising: an optical path folding element (OPFE) for folding light from a first optical axis toward a second optical axis, the first optical axis forming an angle of less than 90 degrees with a normal of the device, the second optical axis being substantially orthogonal to a normal of the device; a lens having a lens axis along the second optical axis; and an image sensor;

[0031] An optical path folding element actuator is provided, the optical path folding element actuator being used to tilt the optical path folding element in one or more directions to point a point of view (POV) of the telephoto folding camera at a portion of a scene;

[0032] Sensing an unwanted rotational movement of the device; and

[0033] Compensation for the undesired rotational motion, wherein the compensation depends on the undesired rotational motion and on the viewpoint of the telephoto folding camera.

[0034] In some embodiments, compensating for the undesired rotational movement includes moving a component of the telescope folding camera.

[0035] In some embodiments, compensating for the undesired rotational movement includes compensating for a rotational movement about the normal direction of the device.

[0036] In some embodiments, a method further includes performing a coordinate transformation to align the coordinates of the telescope with the coordinates of an inertial measurement unit (IMU).

[0037] In some embodiments, a method further includes performing a coordinate transformation to align the coordinates of the inertial measurement unit with the coordinates of the telescope.

[0038] In some embodiments, a method further includes performing a coordinate transformation to align the coordinates of a reference coordinate system with the coordinates of the inertial measurement unit and the coordinates of the telescope.

[0039] In some embodiments, performing the coordinate transformation includes performing the transformation using the Rodrigues rotation formula.

[0040] In some embodiments, sensing an unwanted rotational movement of the device includes sensing the unwanted rotational movement in three directions.

[0041] In some embodiments, the compensation for the undesired rotational movement of the device includes rotating the optical path folding element.

[0042] In some embodiments, the compensation for the undesired rotational movement of the device includes moving the lens.

[0043] In some embodiments, compensating for the undesired rotational movement of the device includes moving the image sensor.

[0044] In some embodiments, compensating for the undesired rotational motion includes calculating a changed viewpoint in the X direction caused by the undesired rotational motion using the following equation: P F P =(P I ·cos(hnd_pitch)+cross(P I ,R P sin(hnd_pitch) + R P ·(dot(P I ,R P )·(1-cos(hnd_pitch)))).

[0045] In some embodiments, compensating for the undesired rotational motion includes calculating a changed viewpoint in the Y direction caused by the undesired rotational motion using the following equation: P F Y =(P I ·cos(hnd_yaw)+cross(P I ,R Y sin(hnd_yaw)+R Y ·(dot(P I ,R Y )·(1-cos(hnd_yaw)))).

[0046] In some embodiments, compensating for the undesired rotational motion includes calculating a changed viewpoint in the X direction caused by the undesired rotational motion using the following equation: P F R =(P I ·cos(hnd_roll)+cross(P I ,R R sin(hnd_roll) + R R ·(dot(P I ,R R )·(1-cos(hnd_roll)))).

[0047] In some embodiments, compensating for the undesired rotational motion includes calculating a direction of a changed viewpoint caused by the undesired rotational motion in the X, Y, and Z directions using the following equation: P F '=P I +(P I -P F P )+(P I -P F Y )+(P I -P F R ).

[0048] In some embodiments, compensating for the undesired rotational motion includes calculating a vector of altered viewpoint caused by the undesired rotational motion in the X, Y, and Z directions using the following equation: P F =P F '·EFL T / P F ' z . Attached Figure Description

[0049] Non-limiting examples of the embodiments disclosed herein are described below with reference to the accompanying drawings, which are set forth later in this paragraph. The drawings and description are intended to illustrate and clarify the embodiments disclosed herein and should not be construed as limiting in any way. Identical elements in different drawings may be indicated by the same reference numerals.

[0050] Figure 1A A perspective view of a known folding scanning camera is schematically shown;

[0051] Figure 1B It shows Figure 1A The image shows the optical path folding element in the telescope rotated 30 degrees from its zero position.

[0052] Figure 1C Examples are shown Figure 1A -B shows a scanning camera that is integrated into a smartphone as a "backward" or "world-facing" camera;

[0053] Figure 2A An exemplary smartphone is shown, comprising a first scanning telephoto camera and a second wide-angle camera at a zero position;

[0054] Figure 2B It shows Figure 2A The smartphone, wherein the telephoto camera is in a non-zero position;

[0055] Figure 2C It shows Figure 2A The smartphone, wherein the telephoto camera is in another non-zero position;

[0056] Figure 3A A two-dimensional (2D) graph is shown for deriving a coordinate system of the telescope camera;

[0057] Figure 3B This illustrates the motion of the rotating device caused by hand tremors. Figure 3A The impact of 2D charts.

[0058] Figure 3C A flowchart illustrating the main steps of a method for scanning the OIS of the telescope camera disclosed herein is shown.

[0059] Figure 4A An embodiment of a handheld device is schematically illustrated in a block diagram, the embodiment including a multi-aperture camera having at least one scanning telescope as disclosed herein; and

[0060] Figure 4B Another embodiment of a handheld device is illustrated schematically in block diagram form, which includes a multi-aperture camera having at least one scanning telescope camera disclosed herein. Detailed Implementation

[0061] Figure 2A An exemplary smartphone 200 is shown, which includes an STC 202 in a zero position and a wide-angle camera P204. The wide-angle camera 204 is not a scanning camera, and its viewpoint ("POV") is... W The device normal N is parallel to the smartphone's normal (parallel to the Z-axis). The device normal N is parallel (or anti-parallel) to the normal with the largest area on the surface of the smartphone 200. The coordinate system of the smartphone 200's IMU (e.g., Figure 4A and 4B IMU in

[0062] 460 (not shown here) can be aligned with a coordinate system of smartphone 200, for example Figure 2A The coordinate system shown, wherein the three axes of the coordinate system are parallel to the three axes of symmetry of the smartphone 200, such that the Z-axis of the IMU (and smartphone 200) coordinate system is parallel to the POV. W The viewpoint ("POV") of STC202 T ") points to its zero position ("POV") T,0 ”), corresponding to such Figure 1A The diagram shows the rotational state of a folding optical path element. (POV) T At zero position, the IMU's coordinate system, wide-angle camera 204, and STC 202 are aligned.

[0063] In the first exemplary method for OIS (“Example 1”), consider the OIS of a wide-angle camera 204, which (for simplicity) can only correct for pitch rotation. To detect unwanted hand movements, data can be collected from the IMU (“X”). IMU The pitch rotation value about the X-axis is read out, and the lens is moved by a specific amount, for example, in a specific direction (dir1), wherein the amount of movement (or travel) is related to the X-axis. IMU Proportional, that is, the lens travel S W Satisfy S W =C W ·X IMU (with some constant C) W This also applies to the STC 202's OIS at position zero. This can be achieved by pressing S in dir1. T =C T ·X IMU (with some constant C) T Moving the lens can compensate for hand movements.

[0064] Figure 2B The image shows the STC 202 in a non-zero position with the smartphone 200. POV T Compared to POV W It has an angle of α degrees. For example, for α = 30 degrees, this corresponds to... Figure 1B The optical path folding element is shown in its rotated state. The coordinate system of the IMU, the wide-angle camera 204, and the STC 202 are no longer aligned.

[0065] Consider Example 1 (hand movement in the pitch direction) of the STC 202 in a non-zero position. OIS of the wide-angle camera 204 can be performed as in Example 1. However, for the OIS of the STC 202, the method of Example 1 no longer allows for hand movement compensation, i.e., (typically) no C... T Therefore, by pressing ST =C T ·X IMU Moving the lens can compensate for hand movements. This is because the coordinate system of the STC202 and the IMU are no longer aligned.

[0066] For a second exemplary method of OIS (“Example 2”), please refer to Figure 2C .and Figure 2A In comparison, POV T Rotated 90 degrees around the Y-axis, i.e., POV T and POV W They are perpendicular to each other. Similar to Example 1, we consider the OIS of the wide-angle camera to be used only for pitch and rotation correction. Hand movements can be detected by reading the rotation X. IMU The IMU value and the result obtained by pressing S in dir1 W =C W ·X IMU (with some constant C) W The movement of one lens (not shown) of the wide-angle camera is fully compensated. However, hand movements cannot be compensated by pressing S in dir1. T =C T ·X IMU (with some constant C) T The movement of a lens (not shown, but similar to lens 102) in the STC is compensated for. Instead, the rotation direction must be changed from dir1 to a specific direction dir2, different from dir1. This hand movement can be achieved by pressing S in dir2. T =C T ·X IMU The STC lens is moved to compensate. Generally, for an STC, the OIS axis depends on the viewpoint or the scanning state of the STC, and is therefore not constant, as is the case with a wide-angle camera.

[0067] Figure 3A A two-dimensional (2D) diagram 320 for deriving the STC using a coordinate system is shown. The STC aperture 322 is located at coordinates (0,0,0). Zero-state STC POV T (POV T,0 )324 corresponds to the line N parallel to the normal of a device (see Figure 2A -C) is a first optical path and can have coordinates (0,0,EFL) T ), of which EFL T It's STC's EFL. FOV T 326 corresponds to STC at POV T,0 FOV at 324 T It also shows the corresponding FOV. T332 Expected or target POV T 328(“POV T,T ”).

[0068] Figure 3B The image shows the handheld device, including the STC, after undergoing a rotational "rolling" motion about the Z-axis. Figure 3A A 2D chart 320, for example, due to the user's hand movements. POV T,0 324 remained unchanged. However, the corresponding FOV... T Transform into a rotating FOV T 326'. Conversely, the rotational motion will cause the POV to... T,T 328 becomes POV T,T 328'. POV (e.g., POV) responding to the movement of a rotating device. T,T The variation of 328) depends not only on the rotation angle or the amount of rotation, but also on the POV. T The location.

[0069] Figure 3C The main steps of the method for STC OIS disclosed herein are illustrated in flowchart form.

[0070] In the first step 302, it is triggered by a human user or a program and by an FOV scanner 442. Figure 4A A command to process FOV T The scan is guided to a region of interest (ROI) within a scene. This scanning can be performed by rotating an optical path folding element using an optical path folding element actuator 414. Figure 4A The FOV scan, performed by rotating the optical path folding element, is not instantaneous but requires a settling time, which may be 1 to 50 milliseconds (ms) for a scan of 2 to 5 degrees and 5 to 500 ms for a scan of 10 to 45 degrees. After the settling time, the STC can be used to capture telephoto images. The STC can be focused on an object by a user command or autonomously. The scan direction of the STC can be determined by an initial (or target) viewpoint vector P. I Provided.

[0071] In step 304, the IMU is read out and provides rotational motion about the pitch, yaw, and roll directions, i.e., X, X, and X respectively. IMU Y IMU and Z IMU Typically, the IMU provides data on angular acceleration, which is integrated to determine the rotation angle. IMU data can be used to calculate the unwanted rotational motion of the device.

[0072] In step 306, a coordinate transformation is performed. This coordinate transformation is necessary because the viewpoint change of the STC caused by an unwanted rotational movement of the device and the sensing of the unwanted rotational movement occur in different coordinate systems.

[0073] A processing unit, such as an AP or MCU, can be configured to perform the coordinate transformation (e.g., AP 440 of device 400 or device 480, or...). Figure 4A (MCU 470 of device 400). In some examples, the AP or MCU can solve the following equation analytically, or the AP or MCU can approximate the solution using a polynomial fit or a linear fit. In other examples, the AP or MCU may not perform the computation but use a look-up table (LUT) for coordinate transformation. In some examples, for example, Figure 4A As shown, the coordinate transformation can be performed by an MCU (e.g., MCU 470) connected to the STC module 410.

[0074] In some examples, the transformation can be performed to express the coordinates of the STC in the IMU's coordinate system. Then, device rotation and compensated motion can be calculated in the IMU's coordinate system.

[0075] In some examples, such as Figure 3B The 2D graph 320 shown can be used to represent the coordinates of the STC in the IMU's coordinate system. Graph 320 may resemble a calibration graph used to calibrate the STC or to calibrate a dual-camera system, such as a telephoto camera and a wide-angle camera. The STC aperture 322 can be located at (0,0,0). The handheld device can be pointed at graph 320 in the "lateral" direction, i.e., for reference. Figure 3B The coordinate system, such as Figure 1C The long side of the smartphone shown can be parallel to the X-axis and the short side parallel to the Y-axis, with the STC aperture pointing towards the graph in the Z-direction. All viewpoints reachable by the STC are given by the "viewpoint vector" or "camera pointing vector" P, which points to coordinates on graph 320. The coordinates of the zero-state position can be (0, 0, EFL). T ), of which EFL T It is the EFL of the STC. At the zero position, the coordinates of the IMU (and the handheld device) overlap with the coordinates of the STC.

[0076] If the STC is pointed to a non-zero viewpoint, a coordinate transformation from IMU coordinates to STC coordinates must be performed. In some cases, the Rodriguez rotation formula can be used. The IMU's pitch / yaw / roll rotation values ​​can be named "hnd_pitch", "hnd_yaw", and "hnd_roll". The IMU provides hnd_pitch, hnd_yaw, and hnd_roll in a one-coordinate system with the following unit vectors:

[0077] -Pitch unit vector R P :R P =(1,0,0),

[0078] -Yaw unit vector R Y :R Y = (0,1,0),

[0079] - Roll unit vector R R :R R = (0,0,1).

[0080] Typically, OIS only corrects for small angles. Therefore, in some cases and approximately, the pitch / yaw / roll rotations can be handled independently. For any (slight) rotation of the device, the Rodrigues rotation formula can be applied individually to the pitch / yaw / roll rotations, which can be represented by the sum of the pitch / yaw / roll rotations. This can be achieved by using only hnd_pitch, or only hnd_yaw, or only hnd_roll (in IMU coordinates R...). P R Y and R R ) Applied to any initial viewpoint vector P I One hand movement may result in the following final viewpoint vector P F ("cross(x,y)" represents the cross product of vectors x and y, and "dot(x,y)" represents the dot product of vectors x and y):

[0081] Around R P The viewpoint vector P after rotating hnd_pitch F P (hnd_yaw,hnd_roll=0): P F P =(P I ·cos(hnd_pitch)+cross(P I ,R P sin(hnd_pitch) + R P ·(dot(P I ,R P)·(1-cos(hnd_pitch))));

[0082] Around R Y The viewpoint vector P after rotating hnd_yaw F Y (hnd_pitch,hnd_roll=0): P F Y =(P I ·cos(hnd_yaw)+cross(P I ,R Y sin(hnd_yaw)+R Y ·(dot(P I ,R Y )·(1-cos(hnd_yaw))));

[0083] Around R R The viewpoint vector P after rotating hnd_roll F R (hnd_pitch,hnd_yaw=0): P F R =(P I ·cos(hnd_roll)+cross(P I ,R R sin(hnd_roll) + R R ·(dot(P I ,R R )·(1-cos(hnd_roll)))).

[0084] For small angles, the final viewpoint vector (before normalization) P after pitch, yaw, and roll rotations is... F It can be given by the following formula:

[0085] P F '=P I +(P I -P F P )+(P I -P F Y )+(P I -P F R )

[0086] Normalization can be performed to ensure that the final viewpoint vector P F Located on chart 320. In some examples, P F You can use EFL T / P F 'z to PF 'Normalization is performed to obtain, where P F 'z is P F The z-part, i.e.:

[0087] P F =P F '·EFL T / P F ' z .

[0088] It is clear from the equations above that, in order to compensate for unwanted hand rotational motion in STC, compared to a non-scanning camera, such as a wide-angle camera 204, where only unwanted hand rotational motion around yaw and pitch can be compensated, it is necessary to compensate for hand rotational motion around yaw, pitch, and roll.

[0089] In other examples of coordinate transformation, the transformation can be performed to express the IMU's coordinates in the STC's coordinate system. The hand rotation and compensation motion can then be calculated in the STC's coordinate system. As described above, the Rodriguez rotation formula can be used.

[0090] In other examples of coordinate transformation, the transformation can be to a third coordinate system (“reference system”). The coordinates of both the STC and IMU are represented in this reference coordinate system. The hand rotation and compensation motion can then be calculated in this reference coordinate system. As described above, the Rodriguez rotation formula can be used.

[0091] In step 308, the movement of the OIS can be performed. In some examples, the OIS can be performed by moving the optical path folding element of the STC. In other examples, a lens such as lens 102 and / or an image sensor such as image sensor 106 can be moved for the OIS. Assuming an ideal OIS, the movement of the optical path folding element and / or the lens and / or the sensor may result in a POC vector correction P. OIS Completely cancels out the influence of hand movement on the viewpoint vector, i.e.: P F +P OIS =

[0092] P I Therefore, after executing step 308, STC points to P again. I In other examples, the entire STC can be moved for the OIS. That is, the optical path folding element, lens, and image sensor move together as a unit of the OIS.

[0093] In some embodiments, steps 304 to 308 can be repeated to continuously stabilize the STC. The OIS loop including steps 304 to 308 can be executed at a frequency of, for example, 500 Hz to 100 kHz. During the execution of the OIS method described above, an STC image or image stream is captured.

[0094] In some embodiments, the IMU can be fixedly attached to the optical path folding element such that when the optical path folding element is moved, the IMU also moves. This allows the use of multiple coordinate systems with multiple identical basic vectors for both the STC and the IMU, thus eliminating the need for the coordinate transformation in step 306.

[0095] In some embodiments, a sensor actuator may activate the image sensor to correct viewpoint aberration in an STC image. As described in commonly owned international patent application PCT / IB2021 / 056311, an STC image experiences viewpoint aberration. An aberration is a rotation of the STC image on the image sensor (“rotational viewpoint aberration”). A viewpoint aberration is introduced when an undesired rotational hand movement is compensated for by moving an optical path folding element as disclosed herein. A sensor actuator may be used to rotate an image sensor about a normal to compensate for rotational viewpoint aberration.

[0096] Figure 4A An embodiment of a handheld device, designated 400, including a multi-aperture camera with at least one STC disclosed herein, is schematically illustrated. Device 400 includes an STC module 410 comprising an optical path folding element 412 and an optical path folding element actuator 414 for FOV scanning and / or OIS, and a telephoto lens module 420 forming a telephoto image recorded by an image sensor 416. A telephoto lens actuator 422 can move the lens module 420 for focusing and / or OIS. The handheld device 400 may also include an application processor (AP) 440, which includes an FOV scanner 442, an OIS controller 444, an image generator 446, and an object tracker 448.

[0097] In other examples, device 400 may include an STC comprising two optical path folding elements and an optical path folding element actuator for each of the two optical path folding elements. In some examples, the optical path folding element actuator may actuate the optical path folding elements to perform OIS as disclosed herein. In other examples, a lens actuator may actuate a lens, or a sensor actuator may actuate a sensor for performing OIS as disclosed herein. For example, in

[0098] PCT / IB2021 / 054186 describes an STC camera based on two optical path folding elements. In such an STC, the optical path within the camera is folded twice, thus it can be described as a double-folding scanning telescope.

[0099] The handheld device 400 also includes a wide-angle (or ultra-wide-angle) camera module 430, which includes a second lens module 434 that forms an image recorded by a second image sensor 432. A second lens actuator 436 can move the lens module 434 for focusing and / or OIS. In some examples, the STC can scan the entire FOV. W Or a larger FOV. In other examples, STC can scan smaller than FOV. W One FOV.

[0100] In some examples, the object tracker 448 can be configured to track the field of view (FOV). W An object is tracked, and tracking data is provided to the FOV scanner 442 and / or the OIS controller 444. Based on the tracking data, the FOV scanner 442 and / or the OIS controller 444 can provide multiple control signals to the optical path folding element actuator 414, which initiates the rotation of an optical path folding element to track an object using STC. As an example, an object can be tracked so that it is within the FOV range. T The center is the center. Examples 3-7 described below relate to this tracking scenario, where the wide-angle camera image data is used to provide tracking information that triggers a telescope FOV scan.

[0101] In some cases, tracking information and OIS information may interfere with each other, and coordination between tracking and OIS may be required to achieve a desired object tracking and / or OIS result.

[0102] As a third exemplary method of OIS, consider a device such as apparatus 400 or 480, which includes a wide-angle camera without OIS and an STC. The STC can track a stationary object such that the center of the object is located at the field of view (FOV). T The tracking may occur in real-time (RT), meaning we assume there is no delay between the detection and compensation of tracking deviations. A rotational motion of a device caused by a user's hand movement will be detected as object movement in the wide-angle camera. In response, a tracking motion of the STC will be triggered, and the object's position in the telescope's field of view (FOV) will be updated. In summary, in an RT scenario, when the object tracker performs OIS, the object will always be within the FOV. T The center of gravity is unaffected by the user's hand movements.

[0103] As a fourth exemplary method of OIS, consider a device such as device 400 or 480, which includes a wide-angle camera without OIS and an STC with OIS. As in Example 3, we assume that in FOV W RT object tracking is performed so that the center of a (non-moving) object is located in the FOV. T The center. OIS can also be performed in RT. The rotational motion of a device caused by a user's hand movement will be detected as object movement in the wide-angle camera. In response, a tracking motion ΔT of the STC will be triggered. At the same time, the rotational motion of the device will also be detected by the OIS of the STC, and an OIS movement ΔOIS of the STC will be triggered in response. The OIS movement can be performed according to the OIS method disclosed herein. ΔT and ΔOIS are the same in direction and magnitude, i.e., 2·ΔT =

[0104] 2. The STC motion of ΔOIS will be triggered, which is (i) keeping the object in FOV. T The required movement is twice that of (i) the center (desired tracking result) and (ii) suppressing the influence of hand movements on the STC image (desired OIS result). In summary, neither telescopic tracking nor telescopic OIS achieved the desired results. Therefore, in some cases, OIS of the STC is disabled when using object tracking.

[0105] As a fifth exemplary method of OIS, consider a device such as device 400 or 480, which includes a wide-angle camera without OIS and an STC with OIS. Object tracking can be performed within the field of view (FOV). W The above is executed so that the center of a (non-moving) object is located in the FOV. T The center of the process. However, object tracking and OIS may not be possible to perform in RT. Typically, OIS is performed at higher frequencies than object tracking. As an example, OIS may be performed at 500 Hz to 100 kHz, while object tracking may be performed at 1 Hz to 100 Hz. In some cases, to prevent unwanted interference between OIS and object tracking, OIS may be disabled when using object tracking. In other embodiments, OIS and object tracking may be controlled separately in the frequency domain. As an example, for rotational motion of the device caused by a user's hand movements (occurring at frequencies higher than, for example, 30 Hz), OIS can be used for device motion correction. For frequencies below, for example, 30 Hz, OIS should not be used for device motion correction. Instead, the low-frequency device motion will be compensated by the object tracker.

[0106] As a sixth exemplary method of OIS, consider a device such as device 400 or 480, which includes a wide-angle camera with OIS and an STC without OIS. Object tracking can be performed within the FOV. W The above is executed so that the center of a (non-moving) object is located in the FOV. T The center. Object tracking and OIS can be performed in RT. For the OIS of the wide-angle camera, rotational motion of the device caused by a user's hand movement will not affect the wide-angle image stream. Because the object is in the FOV W There is no movement, therefore STC tracking will not be triggered. In summary, there is no hand motion compensation, and the object will no longer be within the FOV. T The center of gravity leads to an undesirable object tracking result. In some examples of preventing this undesirable result, the OIS of the wide-angle camera can be disabled during object tracking. In other examples, the object tracking control signals provided to the STC may also include multiple OIS control signals from the wide-angle camera. By superimposing the two signals, the benefits of both wide-angle camera OIS and proper STC tracking can be enjoyed simultaneously.

[0107] As a seventh exemplary method of OIS, consider a device such as device 400 or 480, which has the wide-angle camera and the STC with OIS. We assume RT tracking, so the center of an object is located at FOV. T The rotational motion of the device caused by a user's hand movement will be corrected by an OIS motion in the wide-angle camera and the STC in the RT. In summary, the user's hand movement will not affect the desired output of the object tracker.

[0108] Calibration data can be stored in a first memory 424, such as an electrically erasable programmable read-only memory (EEPROM), and / or in a second memory 438, and / or in a third memory 450 such as non-volatile memory (NVM). The calibration data may include calibration data between the wide-angle camera 430 and the STC 410. The calibration data may also include calibration data between the position of an optical path folding element and the corresponding viewpoint of the STC.

[0109] The handheld device 400 also includes an inertial measurement unit (IMU; for example, a gyroscope) 460, which provides motion information for 400. For example, a microcontroller unit (MCU) may be used to read and process data from the IMU 460. In some examples, the MCU may be controlled by an OIS controller 444, which is part of the AP 440. In some examples, steps 304 and 306 may be performed by the MCU, and step 308 may be performed by an optical path folding element actuator 414 (and / or a lens actuator 436 and / or a sensor actuator 418, where OIS is performed by lens shifting or sensor shifting, respectively). In some examples, the MCU 470 may be integrated into the AP 440.

[0110] Another embodiment of a handheld device, numbered 480 and including a multi-aperture camera with at least one STC as disclosed herein, is, for example... Figure 4B As shown. An MCU (not shown) for reading and processing motion data from the IMU 460 and for providing multiple OIS control signals can be included in the STC module 410, for example, in the driver of the optical path folding element actuator 414.

[0111] In some cases, additional data may be used for hand motion estimation. This additional data could be, for example, image data from the wide-angle camera 430, or data from other sensing units present in the handheld device.

[0112] In some examples, image data from wide-angle camera 430 can be used to estimate "optical flow" from multiple images as known in the art, where OIS controller 444 can use the optical flow data along with data from IMU 460 to estimate the motion of device 400. In other examples, optical flow data estimated solely from image data from camera 410 and / or camera 430 can be used to estimate the motion of device 400.

[0113] Image generator 446 can be configured to generate images and image streams. In some examples, image generator 446 can be configured to use only the first image data from camera 430. In other examples, image generator 446 can use image data from camera 410 and / or camera 430.

[0114] Although this disclosure has been described with reference to certain embodiments and generally associated methods, changes and substitutions to the embodiments and methods will be apparent to those skilled in the art. This disclosure is to be understood not to be limited to the specific embodiments described herein, but only to the scope of the appended claims.

[0115] Unless otherwise stated, the word “and / or” is used between the last two options in the list of options to indicate that it is appropriate to select one or more of the listed options and that a selection can be made.

[0116] It should be understood that when an element is referred to as “a” or “an” in the claims or specification, such reference should not be construed as referring to only one of that element.

[0117] Furthermore, for clarity, the term "substantially" is used herein to imply the possibility that a value may vary within an acceptable range. By one example, the term "substantially" as used herein should be interpreted as implying a possible variation of up to 5% above or below any specified value. By another example, the term "substantially" as used herein should be interpreted as implying a possible variation of up to 2.5% above or below any specified value. By yet another example, the term "substantially" as used herein should be interpreted as implying a possible variation of up to 1% above or below any specified value.

[0118] All patents and patent applications mentioned in this specification are incorporated herein by reference in their entirety, as if each individual reference were specifically and individually designated as incorporated herein by reference. Furthermore, any reference or identification herein should not be construed as an admission that such reference is prior art to the invention.

Claims

1. A device for optical image stabilization, characterized in that, include: A scanning telescope folding camera, the scanning telescope folding camera comprising: a first optical path folding element for folding light from a first optical path toward a second optical path, the first optical path forming an angle of less than 90 degrees with a normal of the device, the second optical path being orthogonal to the normal of the device; a lens having a lens optical axis along the second optical path; and an image sensor, wherein the device is a handheld electronic device. A first optical path folding element actuator is configured to tilt the first optical path folding element in one or more directions to point a viewpoint of the scanning telescope folding camera to a portion of a scene, wherein the tilting of the first optical path folding element introduces a rotating viewpoint aberration corresponding to a misalignment between a coordinate system of the scanning telescope folding camera and a coordinate system of the device. A motion sensor for sensing an undesired rotational motion of the device; A second actuator, configured to move at least one component of the scanning telescope folding camera to compensate for the undesired rotational movement of the device, wherein the compensation depends on the undesired rotational movement of the device and on the viewpoint of the scanning telescope folding camera; and A wide-angle camera, wherein the wide-angle camera has a field of view (FOV). W The field of view (FOV) of the scanning telescope folding camera is greater than that of the camera. T The wide-angle camera provides image data, which is used to track the field of view (FOV). W An object in the process, wherein information obtained from the tracking is used to point the viewpoint of the scanning telescope folding camera at the tracked object, so as to perform object tracking using the scanning telescope folding camera.

2. The apparatus according to claim 1, characterized in that: The movement of at least one component of the scanning telescope folding camera to compensate for the undesired rotational movement of the device is disabled during object tracking using the scanning telescope folding camera.

3. The apparatus according to claim 1, characterized in that: The wide-angle camera includes a wide-angle camera component that is moved to compensate for the undesired rotational movement of the device, wherein the movement of the wide-angle camera component is disabled during object tracking using the scanning telescope folding camera.

4. The apparatus according to claim 1, characterized in that: The movement of at least one component of the scanning telescope folding camera to compensate for the undesired rotational motion of the device is performed in a frequency range different from the frequency range used for object tracking with the scanning telescope folding camera.

5. The apparatus according to claim 4, characterized in that: The frequency range for object tracking is less than 30 Hz, and the frequency range for compensating for the undesired rotational motion of the device is greater than 30 Hz.

6. The apparatus according to claim 1, characterized in that: The scanning telescope folding camera is a double folding telescope camera, which includes a second optical path folding element in addition to the first optical path folding element.

7. The apparatus according to claim 1, characterized in that: The undesired rotational motion occurs around the normal of the device.

8. The apparatus according to claim 1, characterized in that: The sensing of the undesired rotational motion includes sensing the undesired rotational motion in three directions.

9. The apparatus according to claim 1, characterized in that: The compensation for the undesired rotational motion includes compensating for the undesired rotational motion in three directions.

10. The apparatus according to claim 1, characterized in that: The first optical path folding element actuator is used to move the first optical path folding element to compensate for the undesired rotational movement of the device.

11. The apparatus according to claim 1, characterized in that: The second actuator includes a lens actuator that moves the lens.

12. The apparatus according to claim 1, characterized in that: The second actuator includes a sensor actuator that moves the sensor.

13. The apparatus according to claim 1, characterized in that: The handheld electronic device is a smartphone.

14. A device for optical image stabilization, characterized in that, include: A scanning telescope folding camera, the scanning telescope folding camera comprising: an optical path folding element for folding light from a first optical path toward a second optical path, the first optical path forming an angle of less than 90 degrees with a normal of the device, the second optical path being orthogonal to the normal of the device; a lens having a lens optical axis along the second optical path; and an image sensor, wherein the device is a handheld electronic device. An optical path folding element actuator is provided for tilting the optical path folding element in one or more directions to point a viewpoint of the scanning telescope folding camera towards a portion of a scene, wherein the tilting of the optical path folding element introduces a rotational viewpoint aberration corresponding to a misalignment between a coordinate system of the scanning telescope folding camera and a coordinate system of the device; and A sensor actuator, wherein the sensor actuator rotates the image sensor about a normal of the image sensor to compensate for the viewpoint aberration of the rotation.

15. The apparatus according to claim 14, characterized in that: The device further includes: a motion sensor for sensing an unintended rotational movement of the device; and at least one actuator for moving at least one component of the scanning telescope folding camera to compensate for the unintended rotational movement of the device, wherein the compensation depends on the unintended rotational movement of the device and on the viewpoint of the scanning telescope folding camera.

16. A method for optical image stabilization, characterized in that, include: A device is provided including a scanning telescope folding camera, the scanning telescope folding camera comprising: an optical path folding element for folding light from a first optical path toward a second optical path, the first optical path forming an angle of less than 90 degrees with a normal of the device, the second optical path being orthogonal to the normal of the device; a lens having a lens optical axis along the second optical path; and an image sensor, wherein the device is a handheld electronic device; An optical path folding element actuator is provided, the actuator being used to tilt the optical path folding element in one or more directions to point a viewpoint of a scanning telescope folding camera towards a portion of a scene, wherein the tilting of the optical path folding element introduces a rotational viewpoint aberration corresponding to a misalignment between a coordinate system of the scanning telescope folding camera and a coordinate system of the device; and A sensor actuator is provided for rotating the image sensor about a normal to compensate for viewpoint aberrations of the rotation.

17. The method according to claim 16, characterized in that: The method further includes: providing a motion sensor for sensing an undesired rotational movement of the device; and providing at least one actuator for moving at least one component of the scanning telescope folding camera to compensate for the undesired rotational movement of the device, wherein the compensation depends on the undesired rotational movement of the device and on the viewpoint of the scanning telescope folding camera.

Images