Method for correcting viewpoint aberration in scan fold camera and multi-camera comprising such scan fold camera

Abstract

A method for correcting viewpoint aberration in a scan fold camera and a multi-camera including such a scan fold camera is disclosed. In a telefold camera, the telefold camera includes an optical path folding element (OPFE) inclined in one or both directions to align the telefold camera toward a viewpoint of a scene, from which a teleimage or a stream of teleimages is captured, and an image sensor, the telefold camera including an optical path folding element inclined in one or both directions to align the telefold camera toward the viewpoint of the scene, the remote image having a plurality of viewpoint aberrations and the plurality of viewpoint aberrations are digitally corrected to obtain an aberration-corrected image or a stream of a plurality of images.

Description

[0001] This application is a divisional application of application number 202180003962.0 (PCT application number is PCT / IB2021 / 056311), application date July 13, 2021, and invention name “Viewpoint aberration correction of scanning folding camera”.

[0002] Related Applications

[0003] This application is related to and claims priority to U.S. Provisional Patent Application No. 63 / 051,993, filed on July 15, 2020, the entirety of which is incorporated herein by reference. Technical Field

[0004] Embodiments disclosed herein relate generally to digital cameras and, more particularly, to correction of images obtained using folded digital cameras. Background Art

[0005] Compact digital cameras having multiple folded optical elements, also referred to as "folded cameras" or "folded camera modules", are known, see, for example, commonly owned International Patent Application No. PCT / IB2016 / 057366. Figure 1A A folded telephoto camera disclosed therein and numbered 100 is schematically shown from a first perspective. Figure 1B The camera 100 is shown from a second perspective. The camera 100 includes a lens 102 having a lens optical axis 110, an optical path folding element (OPFE) 104, and an image sensor 106. The optical path folding element 104 folds a first optical path that defines a point of view (POV) 108 of the camera 100 and is substantially parallel to the X-axis from an object, scene, or panoramic view portion 114 into a second optical path along the axis 110 that is substantially parallel to the Z-axis. The image sensor 106 has a plane normal aligned (parallel) to the axis 110, and outputs an output image that can be processed by an image signal processor (ISP—not shown). In some embodiments, the image signal processor may be part of the image sensor 106.

[0006] The camera 100 is designed to rotate the optical path folding element 104 relative to the image sensor about an axis 110 (Z axis), as indicated by arrow 112. That is, the camera 100 is a "scanning" telephoto camera ("STC"). The optical path folding element 104 can be rotated over a range of angles as required by various optical requirements (see below), in some cases up to 180 degrees, and in other cases up to 360 degrees. The camera 100 can use its "native" telephoto field of view (N-FOV T ) scans the scene, so it can effectively cover a larger area than the inherent long-range field of view (N-FOV T ) of the scene, which we call the scanning long-range field of view (S-FOV T ). Scanning long-range field of view (S-FOV) T ) is a field of view that includes all scene segments that can be captured by a scanning telephoto camera in multiple scanning telephoto camera images. To scan a two-dimensional scene, the optical path folding element 104 must rotate about two rotation axes. For example, the intrinsic telescopic field of view (N-FOV T )=10-20 degrees, scanning long-range field of view (S-FOV T )=30-80 degrees.

[0007] Figure 1C The optical path folding element 104 is shown after being rotated 30 degrees. Figure 1D The light path folding element 104 is shown after being rotated 180 degrees from a zero position. The rotational positions of 30 degrees and 180 degrees are examples of a range of many rotational positions.

[0008] Multiple images are acquired from certain viewpoints (FOV) of the camera. The viewpoints are determined by the camera aperture position as the starting point and the intrinsic long-range field of view (N-FOV T ) is the direction defined by the unit vector of the vector with the object point at the center as the end point. For example, in the spherical coordinates defined according to the ISO convention In the example, the camera’s viewpoint at r = 0 is given by Definition, polar angle θ and azimuth angle Defined in CN-FOV T The location of the object point. Figure 1A and 1B , the optical path folding element is in a zero rotational position ("zero position"). When the optical path folding element is in the zero position, the image acquired by the sensor (i.e., "generated" by the camera 100) has no viewpoint aberrations. In spherical coordinates as defined above, the zero rotational position is given by (1, 0, 0). When the viewpoint changes, the image acquired by the sensor is subject to viewpoint aberrations. Specifically, the image can be tilted (stretched into a trapezoid) and / or rotated and / or scaled, see e.g. Figure 2Aand 2B .

[0009] Whatever the viewpoint, it is desirable and advantageous to have a scanning telephoto camera image without viewpoint aberrations. Summary of the invention

[0010] The proposed method uses a digital algorithm to correct the viewpoint aberration taking into account the position of the optical path folding element to obtain an image without viewpoint aberration. After acquiring (capturing) the image and correcting it, it is proposed to crop a rectangular area from the corrected image to display the cropped rectangular image on the screen or save the cropped rectangular image to a file. For each optical path folding element position, a pre-calculated geometric transformation (i.e., homography) is applied to the acquired image to produce an image corrected for viewpoint aberration.

[0011] Depending on the position of the optical path folding element after correcting for viewpoint aberrations, the center of the original (uncorrected) image will not coincide with the center of the corrected image. For example, there can be five different cropping options (A, B, C, D, E), see Figure 3D .

[0012] The result of the cropping is a rectangular image with the same aspect ratio AR as the zero position (i.e. height / width = 3 / 4), but with a smaller image area than the zero position image area. The size of the image area depends on the position of the optical path folding element. The corrected and cropped image will be scaled to fit the display size or the saved image size.

[0013] All images can be further cropped so that all light path folding element positions have the same crop size (image area). The maximum crop size that fits all light path folding element positions can be calculated as the minimum size in the set of maximum sizes for each light path folding element position.

[0014] In various embodiments, methods are provided, including: providing a telescopic folded camera, the telescopic folded camera including an optical path folding element (OPFE) and an image sensor; tilting the optical path folding element in one or more directions to aim the telescopic folded camera toward a point of view (POV); capturing a telescopic image or a stream of multiple telescopic images from the point of view, the telescopic images having a viewpoint aberration; and digitally correcting the viewpoint aberration.

[0015] In some embodiments, the viewpoint may have multiple aberrations, and corrections for one, some, or all of the multiple aberrations applied above and below.

[0016] In some embodiments, the step of correcting the viewpoint aberration includes applying a geometric transformation to the captured telephoto image of the aberration to obtain a corresponding aberration-corrected image. In some exemplary embodiments, the geometric transformation uses calibration information captured in a camera calibration procedure.

[0017] In some embodiments, the method further comprises: cropping the aberration-corrected image to obtain an aberration-corrected cropped image, wherein the aberration-corrected cropped image has an aberration-corrected cropped image center, an aberration-corrected cropped image size, and an aberration-corrected cropped image width / height ratio.

[0018] In some embodiments, the method further comprises: scaling the aberration-corrected cropped image to obtain an aberration-corrected cropped and scaled output image, the aberration-corrected cropped and scaled output image having an output image center, an output image size, and an output image width / height ratio. In some embodiments, the steps of tilting the optical path folding element and capturing a telephoto image from the viewpoint are repeated to obtain a plurality of telephoto images captured at a plurality of viewpoints, and the output image center is selected so that the plurality of telephoto images captured for all possible viewpoints cover a maximum rectangular area in a scene. In some embodiments, the steps of tilting the optical path folding element and capturing a telephoto image from the viewpoint are repeated to obtain a plurality of telephoto images captured at a plurality of viewpoints, and the output image center is selected so that the plurality of telephoto images captured for a particular plurality of viewpoints cover a maximum rectangular area in a scene.

[0019] In various embodiments, a variety of systems are provided, including: a wide-angle camera having a wide-angle field of view FOV W ; A telephoto camera with a telescopic field of view FOV T <FOV W , the telephoto camera comprises an optical path folding element (OPFE) and an image sensor, the telephoto camera having a scanning capability, the scanning capability being tilted in one or more directions by the optical path folding element to align the telephoto folded camera toward a viewpoint of a scene, the telephoto image or a stream of multiple telephoto images having a viewpoint aberration, and a processor configured to digitally correct the viewpoint aberration.

[0020] In some embodiments, the viewpoint aberrations may be corrected using calibration data.

[0021] In some embodiments, the calibration data may be stored in a non-volatile memory.

[0022] In some embodiments, the calibration data includes: data of calibration between the tilt position of the optical path folding element in one or two directions and a plurality of corresponding viewpoints in a scene.

[0023] In some embodiments, the calibration data may include: data for calibration between a telephoto image and a wide-angle image.

[0024] In some embodiments, the calibration data includes: data for calibration between a tilt position of the optical path folding element in one or two directions and a telescopic field of view position within a wide-angle field of view.

[0025] In some embodiments, configuring the processor to digitally correct the viewpoint aberration may include applying a geometric transformation to a captured telephoto image or a stream of multiple telephoto images to obtain an aberration-corrected image.

[0026] In some embodiments, the geometric transformation may be a homography transformation.

[0027] In some embodiments, the geometric transformation may include: a motion-based computation of a homography using a stream of multiple frames from the wide-angle camera.

[0028] In some embodiments, the motion-based computation of the homography may further use inertial measurement unit information.

[0029] In some embodiments, the geometric transformation may be a non-affine transformation.

[0030] In some embodiments, the image sensor has an image sensor center, an active sensor width, and an active sensor height, and the output image center coincides with the image sensor center.

[0031] In some embodiments, the output image center may be selected so as to achieve a maximum possible rectangular cropped image size for a particular output image aspect / height ratio.

[0032] In some embodiments, the location of the output image center may be less than 10x the pixel size from an ideal output image center.

[0033] In some embodiments, the output image center may be located less than 10% of the width of the active sensor from an ideal output image center.

[0034] In some embodiments, the output image center may be located less than 10% of the height of the active sensor from an ideal output image center.

[0035] In some embodiments, the output image center may be selected so that the object image magnification M of an object across different viewpoints differs from a constant value by less than 10%.

[0036] In some embodiments, the output image center may be selected so that the output image covers a maximum area in a scene.

[0037] In some embodiments, the output image center may be selected so that the multiple telescopic images captured for all possible viewpoints cover a maximum rectangular area in a scene.

[0038] In some embodiments, the output image center may be selected so that the multiple telescopic images captured for a particular plurality of viewpoints cover a maximum rectangular area in a scene.

[0039] In some embodiments, the output image center may be selected so that the output image displays a region or object of interest in a visually appealing manner.

[0040] In various embodiments, methods are provided, including: providing a telescopic folded camera, the telescopic folded camera including an optical path folding element (OPFE) and an image sensor; tilting the optical path folding element in one or more directions to align the telescopic folded camera toward a plurality of viewpoints (POVs) of a calibration map, each viewpoint being associated with a corresponding optical path folding element position; capturing a corresponding telescopic image of the calibration map at each viewpoint, each telescopic image having a corresponding viewpoint aberration; analyzing data of the telescopic images to derive calibration data between each viewpoint having its corresponding viewpoint aberration and the corresponding optical path folding element position; and digitally correcting the viewpoint aberration using the calibration data.

[0041] In some embodiments, the calibration map may include a plurality of position identifiers that allow the viewpoint for a given position of the optical path folding element to be determined from the corresponding telephoto image.

[0042] In some embodiments, the calibration map may include a plurality of angle identifiers that allow the viewpoint aberration for a given position of the optical path folding element to be determined from each telephoto image.

[0043] In some embodiments, the calibration pattern may be a checkerboard pattern.

[0044] In some embodiments, the calibration data may be represented by a bidirectional function that converts any optical path folding element position to a distant viewpoint and / or its corresponding multiple viewpoint aberrations, and vice versa.

[0045] In some embodiments, the bidirectional function may be a polynomial.

[0046] In some embodiments, the calibration data may be represented by a bidirectional lookup table that converts any optical path folding element position to a distant viewpoint and / or its corresponding multiple viewpoint aberrations, and vice versa.

[0047] In some embodiments, the calibration data may be represented by a lookup table comprising a plurality of optical path folding element positions having a plurality of values ​​associated with a plurality of distant viewpoints and / or their corresponding plurality of viewpoint aberrations.

[0048] In some embodiments, the plurality of light path folding element positions may include more than five light path folding element positions, more than 50 light path folding element positions, or even more than 250 light path folding element positions.

[0049] In some embodiments, a method may further include: providing a wide-angle camera having a field of view FOV W A field of view FOV larger than the telescopic folded camera T .

[0050] In some embodiments, between the analysis of the telephoto image and the use of the calibration data, a method may further comprise: in a first additional step, positioning a telephoto viewpoint within a corresponding wide-angle image field of view at a corresponding optical path folding element position associated with the telephoto viewpoint to capture an additional telephoto image of the calibration map and capturing a wide-angle image of the calibration map, and in a second additional step, using the telephoto and wide-angle image data to derive calibration data between the corresponding optical path folding element position, the telephoto viewpoint within the corresponding wide-angle field of view, and the viewpoint aberration of the telephoto image relative to the wide-angle image. In some such embodiments, the first additional step and the second additional step may be performed simultaneously. In some such embodiments, all steps are performed by the same operator. In some such embodiments, the first to fourth steps may be performed by a first operator, and the first additional step and the second additional step may be performed by a second operator. In some such embodiments, the first to fourth steps may be performed within a time frame of less than 10 seconds, and the first additional step and the second additional step may be performed within a time frame of less than 10 seconds. In some such embodiments, the first and fourth steps may be performed in a time frame of less than 5 seconds, and the first additional step and the second additional step are performed in a time frame of less than 5 seconds. In some such embodiments, the first additional step does not include the capture of any additional images, and the analyzing and deriving of calibration data includes: receiving extrinsic calibration data between the telescopic folded camera and the wide angle camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Non-limiting examples of the embodiments disclosed herein are described below with reference to the drawings listed after this paragraph. The drawings and descriptions are intended to illustrate and clarify the embodiments disclosed herein and should not be considered to be limiting in any way. Similar elements in different drawings may be represented by similar reference numerals.

[0052] Figure 1A A known folded camera having an optical path folding element in a zero position is schematically shown from a perspective;

[0053] Figure 1B Schematically shown from another perspective Figure 1A Folding camera;

[0054] Figure 1C Schematically shows Figure 1A The camera has the optical path folding element in a first non-zero position;

[0055] Figure 1D Schematically shows Figure 1A The camera has the optical path folding element in a second non-zero position;

[0056] Figure 2A Different optical path folding element positions and corresponding fields of view in an object domain are shown;

[0057] Figure 2B shows acquired images, corrected images, and cropped images of corresponding multiple objects at three different optical path folding element positions;

[0058] Figure 2C A method for generating a remote output image as described herein is shown;

[0059] Figure 3A shows a field of view of the entire range of scanning positions of multiple optical path folding elements with an exemplary object;

[0060] Figure 3B Shows Figure 3A Details of the acquired images of the exemplary object described in;

[0061] Figure 3C Shows Figure 3B A detail of a corrected image of the acquired image;

[0062] Figure 3D Shows Figure 3C details of a cropped image of the corrected image;

[0063] Figure 4 An embodiment of an electronic device including a plurality of multi-aperture cameras including at least one scanning telephoto camera is schematically illustrated.

[0064] Figure 5 A method of a scanning telephoto camera calibration procedure described herein is shown;

[0065] Figure 6 Shows the available Figure 5 An exemplary calibration plot of the calibration method described in . DETAILED DESCRIPTION

[0066] Figure 2A Different optical path folding element positions in the “object domain” and their corresponding intrinsic long-range field of view (N-FOV T ). Figure 2B Shown in Figure 2A 1, 2, and 3. The acquired images, corrected images, and cropped images of a plurality of objects at three different (0, 1, and 2) optical path folding element positions shown in . The object domain is defined as the appearance of a scene captured by an ideal camera with a sufficiently large field of view and without any aberrations and distortions. That is, the object domain corresponds to the appearance of the scene as it would appear to a human observer. The object domain is different from the "image domain", which is defined as the appearance of a scene captured by a scanning telephoto camera (such as camera 100).

[0067] exist Figure 2A -B, 202-i (i = 0, 1, 2) represents multiple scanning long-range fields of view (S-FOV T ), 204-i represents multiple objects, 206-i represents multiple image frames, 208-i represents multiple images of objects 204-i, 210-i represents multiple image data boundaries, and 212-i represents rectangular clippings of multiple image frames.

[0068] Block 200 represents the minimum rectangular field of view that encompasses the scanning telescopic field of view, i.e., all image data for all viewpoints, achievable by a scanning telescopic camera in the object domain. The intrinsic telescopic fields of view for three different optical path folding element positions (0, 1, and 2) are represented by 202-0, 202-1, and 202-2. Each optical path folding element position corresponds to a different viewpoint. The intrinsic telescopic field of view for the optical path folding element "zero position" 202-0 is defined as the intrinsic telescopic field of view that produces an image of the object or scene without viewpoint aberrations, i.e., (except for the scaling factor and assuming no camera aberrations and distortions) at the zero position, the object in the object domain is identical to the image of the object in the image domain. As shown, the intrinsic telescopic field of view for any other position (e.g., 202-1 and 202-2) is not a horizontal rectangle (relative to 202-0), but an arbitrary quadrilateral. The same rectangular object is represented by 204-0, 204-1, and 204-2 in the intrinsic long-distance views 202-0, 202-1, and 202-2, respectively.

[0069] In one example, the optical path folding element is positioned at scanning position 1 ( Figure 2A ) at position 1, with an intrinsic telescopic field of view 202-1, which contains object 204-1 and represents the viewpoint of the scanning telescopic camera. Image frame 206-1 is captured (acquired) at position 1 ( Figure 2B In the captured image, the object 204-1 is represented by the captured object image 208-1 ( Figure 2B) is represented by frame 206-1. A geometric transformation is applied on frame 206-1 to obtain a corrected image frame 206'-1. The geometric transformation is related to the rotation angle of the optical path folding element at which the corresponding image is captured. Within the corrected image frame 206'-1, a corrected image 208'-1 of image 208-1 can be seen. 210'-1 marks the boundary of the image data present in the corrected image frame 206'-1, and 212-1 is a possible rectangular cropping containing an image segment of 210'-1. The aspect ratio AR (i.e., the ratio of horizontal width to vertical height) of the rectangular cropping 212-1 may be the same as the aspect ratio AR of the image sensor 106. In other examples, the rectangular cropping 212-1 may have an aspect ratio that is the same as the aspect ratio used to output the zero position image (e.g., 206"-0). The corrected image 206'-1 is then cropped to obtain a corrected cropped image 206"-1.

[0070] In the acquired image row Figure 2B, 206-0, 206-1, and 206-2 are raw image captures (frames) acquired at optical path folding element positions 0, 1, and 2, respectively. 208-0, 208-1, and 208-2 are captured images of objects 204-0, 204-1, and 204-2, respectively. In the corrected image row, 206-0, 206-1, and 206'-2 represent multiple corrected (also referred to as "rectified" or "aberration-corrected") image frames that have undergone image rectification. 208'-0, 208'-1, and 208'-2 represent captured object images 208-0, 208-1, and 208-2 of objects 204-0, 204-1, and 204-2 after image correction, i.e., they represent "rectified images" (or "rectified images") of the objects. The image data present in the corrected image 206'-1 has a boundary 210'-1, and the image data present in the corrected image 206'-2 has a boundary 210'-2. The dashed areas between 206'-1 and 210'-1 and between 206'-2 and 210'-2 have no valid image data to be displayed on the screen or saved to disk (i.e., the areas contain only empty pixels). 212-1 and 212-2 are possible rectangular clips that contain image segments of 210'-1 and 210'-2, respectively. 212-1 and 212-2 may have a specific aspect ratio. In the cropped image row, 206"-0, 206"-1, and 206"-2 are aberration-corrected cropped ("ACC") images, which contain image data of 206'-0, 206'-1, and 206'-2, i.e., they contain corrected image data of 206-0, 206-1, and 206-2. In some embodiments, after cropping, the images are scaled. In images 206"-0, 206"-1, and 206"-2, regions 208-0, 208-1, and 208"-2 represent the corrected, cropped, and scaled object 208-0. 04-0, 204-1 and 204-2. Images 206"-1 and 206"-2 are generated by cropping images 206'-1 and 206'-2 along boundaries 212-1 and 212-2, respectively. Image 206"-0 is not cropped, i.e., all image data of 206'-0 is also present in 206"-0. It should be noted that since position '0' is defined as having no viewpoint aberration, the correction algorithm will have no effect on the acquired image (i.e., 206-0, 206-0 and 206"-0 will be the same). In other examples, the sizes of 206-0, 206-0 and 206"-0 may not be equal, but the size of 206'-0 and / or 206"-0 may be different from the size of 206-0, which is different by some cropping coefficient. The same applies to object images 208-0, 208-0 and 208"-0.

[0071] Figure 2C An exemplary method for generating a telescopic output image disclosed herein is schematically illustrated. In a first step 252, a command triggered by a human user or a program directs the inherent telescopic field of view to a region of interest (ROI) within a scene by scanning. The scanning can be performed by rotating an optical path folding element. The field of view scanning by rotating the optical path folding element is not performed instantaneously, but requires some dwell time, for example, for scanning 2-5 degrees, it can be about 1-30 milliseconds, for scanning 10-25 degrees, it can be about 15-100 milliseconds. After the dwell time, a scanning telescopic camera image (e.g., Figure 2B In step 256, the scanning telephoto camera image is rectified. In the first rectification sub-step (referred to as the "geometric transformation sub-step"), a geometric transformation (e.g., homography, affine, or non-affine transformation) is performed, and the result is shown in FIG. Figure 2B As shown. In the following examples, "homography" is used to represent any meaningful geometric transformation. The homography corrects the aberrations associated with any particular viewpoint and is therefore a function of the viewpoint. A second correction sub-step ("interpolation sub-step") may be performed, which is described in detail below.

[0072] Thus, a corrected (or rectified, or aberration-corrected) image is obtained. The calibration data between the positions of the optical path folding elements and the corresponding viewpoints can be used to select a homography corresponding to a particular viewpoint. In some embodiments, the geometric transformation can include corrections known in the art, such as distortion correction and color correction. In step 258, as Figure 2B The cropped image is shown. In step 260, the cropped image is scaled. In step 262, the cropped and scaled output image is output. The output image can be displayed on an electronic device (e.g., device 400) ( Figure 4 ) and / or stored in a memory (e.g., memory 450) ( Figure 4 ) or any other memory of the device.

[0073] The clipping in step 258 may be performed according to different clipping selection criteria. Some clipping selection criteria may be directed to a specific size of the clipped image. Other clipping selection criteria may enable specific input image coordinates to be transferred to specific image coordinates of the clipped image. Hereinafter, the "clip selection" criterion may be referred to as the "clip criterion".

[0074] The cropping criteria for a specific size of cropped image can be as follows: In one criterion (cropping criterion 1), the image can be cropped so that the resulting image is a rectangular image. In another criterion (cropping criterion 2), the resulting image can be square. Here and in the following, image size and shape are defined by the number and distribution of image pixels, so that the size and shape do not depend on the actual mode in which the image is displayed. For example, a rectangular image has m rows (image height), where each row contains n values ​​(image width). A square image has m rows, each row has m values. If m is satisfied 1 xn 1 >m 2 xn 2 , then we have m 1 Row and n 1 The first rectangular image with a value greater than m 2 Row and n 2 The second rectangular image of the value.

[0075] In another standard (cropping standard 3), the image is cropped so as to obtain a maximum rectangular image with a specific aspect ratio for a specific viewpoint. An example of this standard is Figure 3D The cropping options "D" and "E" shown in . The aspect ratio refers to the width / height ratio of the image. The aspect ratio can be 4:3, 3:2, or 16:9. In another criterion (cropping criterion 4), the image captured at the first viewpoint is cropped so that the resulting image has the same aspect ratio and size as the image captured at the second viewpoint. The second viewpoint is the viewpoint that results in the smallest image, which is obtained by cropping the largest rectangular image with a specific aspect ratio for the second viewpoint. Cropping criterion 4 ensures that the cropped images at all possible viewpoints have the same aspect ratio and shape. In another criterion (cropping criterion 5), the images are cropped so that all output images generated from the scanning telephoto camera images captured from the entire scanning telephoto field of view in step 254 cover the largest area of ​​the rectangular field of view in the object domain. This cropping criterion ensures that the area of ​​the rectangular field of view in the object domain (such as 200) is maximally covered by the scanning telephoto field of view. In another criterion (Crop Criterion 6), the image is rectangularly cropped to obtain the same object-to-image magnification throughout the scanned telescopic field of view. Typically, the image of the object image captured in step 254 is smaller for larger viewpoints. In some embodiments, the "same magnification" condition may be met if the magnification obtained for all viewpoints varies <10% from a constant value. In other examples, the "same magnification" condition may be met if the magnification obtained for all viewpoints varies <5% or <15%.

[0076] The following describes the cropping criteria for mapping specific input image coordinates to specific image coordinates of the cropped image. In general, by applying specific cropping selection criteria, any arbitrary object image point of the image captured in step 254 ("input image") can be defined as the image center of the output image in step 262. In cropping criteria 7, the image can be rectangularly cropped so that the image center of the cropped image contains the same image data as the image data of the center of the input image for a specific viewpoint. The image center can be defined as the center pixel and the surrounding pixels within a radius of, for example, 10 times the pixel size. In some embodiments, the image center can be defined as the center pixel plus the surrounding pixels within a radius of, for example, 5 times or 30 times the pixel size.

[0077] In cropping criterion 8, the image may be cropped rectangularly so that the center of the cropped image contains the same image data as the image data at the center of the input image, and the cropped image additionally satisfies the following condition: any two images captured at any first and second viewpoints are cropped so that the resulting image has the same aspect ratio and size. In yet another example, cropping criterion 8 may additionally satisfy the condition that the cropped image has a maximum size (cropping criterion 9). In another example, the image may be cropped so that a region of interest (ROI) or object of interest (OOI) is displayed on the image output in a visually appealing manner in step 264 (cropping criterion 10). This criterion may support aesthetic image cropping, for example, as described in the article "Deep Network Solution for Attention and Aesthetic Perception Photo Cropping" by Wang et al. in the May 2018 IEEE Transactions on Pattern Analysis and Machine Intelligence. Applications of aesthetic image cropping are also described in co-owned PCT patent application No. PCT / IB2020-061330. In yet another example, the image may be cropped as required for further processing steps, for example, the image may be cropped so as to contain only a certain segment of the field of view in the object domain (cropping criterion 11). Possible further processing may be, for example, the generation of a super image, i.e., the generation of an output image composed of image data of multiple input images. The generation of super images is described in co-owned PCT patent application No. PCT / IB2021-054070. Another possible further processing may be the generation of a panoramic image as known in the art.

[0078] The scaling in step 260 may be performed according to different scaling selection criteria. In some embodiments, scaling may be performed so that the image captured at different viewpoints in step 254 and the image output in step 262 ("output image") have the same size and aspect ratio (scaling criterion 1). In other examples, scaling may be performed so that the pixel density of each object image region in the output image is the same as the pixel density of each region in the object domain present in the image captured in step 254 (scaling criterion 2). In yet other examples, scaling may be performed so that the image size is suitable for the requirements of a program that performs further processing on the image data (scaling criterion 3).

[0079] The above steps 252 - 262 may be performed sequentially, ie one after another.

[0080] In some scanning telephoto camera image correction embodiments, step 256 may be performed as follows: Assume (x 输入 i ,y 输入 j ) is the value of some arbitrary image coordinate (i, j) of the input image (captured in step 254), and assuming that (x 输出 m ,y 输出 n ) is the value of some arbitrary image coordinates (m, n) of the output image (the output image of step 256). In the geometric transformation sub-step, the homography transformation can be (x 输出 ,y 输出 ) = f H (x 输入 ,y 输入 ), where H is a 3x3 homography transformation matrix known in the art. The homography transformation can be performed using f H - 1 =f H-1 Inverted. Clipping transformation (x 输出 ,y 输出 ) = trim(x 输入 ,y 输入 ) can be (x 输出 m ,y 输出 n )=(x 输入 i -clip-start_x i ,y 输入 j -clip-start_y j) is used to assign each coordinate of the input image to a coordinate in the output image, where only coordinates with values ​​greater than 0 are used for the output image. The vector (crop_start_xi, crop_start_yj) defines the size and shape of the cropped image. Inverse crop transformation -1 By (x 输入 m ,y 输入 n )=(x 输出 i +cut-start_x i ,y 输出 j-clip-start_yj) definition. Scaling transformation (x 输出 ,y 输出 ) = zoom(x 输入 ,y 输入 ) can be (x 输出 ,y 输出 )=(s x ·x 输出 ,s y ·y 输入 ) with scaling factors s in the x and y directions respectively x and y . Inverse scaling transformation scaling -1 By (x 输入 ,y 输入 )=(s x -1 ·x 输出 ,s y -1 ·y 输出 ). The transformation function T is defined by applying homography, cropping and scaling in sequence, i.e., T is defined by (x 输出 ,y 输出 )=Scale(Crop(f H (x 输入 ,y 输入 )) and (x 输出 ,y 输出 )=T(x 输入 ,y 输入 ) to define.

[0081] In the interpolation sub-step, the output image (x 输出 ,y 输出 ). For example, the value (x 输出 m ,y 输出 n ). To do this, we compute the input image (x输入 ,y 输入 ) of the output image, which will contain the value (x) at that particular coordinate (m, n) used to calculate the output image. 输出 m ,y 输出 n ). The coordinates (m', n') in the input image can be obtained by applying the inverse transformation function T- 1 Applied to all output coordinates (m, n) to obtain, that is, T- 1 (x 输出 m,y 输出 n) or f H - 1 (Cut- 1 (Zoom- 1 (x 输出 m,y 输出 n)), for all (m, n). In general, T- 1 Instead of mapping each coordinate (m, n) to a coordinate (m', n'), each coordinate is mapped to a segment of adjacent coordinates (m', n'). 输出 m ,y 输出 n ), the entire segment or a partial segment of adjacent coordinates (m', n') can be considered. For the value (x) of the output image obtained at the coordinate (m, n) 输出 m ,y 输出 n ), in the first step, according to T(x 输出’ m ,y 输出’ n ) = resample(T 输入 , x 输入 m’ ,y 输入 n’ ) evaluates a resampling function R known in the art for all neighboring coordinates (m', n'). The resampling may be performed by methods known in the art, such as nearest neighbor, bilinear or bicubic.

[0082] At the value (x 输出 m ,y 输出 n ) is determined, the above steps can be performed to calculate the value (x) at the additional coordinate (o, p) 输出 o ,y 输出 p ) and so on. This is repeated until all values ​​of the output image (x输出 ,y 输出 ). In various embodiments, the calculations described above are performed on multiple output coordinates or even on all output coordinates in parallel. In some scanning telephoto camera image rectification embodiments, the calculations described herein may be performed by a central processing unit (CPU). In other scanning telephoto camera image rectification embodiments, for faster image processing, the calculations described herein may be performed by a graphics processing unit (GPU). Scanning telephoto camera image rectification may be performed in different color gamuts, such as RGB, YUV, YUV420, and other color gamuts known in the art.

[0083] Figure 3A The scanned long-range field of view (S-FOV) in the object domain is shown. T ) 300. 302 represents the inherent long-range field of view (N-FOV) corresponding to the position of a specific optical path folding element in 300 T 304 represents an object point in the scene whose image point is located at the center of the image sensor. 306 represents any selected specific object point in the scene that is included in the inherent long-range field of view 302.

[0084] Figure 3B An acquired telescopic image 310 having a center 304 ′ and an image point 306 ′ containing an object point 306 is shown. Figure 3C A corrected image 312 is shown which is generated by applying the geometric transformation to the image 310. 314 is the boundary of the image data in the corrected image, i.e., there is no available image data outside the boundary (dashed area). 304" and 306" are the positions of image points 304' and 306' in the corrected image, respectively.

[0085] Figure 3D shows the rectangular clipping Figure 3CCorrected image 312 is shown and exemplary different options for generating output images as described herein. It can be seen that the image data located at the center of the output image (OIC) depends on the selected cropping criteria. In the examples labeled "D" and "C", the cropping selection criteria are selected so that the specific image point 306' is located at the center of the output image. The cropping option "D" is selected to meet two criteria: (i) "the specific image point 306' is located at the center of the output image" and (ii) "the largest rectangular image for a given viewpoint is achieved". The cropping option "C" is selected to meet the criterion of "the specific image point 306' is located at the center of the output image". Other cropping selection criteria are described in the examples labeled "A", "B" and "E". The cropping option "A" is selected to meet the criterion "the image center 304' is located at the center of the output image". The cropping option "B" is selected to meet two criteria: (i) "the image center 306' is located at the center of the output image" and (ii) "the largest rectangular image for a given viewpoint is achieved". The cropping option "E" is selected so that the criterion of the output image "the largest rectangular image for a given viewpoint is achieved" is met. In other examples (not shown), the cropping options may be selected to satisfy two criteria: (i) "image center 304' is located at output image center" and (ii) "maximum rectangular image for all possible viewpoints is achieved". It may not always be possible or beneficial to position the output image center exactly at a particular image location ("ideal output image center"), such as the image center, but close to the ideal output image center. The closeness of the ideal output image center may be expressed as a percentage of the image sensor size (e.g., the output image center may be located less than 10% of the image sensor width from the ideal output image center) or as a distance in pixels (e.g., the output image center may be located less than 10 times (10x) the pixel size from the ideal output image center).

[0086] In other examples (not shown), the cropping options may be selected so as to satisfy the criterion of "largest rectangular image for all possible viewpoints". In another example (not shown), the cropping options may be selected so as to satisfy the criterion of "largest rectangular image for a certain number of viewpoints". The certain number of viewpoints may cover all possible viewpoints or a subset thereof.

[0087] Figure 4An embodiment of a mobile electronic device ("electronic device") numbered 400 and including a multi-aperture camera with at least one scanning telephoto camera is schematically shown. The electronic device 400 can be a smartphone, a tablet, a laptop, etc. The electronic device 400 includes a first scanning telephoto camera module 410, which includes an optical path folding element 412 for field of view scanning, and a telephoto lens module 418 to form a telephoto image recorded by a first image sensor 416. The image sensor 416 has an active sensor area defined by an active sensor width and an active sensor height that actually performs light collection, and an inactive area that does not perform light collection. A telephoto lens actuator 422 can move the lens module 418 for focusing and / or optical image stabilization (OIS). The electronic device 400 can also include an application processor (AP) 440, which includes a field of view scanner 442, a motion estimator 444, and an image generator 446. The effective focal length (EFL) of the scanning telephoto camera 410 is EFL=5 mm-50 mm. The sensor diagonal (SD) of the image sensor 416 may be SD=3 mm-15 mm.

[0088] The calibration data may be stored in a first memory 424, such as an electrically erasable programmable read-only memory (EEPROM), a second memory 438, or a third memory 450, such as a non-volatile memory (NVM). The calibration data may include scanning telephoto camera calibration data and dual camera calibration data. The electronic device 400 also includes a wide angle ("W") (or ultra wide angle, "UW") camera module 430 having a FOV W FOV UW >N-FOV T , which includes a second lens module 434 that forms an image recorded by the second image sensor 432. A second lens actuator 436 may move the lens module 434 for focusing and / or optical image stabilization.

[0089] In use, a processing unit such as an application processor 440 may receive corresponding first and second image data from camera modules 410 and 430 and provide camera control signals to camera modules 410 and 430. The field of view scanner 442 may receive commands from a human user or a program for directing the inherent telescopic field of view toward a specific viewpoint in a scene. In some embodiments, the command may include a single request to direct the inherent telescopic field of view toward one specific viewpoint. In other examples, the command may include a series of requests, such as for directing the inherent telescopic field of view toward multiple specific viewpoints in series. The field of view scanner 442 may be configured to calculate a scanning order based on the requested specific viewpoints. The field of view scanner 442 may be configured to provide a control signal to the optical path folding element actuator 414, which may rotate the optical path folding element 412 in response to the control signal to scan the inherent telescopic field of view. In some embodiments, the field of view scanner 442 may also provide a control signal to the optical path folding element actuator 414 to drive the optical path folding element 412 for optical image stabilization.

[0090] The electronic device 400 also includes an inertial measurement unit (IMU or "Gyro") 460 that can provide information about the movement of 400. The motion estimator 444 can use data from the inertial measurement unit 460, for example, to estimate hand movements caused by a human user. In some embodiments, the motion estimator 444 can use additional data. For example, image data from the camera 410 and / or from the camera 430 to estimate "optical flow" from multiple images as is known in the art. The motion estimator 444 can use data from the inertial measurement unit 460, and can also use the optical flow data to estimate the movement of 400 with greater accuracy. The motion information of 400 can be used for optical image stabilization or the above-mentioned homography transformation. In other embodiments, only the optical flow data estimated based on the image data of the camera 410 and / or the camera 430 can be used to estimate the movement of 400. The image generator 446 can be configured to generate images and image streams, such as Figure 2C In some embodiments, image generator 446 may be configured to use only the first image data from camera 430. In other embodiments, image generator 446 may use image data from camera 410 and / or camera 430.

[0091] Figure 5 The method of a scanning telephoto camera calibration procedure described herein is shown. The calibration procedure allows to derive calibration data for a single scanning telephoto camera ("STC calibration") or a dual-camera ("dual-camera, DC"), comprising a scanning telephoto camera and a wide-angle camera with FOV W >N-FOV T(“Dual Camera Calibration (DCcalibration)”). The goal of the calibration procedure is to connect three parameters for all possible viewpoints: a specific optical path folding element position defined by a position sensor value pair, a viewpoint associated with the specific optical path folding element position, and a viewpoint aberration associated with the specific viewpoint.

[0092] In a first example (calibration example 1 or "CE1"), a calibration procedure is provided for a scanning telephoto camera that performs two-dimensional scanning by rotating an optical path folding element along two axes, wherein the magnitude of the rotation is measured by two or more position sensors (e.g., Hall sensors), first and second position sensors P1 and P2, respectively. The viewpoint of the scanning telephoto camera is measured by a value pair p1 and p2 of P1 and P2, respectively. In a first step 502, a calibration chart ("CC") is provided. A suitable calibration chart comprises position identifiers (e.g., position identifiers 602 and 604, see Figure 6 ), which allows the position on the chart to be determined. When a suitable calibration chart is captured at a given distance using a scanning telephoto camera, at least one position identifier is present in the inherent telephoto field of view for all viewpoints in the scanning telephoto field of view. With the help of the position identifiers, the viewpoints of the scanning telephoto camera relative to the calibration chart can be determined. The position identifiers can be, for example, symbols encoding position information, distributed with a sufficiently high frequency on a chessboard. Additionally, a suitable calibration chart contains angle identifiers that allow the relative angular tilt and rotation between the calibration chart and the scanning telephoto camera image of the calibration chart to be determined. The angle identifiers can be, for example, lines in a chessboard. Figure 6 An example of a suitable calibration map is shown in . The size of the calibration map and the distance between the scanning telephoto camera and the calibration map will be chosen so that the entire scanning telephoto field of view is contained within the field of view covered by the calibration map.

[0093] In CE1, a list of N specific value pairs (p1, p2) may be defined for a specific scanning telephoto camera design. In some embodiments, the list may contain N=10 value pairs (p1, p2) 1 , …(p1, p2) 10 In other embodiments, the list may contain N=10 to 20 or more value pairs. According to a first criterion for value pair selection, the value pairs may be selected so that the scanning telephoto camera must capture a minimum number of different viewpoints during the calibration procedure (or, in other words, a minimum number of repetitions required for steps 504 and 506).

[0094] For the second example of dual camera calibration ("CE2"), another calibration pattern may be needed in step 502, where the calibration pattern may or may not be a checkerboard. The scanning telephoto camera of CE2 satisfies the same properties as CE1. Also in CE2, a list of N specific position sensor value pairs (p1, p2) may be defined for a specific scanning telephoto camera design, each value pair being associated with a specific optical path folding element position. In some embodiments, the list may contain N = 200 value pairs (p1, p2) 1 , …(p1, p2) 200 In other embodiments, the list may contain N=100 to 300 or more value pairs.

[0095] By tilting the optical path folding element to a specific optical path folding element position, in CE1 and CE2, for example, (p1, p2) 1 Definition: In step 504, a scanning telephoto camera is directed to an (unknown) viewpoint on a calibration map.

[0096] In step 506, one or more scanning telephoto camera images are captured. For dual camera calibration and CE2, a second sub-step of step 506 is required, where a scanning telephoto camera image is captured together with a wide angle image captured by the wide angle camera. Capturing a scanning telephoto camera image together with a wide angle image here means capturing the image at the same dual camera position and orientation. Typically, the capture may be done simultaneously, but this is not mandatory.

[0097] In some embodiments, capturing the scanning telephoto camera image along with the wide angle image may be performed together and in a single step, such as by the same operator.

[0098] In other examples, these two steps may be performed separately, such as by different operators. For example, to calibrate the scanning telephoto camera relative to a first calibration map, the first operator may capture one or more scanning telephoto camera images at a particular optical path folding element position. The scanning telephoto camera calibrated relative to the first calibration map may be included in a dual camera by a second operator, the dual camera being used to capture a second calibration map (which may be the same as the first calibration map or different from the first calibration map) at a particular optical path folding element position with the scanning telephoto camera, and one or more wide angle images for calibrating the scanning telephoto camera relative to the wide angle camera of the dual camera. Steps 504 and 506 are repeatedly performed according to the N value pairs so as to capture one or more scanning telephoto camera images at each of the N optical path folding element positions (or value pairs). The repetition of steps 502 and 504 for multiple optical path folding element positions may be performed, for example, within a predetermined time frame. The predetermined time frame may be, for example, 10 seconds or 5 seconds. For example, the first operator may be a camera module manufacturer that manufactures the scanning telephoto camera, and the second operator may be a phone manufacturer that includes the scanning telephoto camera in a dual camera and includes the dual camera in a mobile device. In some embodiments, the second sub-step of step 506 does not include capturing additional scanning telephoto camera and wide angle images, but rather includes receiving extrinsic calibration data between the scanning telephoto camera and the wide angle camera.

[0099] In step 508, the scanning telephoto camera image is analyzed. The purpose is to assign a viewpoint and a corresponding viewpoint disparity to a particular optical path folding element (or value pair) position of step 504. The analysis involves using the position identifiers of the calibration map that appear in the scanning telephoto camera image to determine the viewpoint at which it was captured, and using the angle identifiers of the calibration map and the ground truth image to determine the viewpoint disparity.

[0100] For CE1, the analysis consists of using the position identifiers of the calibration map present in the scanning telephoto camera image to determine the viewpoint at which it was captured, and using the angle identifiers of the calibration map and the ground truth image to determine the viewpoint disparity.

[0101] In the first sub-step of CE1, POV i Assigned to the value pair (p1, p2) i .

[0102] In the second sub-step of CE1, the scanning telephoto camera image is compared with the ground truth image of the calibration map at the corresponding viewpoint. In this comparison, it is determined which image transformation parameters transform the scanning telephoto camera image to the ground truth image of the calibration map. In some embodiments, three image transformation parameters can be used. For dual cameras and CE2, the viewpoint and the corresponding viewpoint disparity are determined by comparing the scanning telephoto camera image and the wide angle image captured in step 506.

[0103] In step 508 of CE1, for all pairs of values ​​(p1, p2) 1 , …(p1, p2) N Perform the first and second substeps to add to each pair of values ​​(p1, p2) i Assigning a specific POV i and image transformation parameters.

[0104] In step 510, calibration data is derived based on the analysis in step 508. In some embodiments, the calibration data is represented by a bidirectional data polynomial. In other examples, the calibration data is represented by a bidirectional lookup table (LUT) polynomial. In all examples, the scanning telephoto camera calibration data includes functions that can be used to convert any optical path folding element position to a viewpoint aberration of a scanning telephoto camera image relative to a chessboard and / or a scanning telephoto camera viewpoint. The dual camera calibration data can be used to convert any optical path folding element position to a viewpoint aberration of a scanning telephoto camera image relative to a wide angle camera and / or a wide angle field of view (FOV W ) is a scanned telephoto camera image viewpoint aberration for a scanned telephoto camera viewpoint within a wide angle field of view. Vice versa, any viewpoint aberration of a scanned telephoto camera image relative to a wide angle camera image can be converted to a scanned telephoto camera viewpoint and / or an optical path folding element position within a wide angle field of view (hence "bi-directional"). In yet another example, the scanned telephoto camera calibration data is represented by a bi-directional lookup table comprising a plurality of optical path folding element positions with associated values ​​of scanned telephoto camera image viewpoint aberrations relative to a calibration map and / or a scanned telephoto camera viewpoint. The dual camera calibration data is represented by a bi-directional lookup table comprising a multitude of optical path folding element positions with associated values ​​of rotation angles relative to the wide angle camera image and / or a scanned telephoto camera image of a telephoto viewpoint within a wide angle field of view. For CE1, a function is determined that converts all value pairs (p1, p2) 1 , …(p1, p2) N Rather than assigning a specific viewpoint, the viewpoint 1 , …, viewpoint Nand its assigned image transformation parameters approximate. This function is generalized, meaning that it is used to perform bidirectional conversions between all possible pairs of optical path folding element position values, their viewpoints, and image transformation parameters for image correction. According to a second criterion for value pair selection, the value pairs can be selected so that the generalization of the function leads to a minimum aggregated error ("AE"). The "accumulated error (AE)", which is to be minimized, refers here to an error function that depends on the deviations of the scanning telephoto camera images, which are viewpoint-corrected from their corresponding ground truth images for all possible viewpoints (or a number of viewpoints that is large enough to statistically approximate all possible viewpoints). In some embodiments, some compromise is made between satisfying the first or second criterion for value pair selection.

[0105] For CE2, the derived calibration data is contained in a bidirectional lookup table. The bidirectional lookup table contains N optical path folding element positions (value pairs), the viewpoint associated with each value pair, and its corresponding viewpoint aberration. This means that not all possible optical path folding element positions have clear calibration data. Therefore, for the scanning telephoto camera image corrected with CE2 at the optical path folding element position not included in the bidirectional lookup table, the viewpoint and its viewpoint aberration can be approximated. In some embodiments for approximation, a calibration value associated with an optical path folding element position can be used, wherein the optical path folding element position is the position from all N optical path folding element positions that is closest to the current optical path folding element position. The closest distance can be defined here by a distance metric known in the art, for example, the quadratic distance of the corresponding value pairs (sqrt((p1-p1 c ) 2 +(p2-p2 c ) 2 ) can be minimized, where p1 and p2 are the current positions of the optical path folding elements, p1 c 、p2 c is a value contained in a bidirectional lookup table. In other examples for approximation, a weighted average of multiple calibration values ​​associated with multiple optical path folding element positions that are the positions closest to the current optical path folding element position from among all N optical path folding element positions can be used.

[0106] In step 512, the calibration data is applied to the scanning telephoto camera images to correct for viewpoint aberrations.

[0107] Figure 6 Shows the available Figure 5An example calibration diagram 600 for the calibration method described in . The calibration diagram 600 includes 72 position identifiers, arranged in the form of 9 rows (oriented parallel to x) and 8 columns (oriented parallel to y). Exemplarily, the two first position identifiers of the first row are marked as 602 and 604, respectively. The position identifiers are located at a defined distance from a reference point of the calibration diagram, such as the upper left corner of the calibration diagram. In order to use the calibration diagram 600 for calibration at different camera calibration diagram distances, the size of the calibration diagram 600 can be adjusted so that there is at least one position identifier in the inherent telescopic field of view for all viewpoints in the scanned telescopic field of view. The calibration diagram 600 also includes additional angle identifiers, represented by the vertical and horizontal lines of the chessboard in this example.

[0108] Although the present disclosure has been described in terms of certain embodiments and generally associated methods, variations and combinations of the embodiments and methods will be apparent to those skilled in the art.The present invention is not to be limited by the specific embodiments described herein, but only by the scope of the appended claims.

[0109] Unless otherwise indicated, use of the expression "and / or" between the last two members of a list of options for selection indicates that selection of one or more of the listed options is appropriate and can be made.

[0110] It should be understood that where the claims or description refer to one or an element, the reference should not be interpreted as there being only one of the elements.

[0111] All references mentioned in this specification are incorporated by reference in their entirety into this specification, to the same extent as if each individual reference was specifically and individually indicated to be incorporated by reference into this specification. In addition, any reference cited or identified in this application shall not be construed as an admission that such reference is available as prior art to the present invention.

Claims

1. A method for correcting viewpoint aberrations in multiple cameras, characterized in that: The method comprises: providing a first image stream, wherein the first image stream has a first field of view FOV1; providing a second image stream, the second image stream having a second field of view FOV2, which satisfies FOV2<FOV1, wherein the first image stream includes a first image captured in the first field of view FOV1 and includes an object, the object having a first object-to-image magnification OTM1, and the second image stream includes a second image captured in the second field of view FOV2 and includes the object, the object having a second object-to-image magnification OTM2; and A processor is provided, the processor being configured to perform a geometric transformation on the first image and / or the second image so that the first object-to-image magnification OTM1 and the second object-to-image magnification OTM2 do not differ from each other by more than 10%.

2. The method according to claim 1, characterized in that The first object-to-image magnification OTM1 and the second object-to-image magnification OTM2 differ from each other by no more than 5%.

3. The method according to claim 1, characterized in that The geometric transformation uses calibration data.

4. The method according to claim 3, characterized in that The calibration data is stored in a non-volatile memory.

5. The method according to claim 3, characterized in that The calibration data comprises data regarding calibration between a plurality of object-to-image magnifications and a plurality of corresponding viewpoints POV in a scene.

6. The method according to claim 3, characterized in that The calibration data includes data regarding calibration between the first camera and the second camera.

7. The method according to claim 6, characterized in that The calibration data is represented by a bidirectional function that assigns any object-to-image magnification to a point of view POV of the second camera and vice versa.

8. The method according to claim 1, characterized in that The geometric transformation is a homography transformation.

9. The method according to claim 8, characterized in that The homography transform includes a motion-based calculation using a stream of frames from the first camera.

10. The method according to claim 9, characterized in that The motion-based calculation of the homography transform also uses an inertial measurement unit information.

11. The method according to claim 1, characterized in that The geometric transformation is a non-affine transformation.

12. The method according to claim 1, characterized in that The geometric transformation uses calibration information captured in a camera calibration procedure.

13. The method according to claim 1, characterized in that The second camera is directed toward a viewpoint POV according to a command from a user.

14. The method according to claim 1, wherein: The second camera is directed toward a point of view (POV) according to a command of a program.

15. The method according to any one of claims 1 to 14, characterized in that: The first camera is a wide-angle camera.

16. The method according to any one of claims 1 to 14, characterized in that: The first camera is an ultra-wide-angle camera.

17. The method according to any one of claims 1 to 14, characterized in that The method is executed by a mobile device.

18. The method according to claim 17, characterized in that The mobile device is a smart phone.

Images