Video support in multi-aperture mobile cameras with scan zoom cameras

Abstract

The present invention discloses various systems and methods for obtaining seamless, smooth transitions between a wide-angle field of view (FOV) camera and a scanning telephoto camera by correcting prismatic image aberrations, matching telephoto and wide-angle fields of view, positioning the associated telephoto field of view, and zooming a wide-angle frame to the positioned telephoto field of view.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 130,655, filed on December 26, 2020, the entirety of which is incorporated herein by reference. Technical Field

[0003] Embodiments disclosed herein generally relate to video support in various mobile cameras, particularly multi-aperture cameras in the presence of scanning cameras. Background Art

[0004] Multi-camera is a standard feature of modern mobile electronic handheld devices ("mobile devices", such as smartphones, tablets, etc.), of which dual-camera is a subcategory. Multi-camera usually includes wide angle or wide field of view (FOV W ) camera (“Wide camera” or “WC”) and one or more additional cameras, or a camera with a narrower field of view (FOV) (with a “native long focal field of view (“native FOV T 、native FOV T or n-FOV T ”) or a telephoto or “telephoto” camera with an ultra-wide-angle field of view (FOV UW ).

[0005] A "Scanning Tele camera" ("STC") covers or "scans" larger than n-FOV T This segment is called the "scanning Tele FOV" or "s-FOV T ". Field of view scanning can be performed by rotating one or more optical path folding elements ("OPFE"). For example, a scanning telephoto camera (STC) is described in U.S. Patent No. 10,578,948, which is commonly owned by the inventor of the present application.

[0006] In the case of a dual camera comprising a wide-angle camera (WC) and a telephoto camera, when the user zooms in or out in the video operation mode (image stream), the wide-angle image or the telephoto image is displayed to the user. When the "active" camera is switched, i.e. the camera displaying the output image is switched (from telephoto to wide-angle and vice versa), the user will see a jump or interruption in the video. This jump is caused by the different points of view (POV) of the two or more cameras. The perspective of a camera is defined as a unit vector, which is completely described by an origin and a direction. The origin is the center of the aperture of the respective camera. The direction of the perspective is given by the linear connection between the origin and the center of the specific field of view of the respective camera. In known dual cameras contained in mobile devices, the origins of the two cameras are slightly different, for example 5-25 mm.

[0007] "Smooth Transition" (ST) in the displayed video is a software feature that minimizes jumps when switching between image streams from different cameras (with different viewing angles). In current multi-cameras, the telephoto field of view (FOV T ) is basically centered on the wide-angle field of view (FOV W ) are the same (i.e., the center of the telephoto field of view is substantially the same as the center of the wide-angle field of view). For a mobile device such as a smartphone including multiple cameras, the field of view of the wide-angle camera and the field of view of the telephoto camera are substantially parallel to the normal of the rear surface of the smartphone. However, for a scanning telephoto camera (STC), the n-FOV T and viewing angles are not fixed, but they are scanning telephoto fields of view (s-FOV T ) scanning. Therefore, in general, when included in a smartphone, n-FOV T The center of the focal length is not the same as the center of the wide-angle field of view, and the viewing angle of the scanning telephoto camera (STC) is not parallel to the normal of the rear surface of the smartphone.

[0008] It is necessary and beneficial to have a smooth transition feature in a video image stream for smoothly switching between multiple Wide Angle Images ("WIs") and multiple Scanning Telephoto Images ("STIs"). Summary of the invention

[0009] Disclosed herein are systems and methods for obtaining smooth (i.e., seamless) transitions between cameras or viewing angles, which minimize jump effects during zoom-in (“zoom-in”) when switching from displaying Wide-angle image data to scanning Tele image data, or during zoom-out (“zoom-out”) when switching from displaying scanning Tele image data to Wide-angle image data in a video operation mode. Hereinafter, for simplicity, “Wide-angle image data” may be replaced by “Wide-angle image” or “WI”, and “scanning Tele image data” may be replaced by simply “scanning Tele camera (STC) data”, “scanning Tele image” or “STI”.

[0010] In various examples, a plurality of zoom cameras are provided, including: a wide-angle camera having a wide-angle field of view and configured to output a plurality of wide-angle images (WIs); a scanning telephoto camera (STC) having a telephoto field of view (n-FOV T ), the telephoto field of view is narrower than the wide-angle field of view and is used to output a plurality of scanning telephoto images; and a camera controller operably coupled to the wide-angle camera and the scanning telephoto camera and configured to stream a video image stream, the video image stream showing a continuous zoom-in action toward a non-central area within the wide-angle field of view or a continuous zoom-out action away from the non-central area, wherein the plurality of video images have a smooth transition when switching from displaying a plurality of wide-angle images to displaying a scanning plurality of scanning telephoto images or vice versa.

[0011] In some examples, the smooth transition can be achieved by one or more of the following ways: performing a rotation correction of the plurality of scanning telephoto images, performing a registration between the plurality of wide-angle images and the plurality of scanning telephoto images, or performing a positioning between the plurality of wide-angle images and the plurality of scanning telephoto images to perform position matching. Compared with the position matching before the positioning, the positioning between the plurality of wide-angle images and the plurality of scanning telephoto images improves the accuracy of position matching by more than 2.5 times.

[0012] In some examples, the smooth transition can be achieved by one of the following ways: when switching from displaying multiple wide-angle images to displaying multiple scanning telephoto images, the multiple scanning telephoto images are moved relative to the multiple wide-angle images according to a distance of an object in a region of interest (ROI) of a scanning telephoto image, and / or when switching from displaying multiple scanning telephoto images to displaying multiple wide-angle images, the multiple wide-angle images are moved relative to the multiple scanning telephoto images according to a distance of an object in a region of interest of a wide-angle image; applying blur on the multiple wide-angle images and / or the multiple scanning telephoto images; blending the multiple wide-angle images and the multiple scanning telephoto images; matching the scale and / or brightness and / or color between the multiple wide-angle images and the multiple scanning telephoto images; or cropping the multiple wide-angle images so that the respective cropping offsets of the multiple wide-angle images displayed sequentially are on a line. The cropping of the plurality of Wide images may include cropping such that a distance between respective crop offsets of the plurality of Wide images displayed sequentially changes linearly with a zoom factor, or cropping such that a distance between respective crop offsets of the plurality of Wide images displayed sequentially changes according to the square of a zoom factor. A crop offset and / or a crop center and / or a crop factor may be selected such that a selected object is included in a cropped scanning Tele image displayed to a user. The selected object may be located at a specific position in the cropped scanning Tele image.

[0013] In some examples, the smooth transition can be achieved by cropping the plurality of wide-angle images so that the respective coordinates of the centers of a wide-angle field of view of the plurality of wide-angle images displayed sequentially are located on a line. The cropping of the plurality of wide-angle images can include cropping so that the distance between the respective coordinates of the centers of the wide-angle fields of view of the plurality of wide-angle images displayed sequentially changes linearly with a zoom factor, or cropping so that the distance between the respective coordinates of the centers of the wide-angle fields of view of the plurality of wide-angle images displayed sequentially changes according to the square of a zoom factor.

[0014] In some examples, the smooth transition can be achieved by cropping the multiple wide-angle images so that a specific target area is always displayed in the video image stream, or by cropping the multiple wide-angle images so that the respective coordinates of a specific target area in the multiple wide-angle images displayed sequentially are located on a line. The cropping can cause the distance between the respective coordinates of the specific target area in the multiple wide-angle images displayed sequentially to change linearly or according to a square law.

[0015] In some examples, the switch from displaying multiple Wide-angle images to displaying multiple Scanning Telephoto images may be performed by shifting the ZF value upward (ZF 向上 ) is executed, the switch from displaying multiple scanning telephoto images to displaying multiple wide-angle images is performed by shifting the ZF value (ZF向下 ) where the upward transfer ZF value ≥ the downward transfer ZF value.

[0016] In some examples, the switch from displaying the multiple scanning telephoto images to displaying the multiple wide angle images may be performed by shifting the ZF value (ZF 向下 ) is executed, the downward transfer ZF value (ZF 向下 ) depends on the wide angle field of view (FOV W )Native FOV T (n-FOV T ) from a perspective, namely ZF 向下 =ZF 向下 (x, y), where ZF 向下 (POV) is the wide-angle field of view (FOV W ) is the downward shift ZF value of the center perspective, where ZF 向下 (Peripheral Viewing Angle (POV)) is the downward shifted ZF value of the peripheral viewing angle within the wide-angle field of view, and where ZF 向下 (Center view) <ZF 向下 (Edge view).

[0017] In some examples, the ZF value can be shifted upward (ZF 向上 ) performs switching from displaying a plurality of wide-angle images to displaying a plurality of scanning telephoto images, wherein ZF 向上 Does not depend on the wide angle field of view (FOV W )Native long focal field of view (n-FOV T ) perspective, where [ZF 向下 (x, y)] 最大 is the scanning field of view (s-FOV) of the scanning telephoto camera (STC) T ) for all possible perspectives 向下 The maximum value of (x, y), where ZF 向上 ≥[ZF 向下 (x, y)] 最大 .

[0018] In some examples, the switch from displaying multiple wide-angle images to displaying multiple scanning telephoto images can be ZF 向上 execution, which depends on n-FOV within the wide-angle field of view T perspective, where ZF 向上 =ZF 向上 (x, y).

[0019] In some examples, the switch from displaying multiple wide-angle images to displaying multiple scanning telephoto images can be performed in ZF 向上 Execute, switching from displaying multiple scanning telephoto images to displaying multiple wide-angle images can be ZF 向下Execution, where ZF in video shooting mode 向上 and / or ZF 向下 The value is higher than ZF in still photography mode. 向上 and / or ZF 向下 The value is 5%-30% larger. ZF 向上 and / or ZF 向下 The value of can depend on the aspect ratio of the wide angle and multiple scan telephoto images displayed to the user, respectively. ZF in Digital Image Stabilization Video Mode 向上 and / or ZF 向下 The value can be compared to ZF in non-image stabilized video mode. 向上 and / or ZF 向下 The value is 5%-30% larger.

[0020] In some examples, in digital image stabilization video mode, the image stabilization capability at the center angle of view can be 5%-30% greater than the image stabilization capability at the edge angles of view.

[0021] In some examples, a smooth transition may be achieved by cropping the plurality of scanning Tele images such that a particular position of a selected object in the cropped scanning Tele images displayed to a user does not change by >10 pixels in two consecutive images in the video image stream. In some examples, the particular position of the selected object may not change by >5 pixels. In some examples, the particular position of the selected object in the cropped scanning Tele images displayed to the user is selected based on aesthetic criteria.

[0022] In some examples, the camera controller may be additionally configured to evaluate a no-switch criterion before switching from displaying multiple Wide-angle images to displaying multiple Scanning Telephoto images, and not switch from displaying multiple Wide-angle images to displaying multiple Scanning Telephoto images if the no-switch criterion is met. The no-switch criterion may be selected from a group consisting of: motion blur, electronic noise, jello effect, defocus blur, and incorrect image alignment or obstruction. The no-switch criterion may include a significantly misposition of a region of interest (ROI) in a Scanning Telephoto image relative to its position in a Wide-angle image or a lack of the region of interest in the Scanning Telephoto image; an imperfect roll correction of the Scanning Telephoto image; a check to determine whether multiple target coordinates are contained in a scanning field of view (s-FOV) of the Scanning Telephoto camera. T ) in; a check to determine whether a plurality of target coordinates are close to the s-FOV of the scanning telephoto camera T multiple edges to prevent jumping back and forth between the wide angle image and the scanned telephoto image; or a check to determine whether multiple target coordinates are moving faster than an expected maximum prism scanning speed.

[0023] Zoom cameras above or below may be included in smartphones.

[0024] In some examples, a method is provided, comprising: using a device having a wide angle field of view (FOV W ) to output a plurality of wide-angle images (WI); using an n-FOV having a narrower field of view than the wide-angle field of view T A scanning telephoto camera (STC) is used to output a plurality of scanning telephoto images; and a camera controller is configured, the camera controller is operably coupled to the wide-angle camera and the scanning telephoto camera to stream a video image stream, the video image stream shows a continuous zoom-in action toward a non-central area within the wide-angle field of view or a continuous zoom-out action away from the non-central area, and provides a smooth transition of the plurality of video images when switching from displaying the plurality of wide-angle images to displaying the plurality of scanning telephoto images or vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Non-limiting examples of the disclosed subject matter will be described below with reference to the accompanying drawings listed after this paragraph. The same structure, element, or component appearing in multiple drawings will be labeled with the same reference numerals in the drawings in which they appear. The drawings and descriptions are intended to illustrate and clarify the embodiments disclosed herein and should not be construed as limiting in any way.

[0026] Figure 1 An embodiment of a mobile device including multiple cameras disclosed herein is schematically shown;

[0027] Figure 2 Different optical path folding element (OPFE) positions in the object field and their respective fields of view (FOV) are shown;

[0028] Figure 3 The main steps of the method disclosed herein for positioning a scanning telephoto camera (STC) image within a wide-angle image are shown in the form of a flow chart;

[0029] Figure 4A shows the positioning of the T image within the W image in one view;

[0030] Figure 4B shows the positioning of the T image within the W image in another view;

[0031] Figure 5A shows a zoomed-in scene displayed in a smooth transition video sequence created by a first method embodiment;

[0032] Figure 5B Shows Figure 5AMultiple images of a zoomed-in scene, digitally scaled to have the same aspect ratio;

[0033] Figure 5C Shows Figure 5B Some images and target locations in ;

[0034] Figure 6 The main steps of the cutting method are shown in the form of a flow chart;

[0035] Figure 7 Another embodiment of the method disclosed herein for achieving a smooth transition experience in a multi-camera including at least one camera with scanning capability is shown. DETAILED DESCRIPTION

[0036] Figure 1 An embodiment of a mobile device (e.g., a smartphone) is schematically shown, referenced 100, and includes a dual camera. The dual camera includes a scanning telephoto camera ("STC") 110 having a given native telephoto field of view (FOV T )(“n-FOV T ”) and the effective or scanned FOV T (“Scanning telephoto field of view (s-FOV T )”). The scanning telephoto camera 110 can be a folded camera, comprising an optical path folding element (OPFE) 118, a telephoto lens module 112 having a telephoto lens, and a telephoto image sensor 114. The telephoto lens can have a fixed effective focal length (EFL) that provides a fixed zoom factor (zoom factor, ZF), or have an adjustable (variable) effective focal length that provides an adjustable zoom factor. The adaptation of the EFL can be discrete or continuous. The scanning telephoto camera (STC) 110 also includes a lens actuator 116 for moving the lens module 112, wherein the lens module 112 is used for focusing and / or optical image stabilization (OIS); an optical path folding element actuator 122 for actuating the optical path folding element 118 for optical image stabilization and / or for adjusting the n-FOV T Scan-to-Scan Telephoto Field of View (s-FOV T ) within a specific perspective (POV); and a first memory 124. The memory 124 may be an EEPROM (Electrically Erasable Programmable Read Only Memory). In some embodiments, the first memory 124 may store first calibration data. The scanning telephoto camera (STC) 110 has an EFL of 8 mm to 50 mm or more and a diagonal FOV of n- T It is 10-40 degrees, and the f value is approximately f / #=1.5-6.

[0037] n-FOVT The scan is performed at a limited maximum speed, i.e. some settling time is required. N-FOV T The scanning may be performed on a time scale of about 1-30 milliseconds for scanning 2°-5° and on a time scale of about 10-80 milliseconds for scanning 10-25°. In some embodiments, the scanning telephoto field of view (s-FOV T ) can cover a wide angle of view (FOV W ) area. In some embodiments, the scanning telephoto field of view may cover 100% or more of the wide-angle field of view area.

[0038] In some embodiments, n-FOV T Scanning can be performed by actuating two or more optical path folding elements instead of a single optical path folding element, as described in International Patent Application No. PCT / IB2021 / 059843 commonly owned by the inventors of the present application.

[0039] The dual camera also includes a wide angle camera ("WC") 130, which has a wider angle field of view than the scanning telephoto camera (STC) 110. T . The wide angle camera (WC) 130 includes a wide angle lens module 132 having a wide angle lens and a wide angle image sensor 134. The second lens actuator 136 can move the lens module 132 for focusing and / or optical image stabilization. In some embodiments, the second calibration data can be stored in the second memory 138. In other embodiments, the first calibration data and the second calibration data can be stored in the third memory 170. The first and second calibration data may include calibration data between the wide angle camera (WC) 130 and the scanning telephoto camera (STC) 110. The wide angle camera (WC) may have an EFL of 2.5-20 mm, a diagonal field of view of 50-130 degrees, and an f / # of approximately 1.0-2.5.

[0040] The mobile device 100 may also include an application processor (AP) 140. The application processor 140 may include a wide-angle image signal processor (ISP) 144 and a telephoto image signal processor 142. The application processor 140 may also include a camera controller 150 having a sensor control unit 152, a user control unit 160, a video processing unit 154, and a post-processing unit 146, all of which are operably coupled to the image sensors 114 and 134. The user control unit 160 may include an operation mode selection function 162 for selecting whether to capture an image or a video, a region of interest (ROI) function 164 for selecting a region of interest, a "target coordinate" of the region of interest or a specific viewing angle thereof ("specific viewing angle" and "target coordinate" are used interchangeably hereinafter), and a zoom factor (ZF) module 166 for selecting a zoom factor. The region of interest may be a segment within a wide-angle field of view or a scanned telephoto field of view selected by a user or an algorithm. For a user, the region of interest may have a higher value than other segments, for example, because it contains a specific object or a specific combination of objects. Typically, a wide angle camera (WC) is focused on one of a plurality of regions of interest, and a scanning telephoto camera (STC) is steered toward and / or focused on one of the plurality of regions of interest.

[0041] In use, the application processor 140 may receive respective wide-angle and scanning telephoto camera (STC) image data from the cameras 110 and 130 , and provide camera control signals to the cameras 110 and 130 .

[0042] The sensor control unit 152 is operably connected to the two image signal processors (142 and 144) and the user control unit 160, and can be used to select which image sensor is operable according to the zoom factor and provide a sensor control signal. The video processing unit 154 can be configured to evaluate the no-switching criteria to make a decision about the video output. Specifically, when evaluating the no-switching criteria, if the no-switching criteria are met, the module 154 can be configured to output a zoomed video output image that only contains wide-angle image data in the zoom operation. The post-processing module 146 can be used for image processing, including denoising, sharpening, scaling, etc.

[0043] Here and below, unless otherwise stated, the following definitions shall apply:

[0044] Input image: an image provided by the Telephoto image signal processor 142 or the Wide-angle image signal processor 144 .

[0045] Scanning Telephoto Image (STI): an image provided by the Telephoto image signal processor 142 .

[0046] Wide-angle image (WI): an image provided by the wide-angle image signal processor 144 .

[0047] Output image: The image displayed to the user (usually a specific image in a video stream).

[0048] Telephoto output image: Based on the scanned telephoto image data and the step 608 ( Figure 6 ) or step 708 ( Figure 7 ) is provided in the output image.

[0049] Wide-angle output image: Based on the wide-angle image data and step 608 ( Figure 6 ) or step 708 ( Figure 7 ) is provided in the output image.

[0050] Output image data switching: In an image video stream, the action of switching from displaying a telephoto output image or a wide-angle output image to a user to displaying a wide-angle output image or a telephoto output image to the user, respectively.

[0051] "Image data" and "data" or "image" are used interchangeably hereinafter.

[0052] Zooming in and out in still camera mode

[0053] The definition is as follows: TFOV = tan(camera FOV / 2). "Low zoom factor (ZF)" means that the ZF <TFOV W / TFOV T All zoom factors. "High zoom factor (ZF)" means that ZF>TFOV W / TFOV T "Transfer ZF" or "ZFT" means the zoom factor that meets ZF = TFOV W / TFOV T ZFT represents the minimum zoom factor that allows the output image data to be switched. In one embodiment, zooming in and out in the static mode can be performed as follows:

[0054] Zoom in: At low ZF 向上 Slightly above ZFT, the output image is a digitally zoomed wide-angle output image. 向上 ”), ZF 向上 ≥ZFT, when the output image data is switched from displaying a Wide output image to displaying a Tele output image, the scanning Tele image is moved and corrected as described herein to achieve a smooth transition (ST). For ZF ≥ ZF 向上 For example, the output is a digitally zoomed telephoto output image.

[0055] Zoom out: At high ZF 向下Slightly above ZFT, the output image is a digitally zoomed telephoto output image. 向下 ”), ZF 向下 ≥ZFT, when performing an output image data switch from displaying a Tele output image to displaying a Wide output image, the Wide image is moved and corrected as described herein to achieve a smooth transition. For ZF≤ZF 向下 For 3D images, the output is a digitally zoomed wide-angle output image.

[0056] "Slightly higher than ZFT" means the zoom factor is about 1%-25% higher than ZFT.

[0057] In some examples, a global registration algorithm may be used to achieve a smooth transition.

[0058] In some embodiments, ZF 向上 and / or ZF 向下 Can be a fixed zoom factor, while ZF 向上 ≤ZF 向下 Or ZF 向上 ≥ZF 向下 .

[0059] In other embodiments, ZF 向上 and / or ZF 向下 The zoom factors may not be fixed, but they may vary (“dynamic transfer ZF”), e.g. depending on the viewing angle or the target coordinates (i.e. s-FOV T n-FOV within T , i.e. ZF 向下 =ZF 向下 (x, y) and / or ZF 向上 =ZF 向上 (x, y), "(x, y)" means s-FOV T Coordinates or perspectives within.

[0060] In some embodiments, in the s-FOV T The central area of ​​ZF 向下 May be smaller than s-FOV T The marginal area of ​​ZF 向下 (Central Area) <ZF 向下 (edge ​​area). The "center area" and the "edge area" are defined to be mutually exclusive. In one example, reference Figure 2 s-FOV shown T 200, the central area can be defined by all viewing angles pointing into a rectangular box having the same FOV as s- T200 Same aspect ratio and center, but only s-FOV respectively T 200 height and 50% width. Any direction s-FOV T 200 but not pointing to the center area are included in the edge area. In other examples, the rectangular box may only have s-FOV T 30% height and width of 200, or possibly up to s-FOV T 200, the definition of the center area and the edge area will change accordingly. In a scanning telephoto camera (STC) based on a single optical path folding element, in the s-FOV T The scanning telephoto image captured in the center area of ​​​​the object contains a larger field of view segment in the object domain than the scanning telephoto image captured in the edge area. This is because in order to T The unprocessed (i.e., uncorrected, uncropped, etc.) scan telephoto image at the center of the s-FOV is used to obtain a rectangular corrected scan telephoto image with a specific aspect ratio, which is the same as the image obtained from the s-FOV T Compared to obtaining a rectangular rectified scan tele image with the same specific aspect ratio from an unprocessed scan tele image at the inner edge positions, less cropping is required (i.e., less scene information is lost). The object domain is defined as the actual (or physical) scene, i.e., the scene captured by an ideal camera with a sufficiently large field of view and without any aberrations and distortions. In other words, the object domain corresponds to the appearance of the scene as it would appear to a human observer. Therefore, the ZF of the central region is 向上 and ZF 向下 Can be smaller than the edge area.

[0061] The advantage of using dynamic transfer ZF is that, at least in s-FOV T In the central area of ​​the image, the image quality advantages of using the Scanning Telephoto Camera (STC) can be enjoyed at lower ZF. In still photography mode, using dynamic up and down conversion zoom factors can be particularly beneficial. 向上 and / or ZF 向下 In some embodiments, ZF in video photography mode (i.e., when the user is capturing a video stream) 向上 and / or ZF 向下 The value may be higher than ZF in still photography mode. 向上 and / or ZF 向下 In some embodiments, ZF 向上 and / or ZF 向下 The value of may depend on the aspect ratio of the Wide output image and / or the Tele output image.

[0062] In some embodiments, wherein ZF 向上 =ZF 向上(x,y), ZF 向上 (Central Area) <ZF 向上 (marginal area).

[0063] In some embodiments, ZF 向上 can be fixed, and ZF 向下 Can be dynamic, i.e. ZF 向下 =ZF 向下 (x, y). Fixed ZF 向上 Can satisfy ZF 向上 ≥[ZF 向下 (x, y)] 最大 , i.e. fixed ZF 向上 by ZF 向下 The maximum value of (x, y) is defined when considering s-FOV T All viewing angles within ZF 向下 The maximum value of (x, y) is ZF 向下 This may be beneficial because it may prevent a situation where switching of output image data from a plurality of scanning tele images to a plurality of wide-angle images is performed even if the zoom factor has not changed. An example of such an undesirable situation involves the output image data being switched from a plurality of wide-angle images to a scanning tele image at a center position at the target coordinates (where ZF 向上 The object (i.e., target coordinates) may now move to the s-FOV. T A more edge position within the inner, where due to the n-FOV at this edge position T is not large enough, so the field of view defined by the corresponding ZF can no longer be supported by multiple scanned tele images. Due to the incomplete scanned tele image data, it is necessary to perform an output image data switch from multiple scanned tele images to multiple wide images, although the ZF has not changed. Use fixed ZF 向上 To satisfy the government 向上 ≥[ZF 向下 (x, y)] 最大 Can be particularly useful in video mode.

[0064] Smooth transition

[0065] In order to achieve a smooth transition, matching of the position, scale, brightness and color of the output image may be performed before and / or after the output image data is switched. However, in many cases, image position matching between the entire wide-angle image and the scanning telephoto image is not possible, for example because of parallax. Therefore, in a smooth transition as disclosed herein, position matching may be achieved only in the area of ​​the region of interest, while scale brightness and color are matched for the entire output image area. Specifically, when switching from displaying a plurality of wide-angle images to displaying a plurality of scanning telephoto images, the plurality of scanning telephoto images may be moved relative to the plurality of wide-angle images according to the distance of an object in the region of interest of the scanning telephoto image, and / or when switching from displaying a scanning telephoto image to displaying a wide-angle image, the plurality of wide-angle images may be moved relative to the plurality of scanning telephoto images according to the distance of an object in the region of interest of the wide-angle image.

[0066] Figure 2 Shows different light path folding element locations in the object domain and their respective n-FOVs T , as described in more detail in International Patent Application No. PCT / IB2021 / 056311. The object domain is different from the "image domain", which is defined as the scene captured by the camera with aberrations and deformations. In the present invention, the image domain is defined as the scene captured by a scanning telephoto camera (STC) such as camera 110. Box 200 represents the s-FOV T , i.e., the smallest rectangular field of view, containing all STC image data in all viewing angles, which can be achieved using a specific STC in the target domain. n-FOV for three different optical path folding element positions (0, 1, and 2) T Indicated by 202-0, 202-1 and 202-2. n-FOV of the optical path folding element "zero position" T 202-0 is defined as the n-FOV that produces an image of an object or scene without perspective aberrations T That is, at the null position, the object in the object domain is identical to the image of the object in the image domain (except for scaling, image sensor noise, etc.). In general, at the null position, n-FOV T Overlaps with the center of the wide-angle field of view. As shown in the figure, the n-FOV at any other position (such as 202-1 and 202-2) T It is not a horizontal rectangle (such as 202-0), but an arbitrary quadrilateral. The same rectangular object is represented by T 204-0, 204-1 and 204-2 in 202-0, 202-1 and 202-2 are represented.

[0067] Figure 3The main steps of the method disclosed herein for locating a scanning telephoto image within a wide-angle image are shown; in step 302, a scanning telephoto image is captured. Before capturing, n-FOV T The scanned telephoto image is scanned toward a specific viewing angle. As described in International Patent Application No. PCT / IB2021 / 056311, the scanned telephoto image is corrected in step 304. By using appropriate crop selection criteria, the content contained in the corrected and cropped scanned telephoto image, as well as the position of multiple objects in the corrected and cropped scanned telephoto image, can be manipulated. The crop selection criteria refers to the selection of the image at a specific n-FOV. T A specific scan telephoto image captured at a certain position is used to correct and crop the specific criteria of the specific scan telephoto image. Cropping includes defining a crop offset and / or a crop center and / or a crop factor. The crop offset is the position of the upper left corner of the image in the object domain. The crop factor is usually given by 1 / ZF.

[0068] In some examples, the cropping may be modified so that the selected object is included in the cropped scanned telephoto image.

[0069] In other examples, the cropping may be modified so that the selected object is located at a specific location in the cropped scanned telephoto image.

[0070] In some examples, for video mode, the cropping may be modified so that the position of a selected object in two consecutive frames of the video stream does not change by more than 10%, 5%, or 1% of the width and / or height of the telephoto sensor.

[0071] In other examples, the position of the selected object in two consecutive frames may vary by less than 50 pixels, or less than 20 pixels, or even less than 5 pixels (assuming a pixel size of 1 micron).

[0072] In yet another example, the cropping may be modified so that the position of a selected object in consecutive frames of the video stream changes only slowly, e.g., at a frequency of no more than 10 pixels / second, or 5 pixels / second, or even 2 pixels / second (assuming a pixel size of 1 micron).

[0073] In other examples, the cropping can be modified so that the position of the selected object meets aesthetic criteria. The aesthetic criteria can be based on, for example, the position of the selected object in the cropped scanned telephoto image (aesthetic frame), such as the photographer's rule of thirds, whether additional objects are included or excluded in the cropped scanned telephoto image, such as the photographer's rule of leading lines, etc. For example, Figure 4AAs described above, positioning of the scanning Tele image within the Wide image is performed in step 306. Image registration known in the art is performed between the cropped scanning Tele image and the Wide image in step 308. Output image data switching may be performed in step 310, as outlined in more detail below. The same steps may be performed for the zoom-out case, where a switch may be made from Tele output image data to Wide output image data. In some examples, the switching step may include a blending or fusion of multiple scanning Tele images and multiple Wide images.

[0074] Figure 4A -B shows details of the positioning of the scanned Tele image within the Wide-angle image performed in step 306, as described in more detail in International Patent Application No. PCT / IB2020 / 061461 commonly owned by the inventors of the present application.

[0075] exist Figure 4A , n-FOV is shown at an estimated viewing angle within the wide-angle field of view 404 T The estimated field of view 402 is the field of view estimated based on the calibration data. T The estimate of is calibration dependent and may not be accurate enough in matching the multiple scanned telephoto images with the multiple wide angle images. Typically, the image point coordinates of the same object point in the scanned telephoto image and the wide angle image may deviate by more than 25, 50 or even 100 pixels before localization. Assume that a pixel size is about 1 micron. Localization (step 306) is performed to reduce this deviation, i.e., to achieve a precise n-FOV with higher accuracy. T Estimated. Positioning includes:

[0076] 1. Select Figure 4A The search area 406a is shown. The search area 406 can be determined by using n-FOV T The estimated center is defined by embedding it (e.g. symmetrically) in a rectangular region, where the area of ​​the rectangle may be n-FOV T Estimate double, triple or quadruple the area covered.

[0077] 2. Crop the search area from the wide-angle image.

[0078] 3. Template matching, where the source is represented by a cropped search area from the wide-angle image, and the template can be represented by a scanned telephoto image. Template matching can be performed by correlation of the template at different locations in the search area or over the entire search area. The location with the highest match value can indicate the location within the wide-angle field of view n-FOV T The best estimate of the position. This more accurate precision n-FOV T Estimated Figure 4B408 in the figure, and can be further used to perform steps 308 and 310. Generally, after positioning, the deviation of the image point of the same object point between the wide-angle image and the scanning telephoto image may be less than 5 pixels, or even less than 2 pixels, that is, the accuracy is improved by 5, 10 or even 20 times.

[0079] Figure 5A -C shows a smooth transition video sequence created by the first cropping method ("Method 1") disclosed herein. In the following, the index "0" represents the initial state, the index "f" represents the final (or target) state, and the index "t" represents some intermediate state. "State" refers to an image defined by ZF, crop factor, and crop offset. "Final state" refers to the last image displayed in the smooth transition video sequence, which contains wide-angle output image data. When zooming in further, output image data switching can be performed.

[0080] Figure 5A 5 shows how a zoomed-in scene may be displayed in a cross-fade video sequence created by method 1. Image 502 may represent the first image in the cross-fade video sequence. Image 502 may have ZF=1.0, i.e., it contains the full wide-angle field of view, or it may have ZF>1. Images 504-512 represent subsequent images in the cross-fade video sequence toward a "target" location 514, with coordinates (x f , x f ) and target ZF (“ZF f ”). The target position 514 is the position to which the zoom is pointing. The n-FOV of the scanning telephoto camera (STC) T is scanned so that the target position 514 is located within n-FOV T Center. Location 514 and ZF f Can be selected by the user or program before or at the beginning of the zoom action. Images 504-512 are wide angle output images. The cropping offsets of images 504-512 are represented by image frames 516-528, respectively. An advantage of method 1 is that the cropping is selected so that the target location 514 is included in each of images 502-512, that is, the target location 514 is displayed in the entire cross-fading video sequence.

[0081] Figure 5B Images 504-512 are shown as being subsequently displayed to the user in an up-scaled cross-faded video sequence, ie, digitally scaled to have the same aspect ratio.

[0082] Figure 5C The image 502, the image 512, and the target position 514 are shown. Reference numeral 532 represents the initial X coordinate X of the target position 514 in the coordinate system of the image 502. 0, the number 534 represents the target X coordinate X of the target position 514 in the coordinate system of image 512 f . Accordingly, the Y coordinate Y is defined 0 and Y f . The target position 514 is defined in the object domain. The widths and heights of images 502 and 512 (measured along the x-axis and y-axis respectively) are given by W502 and W512, H502 and H512. The relative positions (“relPos 0，x ”) and (“relPos f，x ”) can be derived from 532 and 534 by relPos 0，x = X 0 / W502 and relPos f，x = x f / W512. The relative position in the Y direction can be defined accordingly. By gradually modifying the relative coordinates of the target position (X-axis and Y-axis), smooth transformation can be achieved. At the target ZF f , the relative position of the target position 514 is relPos f，x = relPos f，y = 1 / 2, that is, the target position 514 coincides with the center of the output image.

[0083] Method 1 can be similarly used for reduction.

[0084] Figure 6 Shows the main steps of Method 1. It can start from the initial (or “0”) state, using ZF 0 <ZFT and a video stream showing a wide-angle output image (such as images 502 - 512). The positioning of n-FOV T within the wide-angle field of view (for example Figure 4A -B) is performed.

[0085] Generally, the relative position at certain intermediate states (“t”) can be calculated according to relPos t = (X t / W t , Y t / H t ), where W t and H t are the width and height of the output image displayed at the intermediate state t respectively, where W t = W / ZF t and H t = H / ZF t are used to define the cropping factor, where W and H are the width and height of the uncropped wide-angle image respectively.

[0086] The initial state relative position relPos 0The target coordinates (X 0 , Y 0 ) in the coordinate system of the image 502 are calculated as relPos 0 = (X 0 / W 0 , Y 0 / H 0 ), where W 0 = W / ZF 0 and H 0 = H / ZF 0 are the width and height of the output image displayed in the initial state 0, respectively. The conversion slope S from relPos 0 to the relative position in the target state relPos f is derived from the following formula:

[0087]

[0088] where ZF P is a preset and fixed ZF value that satisfies ZF P ≥ ZFT.

[0089] In step 602, the program or the user uses a ZF (ZF t1 ) to trigger a command to zoom in to the first intermediate state ("t1"), where ZF t1 ≥ ZFT, so that the output image data switching (from wide-angle image data to scanning telephoto camera (STC) image data) can be performed. The initial output image can be an image that displays the entire wide-angle field of view (i.e., with ZF 0 = 1), or it can be an image with 1 < ZF 0 < ZFT, which displays a (digitally zoomed) section of the wide-angle field of view.

[0090] In step 604, the relative position update is calculated according to relPos t1 = relPos 0 + (ZF t1 - ZF 0 ). Generally, the relative position update can be relPos t = relPos t-1 + (ZF t - ZF t-1 ) · S.

[0091] In step 606, the cropping coefficient and the cropping offset are calculated. The cropping offset (X C , Y C ) is calculated by first calculating the relative coordinates as follows: X t1 = relPos t1，x · Wt1 and Y t1 =relPos t1，y ·H t1 , where relPos t，x and relPos t，y They are relPos t By subtracting the X value from the target coordinates (in the coordinate system of image t1) t1 and Y t1 , calculate the position of the clipping offset (X C , Y C ).

[0092] In step 608, based on the wide-angle image data and according to the cropping offset (X C , Y C ) and has the image size (W t , H t ) is output and displayed in a smooth transition video sequence. If further zooming is performed, the process can start from step 602 again.

[0093] In some examples, the following output image data switching may switch from using only the Wide angle image input to using only the Scanning Telephoto image input.

[0094] In the second cropping method ("Method 2"), a smooth transition can be achieved by linearly modifying the cropping offset. One method can consider the initial cropping offset (X C，0 , Y C，0 ) and the target clipping offset (X C，f , Y C，f ). In method 2, when zooming in or out, the crop offset is always located on this linear connection, such as Figure 5A As shown. In some examples, the position of the crop offset on the linear connection is determined linearly. In other examples, the position is determined by the square law. For example, when zooming from ZF=1 to ZF=2, the difference in the position of the crop offset on the linear connection (i.e., the distance between each crop offset) is 2 times larger than when zooming from ZF=2 to ZF=4.

[0095] In a third cropping method ("Method 3"), a smooth transition may be achieved by linearly modifying the center of the field of view shown in the smooth transition video stream. One may consider a linear connection between the center of the initial field of view and the center of the target field of view. In Method 3, when zooming in or out, the center of the field of view may always be located on this linear connection. The location of the cropping offset on the linear connection may be determined by a linear law or a square law.

[0096] Figure 7An embodiment of the smooth transition method disclosed herein is shown. For each image of the video stream, a Wide-angle input image and a Telephoto input image and the field of view of a scanning telephoto camera (STC) are retrieved in step 702. The Telephoto input image may be pre-corrected to align with the Wide-angle field of view, or corrected in step 702. Optionally, in step 702, the OFPE position may be converted to n-FOV using calibration data T For example, when changing n-FOV by scanning T In this case, a registration procedure can be performed to calculate the translation and rotation parameters of the current scanned Tele image relative to the Wide image. The translation can compensate for parallax and the rotation can compensate for residual rotation between the scanned Tele image and the Wide image, which may be caused by an imperfect calibration.

[0097] In some examples, the multi-camera may include image stabilization (IS), such as optical image stabilization (OIS) or electronic image stabilization (EIS) for the scanning telephoto camera (STC) and / or the wide angle camera (WC). Since image stabilization may move the wide angle image and the scanning telephoto image relative to each other, image stabilization control input data, such as sensor data provided from an inertial measurement unit (IMU), or image stabilization control output data, such as a command to move a specific optical component a given amount, may be read in step 702. The data may be converted to expected pixel shifts in the wide angle image and the scanning telephoto image, and this conversion may be used to compensate for any undesirable effects that the image stabilization mechanism may have on smoothly transitioning video sequences. As is well known, for electronic image stabilization, the input images are cropped and shifted so that a selected object or entire scene remains located at the same (or similar) specific position in two or more consecutive output images of the video stream. To prevent jumping between the scanning telephoto camera (STC) and the wide angle camera (WC), the ZF in the digital image stabilization video mode 向上 and / or ZF 向下 The values ​​are comparable to ZF in non-image stabilized video mode 向上 and / or ZF 向下 The value of 2.5%-50% is greater than that of . In addition, the ability to perform electronic image stabilization (e.g., measured by the maximum number of pixels that can be moved) may be 2.5%-50% greater at the center viewing angle than at the edge viewing angle. Here, image stabilization can be used to achieve two different goals. The first goal can be to stabilize the Wide-angle camera (WC) or the Scanning Telephoto Camera (STC) at about 50-100Hz and higher frequencies, for example to alleviate user hand shake. The first goal can be to firmly frame the scene, that is, to stabilize the Wide-angle camera (WC) or the Scanning Telephoto Camera (STC) at about 20Hz and lower frequencies (e.g., 1Hz) to frame the scene in a wide-angle field of view or n-FOV T inside, or in a wide-angle field of view or n-FOVT The selected scene (of the object domain) is maintained at a specific position within the object domain. In step 704, the wide-angle input image or the telephoto input image is selected as the output image. The wide-angle input image data can be selected for the following situations:

[0098] For any low ZF, or

[0099] For high ZF, but one or more no-switch criteria are met. Evaluating the no-switch criteria avoids switching the output image data in situations where a smooth transition is not possible or may be disadvantageous. After capturing a scan tele image, the following no-switch criteria may be evaluated:

[0100] 1. Poor scan telephoto image quality. Low image quality may be caused by large electronic noise at low scene illumination, for example, due to scan telephoto camera (STC) out-of-focus (defocus blur), or due to motion blur, rolling shutter artifacts, or other artifacts known in the art.

[0101] 2. Inappropriate scanning telephoto image composition. This may include, for example, the following scenarios: (i) the scanning telephoto image shows a scene that is significantly different from the scene shown in the wide-angle image in the smooth transition video stream that is output to the user before the scanning telephoto image, or (ii) the scanning telephoto image shows a scene that is significantly semantically inconsistent with the scene shown in the wide-angle image (e.g., motion occurred between the captured scanning telephoto and wide-angle images), or (iii) the scanning telephoto image does not show the region of interest (e.g., an obstacle) contained in the wide-angle image, or it does show the region of interest at a significantly wrong location relative to the wide-angle image (i.e., the object coordinates in the wide-angle camera (WC) coordinate system are significantly different from the object coordinates in the scanning telephoto camera (STC) coordinate system).

[0102] 3. Imperfect scanning telephoto image correction, such as imperfect rolling correction of scanning telephoto images.

[0103] When capturing a scanned telephoto image or n-FOV T Before scanning to specific target coordinates, the following no-switch criteria can be evaluated:

[0104] 1. Effective scanning telephoto camera (STC) scanning range: evaluate whether the target coordinates are contained in the s-FOV T middle.

[0105] 2. Prevent the jump from scanning telephoto camera (STC) to wide angle camera (WC): evaluate whether the target coordinates are close to s-FOV TIn this case, even a small relative movement of the ROI and the STC can lead to a situation where the ROI can no longer be fully covered by the STC image and the WC image must be used. To prevent jumps between the STC and WC, if the target coordinates are located close to the s-FoV T At a distance of 5%, 10% or 20% (in the horizontal or vertical direction), it is not possible to switch to scanning telephoto image output data.

[0106] 3. Effective ROI velocity: Evaluate the velocity of the ROI (and the corresponding target coordinates) to analyze the n-FOV T Is it possible to follow this movement.

[0107] The input image selected as the output image is cropped in step 706. For example, based on the selected input image, the current ZF, the image cropping required for the input image, etc. are calculated. The calculation may be performed according to cropping method 1, cropping method 2, or cropping method 3. The generated cropped image may be displayed (or output) to the user.

[0108] Optionally, and depending on, for example, a) the selected input image, b) the cropping of the input image, and / or c) image registration parameters, a smoothly transformed output image may be provided in step 708 .

[0109] In other examples, the method disclosed herein may not include the cropping action, but may only provide parameters calculated as described herein, and the actions of steps 706 and 708 (e.g., cropping, providing video, etc.) may be performed on different programs or processors, such as on dedicated hardware (HW) that supports hardware acceleration.

[0110] To compensate for the resolution difference between the multiple scanned telephoto images and the multiple wide-angle images, blurring may be applied on the output images.

[0111] Rotation correction may also be applied in step 708. The difference in lens distortion between the Wide Camera (WC) and the Scanning Telephoto Camera (STC) is another challenge. For a smooth transition, distortion correction may be applied to all input images, or alternatively, digital distortion may be applied only to the Telephoto input images or only to the Wide input images so that they match the corresponding other input images.

[0112] Unless otherwise indicated, use of the expression "and / or" between the last two members of a list of options for selection indicates that selection of one or more of the listed options is appropriate and can be made.

[0113] It should be understood that when the claims or specification refer to "a" or "an" element, such reference should not be construed as meaning that there is only one of the plurality of elements.

[0114] All patents, patent applications and publications mentioned in this specification are fully incorporated into this specification by reference, to the same extent as each individual patent, patent application or publication is expressly and individually indicated to be incorporated into this specification by reference. In addition, any reference document cited or identified in this application shall not be construed as an admission that such reference document is available as prior art for the present disclosure.

Claims

1. A zoom camera, characterized in that: The zoom camera comprises: a wide-angle camera having a wide-angle field of view and configured to output a plurality of wide-angle images; a scanning telephoto camera having a native scanning telephoto field of view n-FOVT, the native scanning telephoto field of view being narrower than the wide-angle field of view and configured to output a plurality of scanning telephoto images; and a camera controller operably coupled to the wide-angle camera and the scanning telephoto camera and configured to stream a video image stream showing a continuous zoom-in motion toward a non-central area within the wide-angle field of view or a continuous zoom-out motion away from the non-central area, Wherein when switching from displaying a plurality of wide-angle images to displaying a plurality of scanning telephoto images or switching from displaying a plurality of scanning telephoto images to displaying a plurality of wide-angle images, a plurality of video images have a smooth transition, and wherein when performing a switch from displaying a plurality of wide-angle images to displaying a plurality of scanning telephoto images, the smooth transition is achieved by moving the plurality of scanning telephoto images relative to the plurality of wide-angle images according to a distance of an object in an area of ​​interest of a scanning telephoto image, and / or when performing a switch from displaying a plurality of scanning telephoto images to displaying a plurality of wide-angle images, the smooth transition is achieved by moving the plurality of wide-angle images relative to the plurality of scanning telephoto images according to a distance of an object in an area of ​​interest of a wide-angle image.

Images