Mobile device

Abstract

Mobile devices for seamless, smooth transition between a wide-angle field (FOV) camera and a scanning telephoto camera by correcting prism image differences, matching telephoto and wide-angle fields, locating an associated telephoto field, and zooming a wide-angle frame to the located telephoto field.

Description

[0001] This application is a divisional application of the application with application number 202180013654.6 (PCT application number PCT / IB2021 / 062305), application date December 24, 2021, and invention title "Video Support in a Multi-Aperture Mobile Camera with a Scanning Zoom Camera".

[0002] Related Applications

[0003] This application claims the priority of U.S. Provisional Patent Application No. 63 / 130,655, filed on December 26, 2020, the entire content of which is incorporated herein by reference. Technical Field

[0004] The embodiments disclosed herein generally relate to various mobile cameras, and more particularly to video support in a multi-aperture camera in the presence of a scanning camera. Background Art

[0005] Multi-cameras are standard in modern mobile electronic handheld devices ("mobile devices" such as smartphones, tablets, etc.), and dual-cameras are a subclass thereof. Multi-cameras typically include a wide-angle (Wide angle) or wide field of view (FOV W ) camera ("wide camera (Wide camera)" or "WC") and one or more additional cameras, or have a narrower field of view (FOV) (with a "native telephoto field of view ("native FOV T , native FOV T or n-FOV T ") telephoto or "tele" camera), or have an ultra-wide field of view (FOV UW ).

[0006] A "scanning tele camera" ("STC") covers or "scans" a segment of a scene larger than the n-FOV T . This segment is called the "scanning telephoto field of view" or "s-FOV T ". The field of view scanning can be performed by rotating one or more optical path folding elements ("OPFE"). For example, the scanning tele camera (STC) is described in U.S. Patent No. 10578948, jointly owned by the inventors of the present application.

[0007] For a dual camera that includes a wide-angle camera (WC) and a telephoto camera, when the user zooms in or out in the video operation mode (image stream), a wide-angle image or a telephoto image is displayed to the user. When switching the "active" camera, that is, switching the camera that displays the output image (from telephoto to wide-angle, or vice versa), the user will see a jump or interruption in the video. This jump is caused by the different points of view (POV) of two or more cameras. The point of view of a camera is defined as a unit vector and is completely described by an origin and a direction. The origin is the center of the aperture of the corresponding camera. The direction of the point of view is given by the linear connection between the origin and the center of the specific field of view of the corresponding camera. In known dual cameras incorporated in mobile devices, the origins of the two cameras are slightly different, for example, 5 - 25 millimeters.

[0008] "Smooth transition" (ST) in the displayed video is a software function that minimizes the jump when switching between image streams of different cameras (with different points of view). In current multi-cameras, the center of the telephoto field of view (FOV T ) is substantially the same as the center of the wide-angle field of view (FOV W ), that is, the center of the telephoto field of view is substantially the same as the center of the wide-angle field of view. For a mobile device such as a smartphone incorporating a multi-camera, the point of view of the wide-angle camera and the point of view of the telephoto camera are substantially parallel to the normal of the rear surface of the smartphone. However, for a scanning telephoto camera (STC), the n-FOV T and the point of view are not fixed, but they scan within the scanning telephoto field of view (s-FOV T ). Therefore, generally speaking, when incorporated in a smartphone, the center of the n-FOV T is not the same as the center of the wide-angle field of view, and the point of view of the scanning telephoto camera (STC) is not parallel to the normal of the rear surface of the smartphone.

[0009] It is necessary and beneficial to have a smooth transition feature for smooth switching between multiple wide-angle images ("WIs") and multiple scanning telephoto images ("STIs") in the video image stream. SUMMARY OF THE INVENTION

[0010] The present disclosure provides systems and methods for obtaining smooth (i.e., seamless) transitions between cameras or viewpoints, minimizing jump effects in video operation modes during zoom-in when switching from displaying wide-angle image data to scanning long-focus image data, or during zoom-out when switching from displaying scanning long-focus image data to wide-angle image data. Hereinafter, for simplicity, "wide-angle image data" may be replaced by "wide-angle image" or "WI", and "scanning long-focus image data" may be replaced only by "scanning telecamera (STC) data", "scanning long-focus image", or "STI".

[0011] In various examples, a plurality of zoom cameras are provided, including: a wide-angle camera having a wide-angle field of view and configured to output a plurality of wide-angle images (WIs); a scanning telecamera (STC) having a long-focus field of view (n-FOV T ) that is narrower than the wide-angle field of view and configured to output a plurality of scanned long-focus images; and a camera controller operatively coupled to the wide-angle camera and the scanning telecamera and configured to stream a video image stream that displays a continuous zoom-in action towards a non-central region within the wide-angle field of view or a continuous zoom-out action away from the non-central region, wherein the plurality of video images have a smooth transition when switching from displaying a plurality of wide-angle images to displaying a plurality of scanned long-focus images or vice versa.

[0012] In some examples, the smooth transition may be achieved by one or more of the following: performing a rotation correction of the plurality of scanned long-focus images, performing a registration between the plurality of wide-angle images and the plurality of scanned long-focus images, or performing a positioning between the plurality of wide-angle images and the plurality of scanned long-focus images to perform a position match. The positioning between the plurality of wide-angle images and the plurality of scanned long-focus images improves the accuracy of the position match by more than 2.5 times compared to the position match before the positioning.

[0013] In some examples, the smooth transition may be achieved in one of the following ways: when switching from displaying multiple wide-angle images to displaying multiple scanned telephoto images, move the multiple scanned telephoto images according to the distance of an object in a region of interest (ROI) of a scanned telephoto image relative to the multiple wide-angle images, and / or when switching from displaying multiple scanned telephoto images to displaying multiple wide-angle images, move the multiple wide-angle images according to the distance of an object in a region of interest of a wide-angle image relative to the multiple scanned telephoto images; apply blur to the multiple wide-angle images and / or the multiple scanned telephoto images; blend the multiple wide-angle images and the multiple scanned telephoto images; match the scale and / or brightness and / or color between the multiple wide-angle images and the multiple scanned telephoto images; or crop the multiple wide-angle images such that the respective crop offsets of the multiple wide-angle images displayed in sequence are on a line. The cropping of the multiple wide-angle images may include cropping such that the distance between the respective crop offsets of the multiple wide-angle images displayed in sequence changes linearly with a zoom factor, or cropping such that the distance between the respective crop offsets of the multiple wide-angle images displayed in sequence changes according to the square of a zoom factor. A crop offset and / or a crop center and / or a crop coefficient may be selected such that a selected object is included in a cropped scanned telephoto image displayed to a user. The selected object may be located at a specific position in the cropped scanned telephoto image.

[0014] In some examples, the smooth transition may be achieved by cropping the multiple wide-angle images such that the respective coordinates of the center of a wide-angle field of view of the multiple wide-angle images displayed in sequence are on a line. The cropping of the multiple wide-angle images may include cropping such that the distance between the respective coordinates of the center of the wide-angle field of view of the multiple wide-angle images displayed in sequence changes linearly with a zoom factor, or cropping such that the distance between the respective coordinates of the center of the wide-angle field of view of the multiple wide-angle images displayed in sequence changes according to the square of a zoom factor.

[0015] In some examples, the smooth transition may be achieved by cropping the multiple wide-angle images such that a specific target area is always displayed in the video image stream, or by cropping the multiple wide-angle images such that the respective coordinates of a specific target area in the multiple wide-angle images displayed in sequence are on a line. The cropping may be such that the distance between the respective coordinates of the specific target area in the multiple wide-angle images displayed in sequence changes linearly, or changes according to a square law.

[0016] In some examples, the switch from displaying multiple wide-angle images to displaying multiple scanned telephoto images may be performed with an upward shift of the ZF value (ZF 向上 ) and the switch from displaying multiple scanned telephoto images to displaying multiple wide-angle images is at a downward shift of the ZF value (ZF向下 ) is executed with the upward transfer ZF value ≥ the downward transfer ZF value.

[0017] In some examples, the switch from displaying multiple scanned long - focal images to displaying multiple wide - angle images can be made with a downward transfer ZF value (ZF 向下 ) executed, and the downward transfer ZF value (ZF 向下 ) depends on a viewpoint within the wide - angle field of view (FOV W ) of the native FOV T (n - FOV T ), that is, ZF 向下 = ZF 向下 (x, y), where ZF 向下 (central point of view (POV)) is the downward transfer ZF value of the central point of view within the wide - angle field of view (FOV W ), where ZF 向下 (edge point of view (POV)) is the downward transfer ZF value of the edge point of view within the wide - angle field of view, and where ZF 向下 (central point of view) < ZF 向下 (edge point of view).

[0018] In some examples, the switch from displaying multiple wide - angle images to displaying multiple scanned long - focal images can be made with an upward transfer ZF value (ZF 向上 ), where ZF 向上 does not depend on the viewpoint of the native long - focal field of view (n - FOV W ) within the wide - angle field of view (FOV T ), where [ZF 向下 (x, y)] 最大 is the maximum value of ZF T (x, y) for all possible viewpoints within the scanning field of view (s - FOV 向下 ) of the scanning long - focal camera (STC), where ZF 向上 ≥ [ZF 向下 (x, y)] 最大 .

[0019] In some examples, the switch from displaying multiple wide - angle images to displaying multiple scanned long - focal images can be made with ZF 向上 executed, which depends on the viewpoint of the n - FOV T within the wide - angle field of view, where ZF 向上 = ZF 向 up(x, y).

[0020] In some examples, the switch from displaying multiple wide - angle images to displaying multiple scanned long - focal images can be made with ZF 向上 executed, and the switch from displaying multiple scanned long - focal images to displaying multiple wide - angle images can be made with ZF 向下Execute, where ZF in video shooting mode 向上 and / or ZF 向下 has a value that is 5% - 30% greater than ZF in still shooting mode 向上 and / or ZF 向下 . The value of ZF 向上 and / or ZF 向下 can depend on the aspect ratios of the wide-angle and multiple scanned telephoto images presented to the user, respectively. ZF in digital image stabilization video mode 向上 and / or ZF 向下 can have a value that is 5% - 30% greater than ZF in non-image stabilization video mode 向上 and / or ZF 向下 .

[0021] In some examples, in digital image stabilization video mode, the image stabilization ability at the central viewing point can be 5% - 30% greater than the image stabilization ability at the peripheral viewing points.

[0022] In some examples, smooth transition can be achieved by cropping the multiple scanned telephoto images such that a specific position of a selected object in the cropped scanned telephoto images presented to a user in two consecutive images of the video image stream does not change by > 10 pixels. In certain examples, the specific position of the selected object may not change by > 5 pixels. In some examples, the specific position of the selected object in the cropped scanned telephoto images presented to the user is selected according to aesthetic criteria.

[0023] In some examples, the camera controller can additionally be configured to evaluate a non-switching criterion before switching from displaying multiple wide-angle images to displaying multiple scanned telephoto images, and if the non-switching criterion is met, not switch from displaying multiple wide-angle images to displaying multiple scanned telephoto images. The non-switching criterion can be selected from a group consisting of: motion blur, electronic noise, jelly effect, defocus blur, and incorrect image alignment or obstacles. The non-switching criterion can include a significant misposition of an area of interest (ROI) in a scanned telephoto image relative to its position in a wide-angle image or the lack of the area of interest in the scanned telephoto image; an imperfect roll correction of the scanned telephoto image; a check to determine whether multiple target coordinates are included in a scan field of view (s-FOV T ) of the scanned telephoto camera; a check to determine whether multiple target coordinates are close to multiple edges of the s-FOV T of the scanned telephoto camera to prevent jumping back and forth between the wide-angle image and the scanned telephoto image; or a check to determine whether multiple target coordinates are moving faster than an expected maximum prism scan speed.

[0024] A zoom camera above or below can be included in a smart phone.

[0025] In some examples, a method is provided, including: using a wide-angle camera with a wide field of view (FOV W ) to output a plurality of wide-angle images (WI); using a scanning telephoto camera (STC) with an n-FOV narrower than the wide field of view T to output a plurality of scanned telephoto images; and configuring a camera controller operably coupled to the wide-angle camera and the scanning telephoto camera to stream a video image stream that shows a continuous zoom-in action towards a non-central area or a continuous zoom-out action away from the non-central area within the wide field of view, and to provide a smooth transition for a plurality of video images when switching from displaying the plurality of wide-angle images to displaying the plurality of scanned telephoto images or vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Non-limiting examples of the subject matter of the present disclosure will be described below with reference to the drawings listed later in this section. The same structures, elements, or components that appear in multiple drawings will be labeled with the same reference numerals in the drawings in which they appear. The drawings and the description are intended to illustrate and clarify the embodiments disclosed herein and should not be considered to limit in any way.

[0027] Figure 1 An embodiment of a mobile device including a multi-camera disclosed herein is schematically shown;

[0028] Figure 2 Different optical path folding element (OPFE) positions in the object domain and their respective fields of view (FOV) are shown;

[0029] Figure 3 The main steps of a method disclosed herein for positioning a scanning telephoto camera (STC) image within a wide-angle image are shown in the form of a flowchart;

[0030] Figure 4A The positioning of a T image within a W image in one view is shown;

[0031] Figure 4B The positioning of a T image within a W image in another view is shown;

[0032] Figure 5A An enlarged scene shown in a smooth transition video sequence created by a first method embodiment is shown;

[0033] Figure 5B Shows Figure 5AMultiple images in an enlarged scene, which have been digitally scaled to have the same aspect ratio;

[0034] Figure 5C shows Figure 5B some of the images and the target locations in;

[0035] Figure 6 shows the main steps of the cropping method in the form of a flowchart;

[0036] Figure 7 shows another embodiment of a method disclosed herein for achieving a smooth transition experience in a multi-camera system that includes at least one camera with scanning capabilities. Detailed Description

[0037] Figure 1 Schematically shows an embodiment of a mobile device (e.g., a smartphone) labeled 100, which includes a dual camera. The dual camera includes a scanning telephoto camera ("STC") 110 having a given native telephoto field of view (FOV T )(“n-FOV T ”) and an effective or scanned FOV T (“scanning telephoto field of view (s-FOV T )”). The scanning telephoto camera 110 can be a folding camera that includes an optical path folding element (OPFE) 118, a telephoto lens module 112 having a telephoto lens, and a telephoto image sensor 114. The telephoto lens can have a fixed effective focal length (EFL) that provides a fixed zoom factor, or an adjustable (variable) effective focal length that provides an adjustable zoom factor. The adaptation of the EFL can be discrete or continuous. The scanning telephoto camera (STC) 110 also includes a lens actuator 116 for moving the lens module 112, which is used for focusing and / or optical image stabilization (OIS); an optical path folding element actuator 122 for actuating the optical path folding element 118 for optical image stabilization and / or for scanning the n-FOV T into a specific point of view (POV) within the scanning telephoto field of view (s-FOV T ); and a first memory 124. The memory 124 can be an EEPROM (electrically erasable programmable read-only memory). In some embodiments, the first memory 124 can store first calibration data. The EFL of the scanning telephoto camera (STC) 110 is 8 millimeters to 50 millimeters or greater, the diagonal n-FOV T is 10 - 40 degrees, and the f-number is approximately f / # = 1.5 - 6.

[0038] n-FOVT Scanning of T is performed at a limited maximum speed, i.e., some setup time is required. N-FOV T Scanning of T can be performed on a time scale of about 1 - 30 milliseconds for scanning 2° - 5°, and on a time scale of about 10 - 80 milliseconds for scanning 10 - 25°. In some embodiments, scanning of the long focal length field of view (s-FOV T ) can cover about 50% of the area of the wide field of view (FOV W ). In some embodiments, scanning of the long focal length field of view can cover 100% or more of the area of the wide field of view.

[0039] In some embodiments, n-FOV T scanning can be performed by actuating two or more optical path folding elements instead of a single optical path folding element, as described in International Patent Application No. PCT / IB2021 / 059843 co-owned by the inventors of the present application.

[0040] The dual camera also includes a wide-angle camera ("WC") 130, which has a wide field of view greater than the n-FOV of the scanning long focal length camera (STC) 110 T . The wide-angle camera (WC) 130 includes a wide-angle lens module 132 having a wide-angle lens and a wide-angle image sensor 134. A second lens actuator 136 can move the lens module 132 for focusing and / or optical image stabilization. In some embodiments, second calibration data can be stored in a second memory 138. In other embodiments, the first calibration data and the second calibration data can be stored in a third memory 170. The first and second calibration data can include calibration data between the wide-angle camera (WC) 130 and the scanning long focal length camera (STC) 110. The EFL of the wide-angle camera (WC) may be 2.5 - 20 millimeters, the diagonal field of view is 50 - 130 degrees, and the f / # is about 1.0 - 2.5.

[0041] The mobile device 100 may also include an application processor (AP) 140. The application processor 140 may include a wide-angle image signal processor (ISP) 144 and a telephoto image signal processor 142. The application processor 140 may also include a camera controller 150 having a sensor control unit 152, a user control unit 160, a video processing unit 154, and a post-processing unit 146, all of which are operatively coupled to the image sensors 114 and 134. The user control unit 160 may include an operation mode selection function 162 for selecting whether to capture an image or video, a region of interest (ROI) function 164 for selecting a region of interest, the “target coordinates” of the region of interest or its specific viewing point (the “specific viewing point” and “target coordinates” are used interchangeably hereinafter), and a zoom factor (ZF) module 166 for selecting a zoom factor. The region of interest may be a section within the wide-angle field of view or the scanned telephoto field of view selected by the user or an algorithm. For the user, the region of interest may have a higher value than other segments, for example, because it contains a specific object or a specific combination of objects. Generally, the wide-angle camera (WC) focuses on one of the multiple regions of interest, and the scanned telephoto camera (STC) turns towards and / or focuses on one of the multiple regions of interest.

[0042] In use, the application processor 140 may receive respective wide-angle and scanned telephoto camera (STC) image data from the cameras 110 and 130 and provide camera control signals to the cameras 110 and 130.

[0043] The sensor control unit 152 is operatively connected to the two image signal processors (142 and 144) and the user control unit 160 and may be used to select which image sensor is operable according to the zoom factor and provide sensor control signals. The video processing unit 154 may be configured to evaluate a non-switching criterion to make a decision regarding the video output. Specifically, when evaluating the non-switching criterion, if the non-switching criterion is met, the module 154 may be configured to output a scaled video output image that includes only wide-angle image data during the zoom operation. The post-processing module 146 may be used for image processing, including denoising, sharpening, scaling, etc.

[0044] Here and hereinafter, unless otherwise stated, the following definitions will be used:

[0045] Input image: An image provided by the telephoto image signal processor 142 or the wide-angle image signal processor 144.

[0046] Scanned telephoto image (STI): An image provided by the telephoto image signal processor 142.

[0047] Wide-angle image (WI): An image provided by the wide-angle image signal processor 144.

[0048] Output image: The image shown to the user (usually a specific image of a video stream).

[0049] Telephoto output image: Based on the scanned telephoto image data and the output image provided in step 608 ( Figure 6 ) or step 708 ( Figure 7 ).

[0050] Wide - angle output image: According to the wide - angle image data and the output image provided in step 608 ( Figure 6 ) or step 708 ( Figure 7 ).

[0051] Output image data switching: In the image video stream, the action of switching from showing the telephoto output image or the wide - angle output image to the user to showing the wide - angle output image or the telephoto output image to the user respectively.

[0052] "Image data" and "data" or "image" are used interchangeably hereinafter.

[0053] Zooming in and out in the still - camera mode

[0054] Defined as follows: TFOV = tan(camera FOV / 2). "Low zoom factor (ZF)" refers to all zoom factors that satisfy ZF < TFOV W / TFOV T . "High zoom factor (ZF)" refers to all zoom factors that satisfy ZF > TFOV W / TFOV T . "Transfer zoom factor (Transfer ZF)" or "ZFT" refers to the zoom factor that satisfies ZF = TFOV W / TFOV T . ZFT represents the minimum zoom factor that allows output image data switching. In one embodiment, zooming in and out in the still mode can be performed as follows:

[0055] Zooming in: When the ZF is slightly higher than ZFT in the low ZF 向上 , the output image is a digitally scaled wide - angle output image. When the up - transfer ZF ( "ZF 向上 "), ZF 向上 ≥ZFT, when performing the switching of output image data from showing the wide - angle output image to showing the telephoto output image, move and correct the scanned telephoto image as described herein to achieve smooth transition (ST). For ZF ≥ ZF 向上 , the output is a digitally scaled telephoto output image.

[0056] Zooming out: When the ZF is high 向下When slightly above ZFT, the output image is a digitally zoomed telephoto output image. When down-transferring the zoom factor (down-transfer ZF, "ZF 向下 "), ZF 向下 ≥ZFT, when performing the switching of output image data from displaying the telephoto output image to displaying the wide-angle output image, move and correct the wide-angle image as described herein to achieve a smooth transition. For ZF ≤ ZF 向下 the output is a digitally zoomed wide-angle output image.

[0057] "Slightly above ZFT" means that the zoom factor is about 1% - 25% higher than ZFT.

[0058] In some examples, a global registration algorithm can be used to achieve a smooth transition.

[0059] In some embodiments, ZF 向上 and / or ZF 向下 can be a fixed zoom factor, while ZF 向上 ≤ ZF 向下 or ZF 向上 ≥ ZF 向下 .

[0060] In other embodiments, ZF 向上 and / or ZF 向下 may not be fixed zoom factors, but they may vary ("dynamic transfer ZF"), for example, depending on the viewpoint or target coordinates (i.e., the position of n-FOV T within s-FOV T ), i.e., ZF 向下 = ZF 向下 (x, y) and / or ZF 向上 = ZF 向上 (x, y), where "(x, y)" represents the coordinates or viewpoint within s-FOV T .

[0061] In some embodiments, in the central region of s-FOV T ZF 向下 may be less than that in the edge region of s-FOV T , i.e., ZF 向下 (central region) < ZF 向下 (edge region). The "central region" and "edge region" are defined as mutually exclusive. In one example, referring to the s-FOV Figure 2 shown in T 200, the central region can be defined by all viewpoints pointing within a rectangular box that has the same aspect ratio as s-FOV T200 with the same aspect ratio and center, but only 50% of the height and width of 200 respectively for the s-FOV. Any viewing point that points to 200 but not to the central region is included in the edge region. In other examples, the rectangular box may have only 30% of the height and width of 200, or may have up to 80% of the height and width of 200, and the definitions of the central region and the edge region will change accordingly. In a scanning telephoto camera (STC) based on a single optical path folding element, the scanned telephoto image captured in the central region of the s-FOV contains a larger field of view section in the object domain than the scanned telephoto image captured in the edge region. This is because in order to obtain a rectangular corrected scanned telephoto image with a specific aspect ratio from the unprocessed (i.e., uncorrected, uncropped, etc.) scanned telephoto image at the central position within the s-FOV, less cropping amount (i.e., less loss of scene information) is required compared to obtaining a rectangular corrected scanned telephoto image with the same specific aspect ratio from the unprocessed scanned telephoto image at the edge position within the s-FOV. The object domain is defined as the actual (or physical) scene, i.e., the scene captured by an ideal camera with a large enough field of view and without any aberrations and distortions. That is, the object domain corresponds to the appearance of the scene as it appears to a human observer. Therefore, the ZF in the central region and the ZF may be smaller than that in the edge region. T 200's height and width. Any viewing point that points to 200 but not to the central region is included in the edge region. T 200 but not pointing to the central region is included in the edge region. In other examples, the rectangular box may have only 30% of the height and width of 200, or may have up to 80% of the height and width of 200, and the definitions of the central region and the edge region will change accordingly. In a scanning telephoto camera (STC) based on a single optical path folding element, the scanned telephoto image captured in the central region of the s-FOV contains a larger field of view section in the object domain than the scanned telephoto image captured in the edge region. T 200's 30% height and width, or may have up to 80% of the height and width of 200, and the definitions of the central region and the edge region will change accordingly. In a scanning telephoto camera (STC) based on a single optical path folding element, the scanned telephoto image captured in the central region of the s-FOV contains a larger field of view section in the object domain than the scanned telephoto image captured in the edge region. T 200's 80% height and width, the definitions of the central region and the edge region will change accordingly. In a scanning telephoto camera (STC) based on a single optical path folding element, the scanned telephoto image captured in the central region of the s-FOV contains a larger field of view section in the object domain than the scanned telephoto image captured in the edge region. T The central region of the s-FOV contains a larger field of view section in the object domain than the scanned telephoto image captured in the edge region. This is because in order to obtain a rectangular corrected scanned telephoto image with a specific aspect ratio from the unprocessed (i.e., uncorrected, uncropped, etc.) scanned telephoto image at the central position within the s-FOV, less cropping amount (i.e., less loss of scene information) is required compared to obtaining a rectangular corrected scanned telephoto image with the same specific aspect ratio from the unprocessed scanned telephoto image at the edge position within the s-FOV. T The central position within the s-FOV to obtain a rectangular corrected scanned telephoto image with a specific aspect ratio, compared to obtaining a rectangular corrected scanned telephoto image with the same specific aspect ratio from the unprocessed scanned telephoto image at the edge position within the s-FOV, requires less cropping amount (i.e., less loss of scene information). T The edge position within the s-FOV to obtain a rectangular corrected scanned telephoto image with the same specific aspect ratio, less cropping amount (i.e., less loss of scene information) is required. 向上 and ZF 向下 can be smaller than that in the edge region.

[0062] The advantage of using dynamic transfer ZF is that at least in the central region of the s-FOV, the image quality advantage of using a scanning telephoto camera (STC) can be enjoyed at a lower ZF. In the still photography mode, using dynamic up and down conversion zoom factors may be particularly beneficial. In some embodiments with fixed or dynamic ZF T and / or ZF 向上 and / or ZF 向下 In some embodiments, during the video photography mode (i.e., when the user captures a video stream), the value of ZF 向上 and / or ZF 向下 may be 5% - 30% larger than the value of ZF 向上 and / or ZF 向下 in the still photography mode. In some embodiments, the value of ZF 向上 and / or ZF 向下 may depend on the aspect ratio of the wide-angle output image and / or the telephoto output image.

[0063] In some embodiments, where ZF 向上 = ZF 向上(x, y), ZF 向上 (central region) <ZF 向上 (peripheral region).

[0064] In some embodiments, ZF 向上 can be fixed, while ZF 向下 can be dynamic, i.e., ZF 向下 = ZF 向下 (x, y). Fixed ZF 向上 can satisfy ZF 向上 ≥ [ZF 向下 (x, y)] 最大 , i.e., fixed ZF 向上 is defined by the maximum value of ZF 向下 (x, y). When considering all viewpoints within the s-FOV T , the maximum value of ZF 向下 (x, y) is the maximum value of ZF 向下 . This can be beneficial because it can prevent the need to perform a switch of output image data from multiple scanned telephoto images to multiple wide-angle images even when the zoom factor does not change. An example of such an undesirable situation involves switching the output image data from multiple wide-angle images to a scanned telephoto image at the central position at the target coordinates (where ZF 向上 can be less than the peripheral region) and then using a scanned telephoto camera (STC) to track a moving object. The object (i.e., the target coordinates) may now move to a more peripheral position within the s-FOV T , where the n-FOV T of this peripheral position is not large enough, so the field of view defined by the corresponding ZF can no longer be supported by multiple scanned telephoto images. Due to the incomplete scanned telephoto image data, a switch of output image data from multiple scanned telephoto images to multiple wide-angle images needs to be performed even though ZF has not changed. Using a fixed ZF 向上 to satisfy ZF 向上 ≥ [ZF 向下 (x, y)] 最大 can be particularly useful in video mode.

[0065] Smooth transition

[0066] For smooth transition, matching of the position, scale, brightness, and color of the output image can be performed before and / or after switching the output image data. However, in many cases, image position matching between the entire wide-angle image and the scanned telephoto image is impossible, for example because of parallax. Thus, in the smooth transition as disclosed herein, position matching can be achieved only in the region of the region of interest, while scale, brightness, and color are matched for the entire output image region. Specifically, when switching from displaying multiple wide-angle images to displaying multiple scanned telephoto images, the multiple scanned telephoto images can be moved according to the distance of an object in the region of interest of the scanned telephoto images relative to the multiple wide-angle images, and / or when switching from displaying scanned telephoto images to displaying wide-angle images, by moving the multiple wide-angle images according to the distance of an object in the region of interest of the wide-angle images relative to the multiple scanned telephoto images.

[0067] Figure 2 Shows different optical path folding element positions in the object domain and their respective n-FOVs T , as described in more detail in International Patent Application No. PCT / IB2021 / 056311. The object domain is different from the "image domain", which is defined as the scene with aberration and distortion captured by the camera. In the present invention, the image domain is defined as the scene captured by a scanning telephoto camera (STC) such as camera 110. Box 200 represents the s-FOV T , i.e., the smallest rectangular field of view that contains all the scanning telephoto camera (STC) image data in all viewpoints and can be achieved using a specific scanning telephoto camera (STC) in the target domain. The n-FOVs T for three different optical path folding element positions (0, 1, and 2) are represented by 202-0, 202-1, and 202-2. The n-FOV T 202-0 of the optical path folding element "zero position" is defined as the n-FOV that produces an object or scene image without viewpoint aberration T . That is, at the zero position, the object in the object domain is the same as the object image in the image domain (except for scaling, image sensor noise, etc.). Generally speaking, in the zero position, the n-FOV T overlaps with the center of the wide-angle field of view. As shown, the n-FOVs T at any other position (such as 202-1 and 202-2) are not horizontal rectangles (such as 202-0), but arbitrary quadrilaterals. The same rectangular object is represented by 204-0, 204-1, and 204-2 in the n-FOVs T 202-0, 202-1, and 202-2 respectively.

[0068] Figure 3shows the main steps of the method for locating a scanned telephoto image within a wide-angle image; a scanned telephoto image is captured in step 302. Before capture, the n-FOV T is scanned towards a specific viewpoint. As described in International Patent Application No. PCT / IB2021 / 056311, in step 304, the scanned telephoto image is corrected. By using appropriate cropping selection criteria, the content included in the corrected and trimmed scanned telephoto image, as well as the positions of multiple objects in the corrected and trimmed scanned telephoto image, can be manipulated. The cropping selection criteria refer to the specific criteria for using a specific scanned telephoto image captured at a specific n-FOV T to correct and crop the specific scanned telephoto image. Cropping includes defining a cropping offset and / or a cropping center and / or a cropping coefficient. The cropping offset is the position of the upper left corner of the image in the object domain. The cropping coefficient is usually given by 1 / ZF.

[0069] In some examples, the cropping can be modified so that the selected object is included in the cropped scanned telephoto image.

[0070] In other examples, the cropping can be modified so that the selected object is located at a specific position in the cropped scanned telephoto image.

[0071] In some examples, for video mode, the cropping can be modified so that the change in the position of the selected object between two consecutive frames of the video stream does not exceed 10%, 5%, or 1% of the width and / or height of the telephoto sensor.

[0072] In other examples, the position of the selected object between two consecutive frames can vary by less than 50 pixels or less than 20 pixels, or even less than 5 pixels (assuming a pixel size of 1 micron).

[0073] In yet another example, the cropping can be modified so that the position of the selected object in consecutive frames of the video stream changes only slowly, for example, at a frequency of no higher than 10 pixels / second or 5 pixels / second or even 2 pixels / second (assuming a pixel size of 1 micron).

[0074] In other examples, the cropping can be modified so that the position of the selected object meets aesthetic criteria. The aesthetic criteria can be based on, for example, the position of the selected object in the cropped scanned telephoto image (aesthetic frame), such as the rule of thirds for photographers, whether additional objects are included or excluded in the cropped scanned telephoto image, such as the leading line method for photographers, etc. For example, as Figure 4A-B As described, in step 306, the positioning of the scanned telephoto image within the wide-angle image is performed. In step 308, image registration known in the art is performed between the cropped scanned telephoto image and the wide-angle image. The output image data switching can be performed in step 310, as outlined in more detail below. The same steps can be performed for the zoom-out case, where the output image data can be switched from the telephoto output image data to the wide-angle output image data. In some examples, the switching step may include the mixing or fusing of multiple scanned telephoto images and multiple wide-angle images.

[0075] Figure 4A -B shows the details of the positioning of the scanned telephoto image within the wide-angle image performed in step 306, as described in more detail in International Patent Application No. PCT / IB2020 / 061461 jointly owned by the inventors of the present application.

[0076] In Figure 4A an estimated viewpoint within the wide-angle field of view 404 is shown, where an estimate 402 of the n-FOV T is shown. The estimated viewpoint refers to the viewpoint estimated based on calibration data. The estimate of the n-FOV T depends on calibration and may not be accurate enough in terms of matching multiple scanned telephoto images with multiple wide-angle images. Typically, before positioning, the image point coordinates of the same object point in the scanned telephoto image and the wide-angle image may deviate by more than 25, 50, or even 100 pixels. Assume that the size of one pixel is approximately 1 micron. Positioning (step 306) is performed to reduce this deviation, i.e., to achieve a more accurate and precise n-FOV T estimate. The positioning includes:

[0077] 1. Select a search area 406a as shown in Figure 4A . The search area 406 can be defined by using the center of the n-FOV T estimate and (e.g., symmetrically) embedding it in a rectangular area, where the area of the rectangle can be two, three, or four times the area covered by the n-FOV T estimate.

[0078] 2. Crop the search area from the wide-angle image.

[0079] 3. Template matching, where the source is represented by the cropped search area from the wide-angle image, and the template can be represented by the scanned telephoto image. The template matching can be performed by correlating the template at different positions or over the entire search area. The position with the highest matching value can indicate the best estimate of the n-FOV T position within the wide-angle field of view. This more accurate and precise n-FOV T estimate is in Figure 4BIt is represented as 408 in the figure and can be further used to execute steps 308 and 310. Generally, after positioning, the deviation of the image points of the same object point between the wide-angle image and the scanned telephoto image may be less than 5 pixels, or even less than 2 pixels, that is, the accuracy is increased by 5, 10, or even 20 times.

[0080] Figure 5A -C shows a smooth transition video sequence created by the first cropping method ("Method 1") disclosed herein. Hereinafter, the index "0" represents the initial state, the index "f" represents the final (or target) state, and the index "t" represents some intermediate state. "State" refers to the image defined by ZF, the cropping coefficient, and the cropping offset. The "final state" refers to the last image shown in the smooth transition video sequence, which contains the wide-angle output image data. When further magnified, the output image data switching can be performed.

[0081] Figure 5A Shows how the magnified scene is displayed in the smooth transition video sequence created by Method 1. Image 502 can represent the first image in the smooth transition video sequence. Image 502 may have ZF = 1.0, that is, it contains the full wide-angle field of view, or it may have ZF>1. Images 504 - 512 represent subsequent images in the smooth transition video sequence towards the "target" position 514, with coordinates (xf, xf) and the target ZF ("ZFf"). The target position 514 is the position towards which the magnification is directed. The n-FOV of the scanned telephoto camera (STC) T is scanned such that the target position 514 is located at the center of the n-FOV T The position 514 and ZFf can be selected by the user or the program before or at the start of the magnification action. Images 504 - 512 are wide-angle output images. The cropping offsets of images 504 - 512 are represented by image frames 516 - 528 respectively. The advantage of Method 1 is that the cropping is selected such that the target position 514 is included in each of images 502 - 512, that is, the target position 514 is shown throughout the smooth transition video sequence.

[0082] Figure 5B Shows images 504 - 512 subsequently displayed to the user in the magnified smooth transition video sequence, that is, they are digitally scaled to have the same aspect ratio.

[0083] Figure 5CDisplays image 502, image 512, and target position 514. Reference numeral 532 represents the initial X coordinate X0 of target position 514 in the coordinate system of image 502, and the number 534 represents the target X coordinate Xf of target position 514 in the coordinate system of image 512. The Y coordinates Y0 and Yf are defined correspondingly. The target position 514 is defined in the object domain. The widths and heights of images 502 and 512 (measured along the x-axis and y-axis respectively) are given by W502 and W512, H502 and H512. The relative positions (“relPos 0， x”) and (“relPosf ， x”) can be derived from 532 and 534 by relPos 0， x = X0 / W502 and relPosf ， x = xf / W512. The relative positions in the Y direction can be defined correspondingly. By gradually modifying the relative coordinates of the target position (in the X-axis and Y-axis), a smooth transition can be achieved. At the target ZFf, the relative position of target position 514 is relPosf ， x = relPosf ， y = 1 / 2, that is, the target position 514 coincides with the center of the output image.

[0084] Method 1 can be similarly used for reduction.

[0085] Figure 6 Shows the main steps of Method 1. It can start from the initial (or “0”) state, use ZF0 < ZFT and display a video stream of a wide-angle output image (such as images 502 - 512). The positioning within the wide-angle field of view of n-FOV T (for example Figure 4A - as described in - B) is performed.

[0086] Generally, the relative position at some intermediate state (“t”) can be calculated according to relPost = (Xt / Wt, Yt / Ht), where Wt and Ht are the width and height of the output image displayed at the intermediate state t respectively, where Wt = W / ZFt and Ht = H / ZFt, to define the cropping factor, where W and H are the width and height of the untrimmed wide-angle image respectively.

[0087] The initial state relative position relPos0 is calculated from the target coordinates (X0, Y0) in the coordinate system of image 502 as relPos0 = (X0 / W0, Y0 / H0), where W0 = W / ZF0 and H0 = H / ZF0 are the width and height of the output image displayed in the initial state 0 respectively. The conversion slope S from relPos0 to the relative position in the target state relPosf is derived from the following formula:

[0088]

[0089] where ZF P is a preset and fixed ZF value that satisfies ZF P ≥ ZFT.

[0090] In step 602, the program or user uses a ZF (ZFt1) to trigger a command to zoom to a first intermediate state ("t1"), where ZFt1 ≥ ZFT, enabling the execution of output image data switching (from wide-angle image data to scanned telecamera (STC) image data). The initial output image can be an image showing the entire wide-angle field of view (i.e., having ZF0 = 1), or an image having 1 < ZF0 < ZFT that shows a (digitally zoomed) section of the wide-angle field of view.

[0091] In step 604, the relative position update is calculated according to relPost1 = relPos0 + (ZFt1 - ZF0). Generally, the relative position update can be relPost = relPost -1 + (ZFt - ZFt -1 ) · S.

[0092] In step 606, the cropping factor and cropping offset are calculated. The cropping offset (X C , Y C ) is calculated by first calculating the relative coordinates as follows: Xt1 = relPost1,x · Wt1 and Yt1 = relPost1,y · Ht1, where relPost ， x and relPost ， y are the x value and y value of relPost respectively. The position of the cropping offset (X C , Y C ) is calculated by subtracting Xt1 and Yt1 from the target coordinates (in the coordinate system of image t1).

[0093] In step 608, an image based on the wide-angle image data and cropped according to the cropping offset (X C , Y C ) and having an image size (Wt, Ht) is output and displayed in a smoothly transitioning video sequence. If further zooming is performed, the process can start again from step 602.

[0094] In some examples, the following output image data switching can be from using only wide-angle image input to using only scanned telephoto image input.

[0095] In the second cropping method ("Method 2"), smooth transition can be achieved by linearly modifying the cropping offset. One such method can consider the initial cropping offset (X C，0 , Y C，0) and the linear connection between the target cropping offset (X C， f, Y C， f). In Method 2, when zooming in or out, the cropping offset is always located on this linear connection, as shown in Figure 5A . In some examples, the position of the cropping offset on the linear connection is determined linearly. In other examples, the position is determined by the square law. For example, when zooming from ZF = 1 to ZF = 2, the difference in the cropping offset position (i.e., the distance between the respective cropping offsets) on the linear connection is 2 times greater than when zooming from ZF = 2 to ZF = 4.

[0096] In a third cropping method (“Method 3”), smooth transition can be achieved by linearly modifying the center of the field of view shown in the smooth transition video stream. One possibility is to consider the linear connection between the initial center of the field of view and the target center of the field of view. In Method 3, when zooming in or out, the center of the field of view may always be located on this linear connection. The position of the cropping offset on the linear connection can be determined by the linear law or the square law.

[0097] Figure 7 An embodiment of the smooth transition method disclosed herein is shown. For each image of the video stream, in step 702, the wide-angle input image, the telephoto input image, and the viewpoint of the scanning telephoto camera (STC) are retrieved. The telephoto input image can be pre-corrected to align with the wide-angle viewpoint or corrected in step 702. Optionally, in step 702, calibration data can be used to convert the OFPE position to an n-FOV T estimation. For example, in the case of changing the n-FOV T by scanning, a registration procedure can be performed to calculate the translation and rotation parameters of the current scanned telephoto image relative to the wide-angle image. The translation can compensate for parallax, and the rotation can compensate for the residual rotation between the scanned telephoto image and the wide-angle image, which may be caused by imperfect calibration.

[0098] In some examples, the multi-camera may include image stabilization (IS), such as optical image stabilization (OIS) or electronic image stabilization (EIS) for a scanning tele-camera (STC) and / or a wide-angle camera (WC). Since image stabilization may shift the wide-angle image and the scanning tele-image relative to each other, in step 702, image stabilization control input data may be read, such as sensor data provided from an inertial measurement unit (IMU), or image stabilization control output data, such as a command to move a particular optical component by a given amount. The data may be converted to an expected pixel offset in the wide-angle image and the scanning tele-image, and this conversion may be used to compensate for any undesirable effects that the image stabilization mechanism may have on the smooth transition of the video sequence. As is well known, for electronic image stabilization, the input image is cropped and shifted so that in two or more consecutive output images of the video stream, a selected object or the entire scene remains located at the same (or similar) particular position. To prevent jumps between the scanning tele-camera (STC) and the wide-angle camera (WC), the ZF 向上 and / or ZF 向下 value in the digital image stabilization video mode may be 2.5% - 50% larger than the ZF 向上 and / or ZF 向下 value in the non-image stabilization video mode. Additionally, the ability to perform electronic image stabilization (e.g., measured by the maximum number of pixels that can be moved) may be 2.5% - 50% greater at the center viewing point than at the edge viewing point. Here, image stabilization can be used to achieve two different goals. The first goal may be to stabilize the wide-angle camera (WC) or the scanning tele-camera (STC) at approximately 50 - 100 Hz and higher frequencies, for example, to mitigate the user's hand shake. The first goal may be to stably construct the scene, i.e., to stabilize the wide-angle camera (WC) or the scanning tele-camera (STC) at approximately 20 Hz and lower frequencies (e.g., 1 Hz) to maintain a selected scene (of the object domain) within the wide-angle field of view or n-FOV T or at a particular position within the wide-angle field of view or n-FOV T . In step 704, a wide-angle input image or a tele-input image is selected as the output image. The wide-angle input image data may be selected for the following cases:

[0099] for any low ZF, or

[0100] for high ZF, but meeting one or more non-switching criteria. Evaluating the non-switching criteria avoids switching the output image data in situations where a smooth transition cannot be achieved or may be disadvantageous. After capturing a scanning tele-image, the following non-switching criteria may be evaluated:

[0101] 1. Low-scan long-focus image quality. For example, due to defocus (blur) of a scanning long-focus camera (STC), or due to motion blur, jelly effect artifacts, or other artifacts known in the art, low image quality may be caused by large electronic noise from low scene illumination.

[0102] 2. Inappropriate composition of the scanning long-focus image. For example, this may include the following scenarios: (i) the scene shown in the scanning long-focus image is significantly different from the scene shown in the wide-angle image, and in a smoothly transitioning video stream, it is output to the user before the scanning long-focus image, or (ii) the scanning long-focus image shows a scene that is semantically significantly inconsistent with the scene shown in the wide-angle image (e.g., motion occurring between the captured scanning long-focus image and the wide-angle image), or (iii) the scanning long-focus image does not show the region of interest (e.g., an obstacle) included in the wide-angle image, or it does show it at a significantly incorrect location relative to the region of interest in the wide-angle image (i.e., the target coordinates in the wide-angle camera (WC) coordinate system are significantly different from the target coordinates in the scanning long-focus camera (STC) coordinate system).

[0103] 3. Imperfect correction of the scanning long-focus image, such as imperfect rolling correction of the scanning long-focus image.

[0104] Before capturing the scanning long-focus image or scanning to specific target coordinates for the n-FOV T the following non-switching criteria can be evaluated:

[0105] 1. Effective scanning range of the scanning long-focus camera (STC): Evaluate whether the target coordinates are included in the s-FOV T or not.

[0106] 2. Preventing jumps between the scanning long-focus camera (STC) and the wide-angle camera (WC): Evaluate whether the target coordinates are close to certain thresholds at the edge of the s-FOV T . In this case, even a small relative movement of the region of interest and the scanning long-focus camera (STC) may result in a situation where the region of interest can no longer be completely covered by the scanning long-focus image, so the wide-angle image must be used. To prevent jumps between the scanning long-focus camera (STC) and the wide-angle camera (WC), if the target coordinates are located at a distance of 5% or 10% or 20% (in the horizontal or vertical direction) from the s-FoV T , the output data of the scanning long-focus image cannot be switched to.

[0107] 3. Effective speed of the region of interest: Evaluate the speed of the region of interest (and the corresponding target coordinates) to analyze whether the n-FOV T can follow this movement.

[0108] In step 706, the input image selected as the output image is cropped. For example, based on the selected input image, the current ZF, the image cropping required for the input image, etc. are calculated. The calculation can be performed according to cropping method 1, cropping method 2, or cropping method 3. The generated cropped image can be displayed (or output) to the user.

[0109] Optionally, and depending on, for example, a) the selected input image, b) the cropping of the input image, and / or c) the image registration parameters, a smoothly transformed output image can be provided in step 708.

[0110] In other examples, the methods disclosed herein may not include the action of cropping, but may only provide the parameters calculated as described herein, and the actions of steps 706 and 708 (such as cropping, providing a video, etc.) may be performed on different programs or processors, for example, on dedicated hardware (HW) that supports hardware acceleration.

[0111] To compensate for the resolution difference between multiple scanned telephoto images and multiple wide-angle images, blurring can be applied to the output image.

[0112] Rotation correction can also be applied in step 708. The lens distortion difference between the wide-angle camera (WC) and the scanned telephoto camera (STC) is another challenge. For smooth transformation, distortion correction can be applied to all input images, or alternatively, digital distortion can be applied only to the telephoto input images or only to the wide-angle input images so that they match the corresponding other input images.

[0113] Unless otherwise specified, the expression "and / or" used between the last two members of an option list for selection indicates that it is appropriate to select one or more of the listed options and selection can be made.

[0114] It should be understood that when a claim or specification refers to "a (a, an)" element, such reference should not be construed as meaning only one of a plurality of elements.

[0115] All patents, patent applications, and publications mentioned in this specification are hereby incorporated by reference in their entirety to the same extent as if each individual patent, patent application, or publication was specifically and individually indicated to be incorporated by reference into this specification. Additionally, any reference document cited or identified in this application should not be construed as an admission that such reference document can be used as prior art for this disclosure.

Claims

1. A mobile device, characterized in that, The mobile device includes: a processor operable to receive a first video image stream of a plurality of first images having a first field of view FOV1 and a first resolution, and a second video image stream of a plurality of second images having a second field of view FOV2 and a second resolution different from the first resolution, wherein FOV2 < FOV1, and a camera controller configured to receive a position within the first field of view FOV1 selected by a user of the mobile device or by a program running on the processor, wherein the camera controller is configured to stream a video image stream that shows a continuous zoom-in action towards a non-central area including the position within the first field of view FOV1 or a continuous zoom-out action away from the non-central area, and wherein when switching from displaying the plurality of first images to displaying the plurality of second images or vice versa, the plurality of video images have a smooth transition.

2. The mobile device according to claim 1, wherein, The plurality of first images undergo a first image signal processing, and the plurality of second images undergo a second image signal processing.

3. The mobile device according to claim 1, characterized in that, The position is selected by the program, and the selection of the position is made according to aesthetic criteria.

4. The mobile device according to claim 1, characterized in that, The smooth transition is achieved by performing a rotation correction of the plurality of second images.

5. The mobile device according to claim 1, characterized in that, The smooth transition is achieved by performing a registration between the plurality of first images and the plurality of second images.

6. The mobile device according to claim 1, characterized in that, The smooth transition is achieved by performing positioning between the plurality of first images and the plurality of second images to perform position matching.

7. The mobile device according to claim 6, characterized in that, Relative to the position matching before the positioning, the positioning between the plurality of first images and the plurality of second images improves the accuracy of the position matching by more than 2.5 times.

8. The mobile device according to claim 6, characterized in that, Relative to the position matching before the positioning, the positioning between the plurality of first images and the plurality of second images improves the accuracy of the position matching by more than 10 times.

9. The mobile device according to claim 1, characterized in that, When performing the switch from displaying the plurality of first images to displaying the plurality of second images, the smooth transition is achieved by moving the plurality of second images relative to the plurality of first images according to a distance of an object in a region of interest ROI of a second image, and / or when performing the switch from displaying the plurality of second images to displaying the plurality of first images, moving the plurality of first images relative to the plurality of second images according to a distance of an object in a region of interest ROI of a first image.

10. The mobile device according to claim 1, characterized in that, The smooth transition is achieved by applying blur to the plurality of first images and / or the plurality of second images to compensate for the resolution difference between the plurality of first images and the second images.

11. The mobile device according to claim 1, characterized in that, The smooth transition is achieved by blending the plurality of first images and the plurality of second images.

12. The mobile device according to claim 1, characterized in that, The smooth transition is achieved by matching the scale and / or brightness and / or color between the plurality of first images and the plurality of second images.

13. The mobile device according to claim 1, characterized in that, The smooth transition is achieved by cropping the plurality of first images such that a specific target area is always displayed in the video image stream.

14. The mobile device according to claim 1, characterized in that, The switching from displaying a plurality of first images to displaying a plurality of second images is performed with an upward shift of the ZF value ZF 向上 and the switching from displaying a plurality of second images to displaying a plurality of first images is performed with a downward shift of the ZF value ZF 向下 where ZF 向上 ≥ZF 向下 .

15. The mobile device according to claim 1, characterized in that, The switching from displaying a plurality of second images to displaying a plurality of first images is performed depending on a downward transfer ZF value ZF of a viewpoint POV within the second field of view FOV2 in the first field of view FOV1. 向下 is performed.

16. The mobile device according to claim 1, characterized in that, The switching from displaying a plurality of first images to displaying a plurality of second images is performed at an upward shift of the ZF value ZF 向上 and the switching from displaying a plurality of second images to displaying a plurality of first images is performed at a downward shift of the ZF value ZF 向下 wherein the ZF 向上 and / or the ZF 向下 value in a video shooting mode is 5% to 30% greater than the ZF 向上 and / or the ZF 向下 value in a still image shooting mode.

17. The mobile device according to claim 1, characterized in that, The switching from displaying a plurality of first images to displaying a plurality of second images is performed at an upward shift of the ZF value ZF 向上 and the switching from displaying a plurality of second images to displaying a plurality of first images is performed at a downward shift of the ZF value ZF 向下 wherein the ZF 向上 and / or ZF 向下 value in a digital image stabilization video mode is 5% to 30% greater than the ZF 向上 and / or ZF 向下 value in a non-image stabilization video mode.

18. The mobile device according to claim 1, characterized in that, In a digital image stabilization video mode, an image stabilization ability at a central point of view (POV) is 5% to 30% greater than an image stabilization ability at a peripheral POV.

19. The mobile device according to any one of claims 1 to 18, characterized in that, The mobile device is a smart phone.

Images