SMOOTH CONTINUOUS ZOOMING IN A MULTI-CAMERA SYSTEM THROUGH IMAGE-BASED VISUAL FEATURES AND OPTIMIZED GEOMETRIC CALIBRATIONS

Abstract

An example method includes displaying an initial preview of a scene captured by a first camera operating within a first focal length range. The method includes detecting a zoom event predicted to cause the first camera to reach a boundary of the first range. The method includes activating a second camera operating within a second focal length range to capture a zoomed preview of the scene. The method includes updating a geometry-based warp transform based on a comparison of respective image features from the initial preview and the zoomed preview. The method includes aligning the zoomed preview with the initial preview by applying the updated warp transform.The method includes displaying the aligned magnified preview of the image captured by the second camera while working in the second area.

Description

[0001] This application claims priority over Provisional Patent Application No. 63 / 377,581, filed September 29, 2022, which is hereby incorporated by reference. GENERAL STATE OF THE ART

[0002] Many modern computing devices, including mobile phones, PCs, and tablets, incorporate image capture devices. Some image capture devices are configured with multi-camera systems. The camera systems are configured to use their respective specifications to collectively meet various image capture requirements. A smartphone can integrate multiple camera types with different focal lengths to capture objects at different distances and scenes in different fields of view (FOV). SUMMARY

[0003] The present disclosure generally relates to smooth transitioning between multiple cameras. In one aspect, an image capture device may include multiple cameras. Transitioning between cameras may result in perceptible distortions of the image, such as binocular disparity, due, for example, to a change in field of view. As described herein, a distortion transform is estimated from available geometric metadata as well as image-based data to distort the image of one camera to nearly align with the image of the other camera, thereby reducing the perceptible image distortions upon camera transition.

[0004] In a first aspect, a computer-implemented method is provided. The method includes displaying, through a screen of a computing device, an initial preview of a scene captured by a first image capture device of the computing device, wherein the first image capture device operates within a first focal length range. The method further includes detecting, by the computing device, a zoom operation that is expected to cause the first image capture device to reach a limit of the first focal length range. The method further includes, in response to the detecting, activating a second image capture device of the computing device to capture a magnified preview of the scene, wherein the second image capture device is configured to operate within a second focal length range.The method additionally includes updating a geometry-based distortion transformation based on a comparison of the respective image features from the initial preview and the magnified preview. The method further includes aligning the magnified preview with the initial preview by applying the updated distortion transformation, wherein the updated distortion transformation reduces one or more viewing artifacts caused by a change in a field of view when transitioning from the initial preview to the magnified preview. The method further includes displaying the aligned magnified preview of the image captured by the second image capture device while operating within the second focal length range through the screen of the computing device.

[0005] In a second aspect, a computing device is provided. The computing device includes a display, a first image capture device configured to operate within a first focal length range, a second image capture device configured to operate within a second focal length range, one or more processors, and a data storage, wherein the data storage stores computer-executable instructions that, when executed by the one or more processors, cause the mobile device to perform functions. The operations include displaying an initial preview of a scene captured by the first image capture device on the display; detecting, by the computing device, a zoom operation that is likely to cause the first image capture device to reach a limit of the first focal length range;Activating the second image capture device in response to the detection to capture a zoomed preview of the scene; updating a geometry-based distortion transformation based on a comparison of respective image features from the initial preview and the zoomed preview; aligning the zoomed preview with the initial preview by applying the updated distortion transformation, wherein the updated distortion transformation reduces one or more viewing artifacts caused by a change in a field of view when transitioning from the initial preview to the zoomed preview; and displaying the aligned zoomed preview of the image captured by the second image capture device while operating within the second focal length range through the display screen of the computing device.

[0006] In a third aspect, an article of manufacture is provided. The article of manufacture may include a non-transitory, computer-readable medium having stored thereon program instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations. The operations include displaying an initial preview of a scene captured by the first image capture device on the screen; detecting, by the computing device, a zoom operation that is likely to cause the first image capture device to reach a limit of the first focal length range; activating, in response to the detection, the second image capture device to capture a zoomed preview of the scene; updating a geometry-based warp transform based on a comparison of respective image features from the initial preview and the zoomed preview;Aligning the zoomed preview with the initial preview by applying the updated distortion transformation, wherein the updated distortion transformation reduces one or more viewing artifacts caused by a change in a field of view when transitioning from the initial preview to the enlarged preview; and displaying the aligned enlarged preview of the image captured by the second image capture device while operating within the second focal length range through the display screen of the computing device;

[0007] In a fourth aspect, a system is provided. The system includes means for displaying an initial preview of a scene captured by the first image capture device on the screen; means for detecting, by the computing device, a zoom operation that is likely to cause the first image capture device to reach a limit of the first focal length range; means for activating the second image capture device in response to the detection to capture a zoomed preview of the scene; means for updating a geometry-based distortion transform based on a comparison of respective image features from the initial preview and the zoomed preview;Means for aligning the zoomed preview with the initial preview by applying the updated distortion transformation, wherein the updated distortion transformation reduces one or more viewing artifacts caused by a change in a field of view when transitioning from the initial preview to the magnified preview; and means for displaying the aligned magnified preview of the image captured by the second image capture device while operating within the second focal length range through the display screen of the computing device;

[0008] Other aspects, embodiments and implementations will become apparent to those skilled in the art upon reading the following detailed description, with reference, where appropriate, to the accompanying drawings. SHORT DESCRIPTION OF THE CHARACTERS Fig.1 illustrates binocular disparity in a multi-camera system according to the exemplary embodiments. Fig. 2 is a flowchart of a workflow for image-based computation of a distortion transform according to the example embodiments. Fig. 3 is an example of a sparse feature workflow for smooth continuous zooming in a multi-camera system according to the example embodiments. Fig. 4A illustrates temporal feature tuning and tracking according to the example embodiments. Fig. 4B illustrates example images for temporal feature matching and tracking according to example embodiments. Fig. 4C illustrates an example application 400 for temporal feature matching and tracking according to example embodiments. Fig.5 is an example of a dense feature workflow for smooth continuous zooming in a multi-camera system according to the example embodiments. Fig. 6 illustrates exemplary handling of delta data during camera transitions according to the exemplary embodiments. Fig. 7 is a table illustrating various cases for switching between a telephoto camera and a wide-angle camera according to the exemplary embodiments. Fig. 8A depicts an example geometric relationship at each pair of matching pixels according to the example embodiments. Fig. 8B depicts a workflow for determining a geometric relationship at each pair of matching pixels, according to example embodiments. Fig.9 depicts an example workflow for smooth continuous zooming in a multi-camera system according to the example embodiments. Fig. 10 illustrates a distributed computing architecture, according to example embodiments. Fig. 11 is a block diagram of a computing device according to example embodiments. Fig. 12 is a flowchart of a method, according to example embodiments. DETAILED DESCRIPTION

[0009] Example methods, apparatus, and systems are described herein. It should be understood that the terms "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." An embodiment or feature described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein.

[0010] Therefore, the exemplary embodiments described herein are not to be considered limiting. Aspects of the present disclosure, as generally described herein and illustrated in the figures, may be implemented, substituted, combined, separated, and configured in a wide variety of different configurations, all of which are contemplated herein.

[0011] Furthermore, unless the context suggests otherwise, the features illustrated in the individual figures may be used in combination with one another. Therefore, the figures should generally be considered component aspects of one or more overall embodiments, with the understanding that not all illustrated features are required for every implementation. Overview

[0012] A smartphone or other mobile device that supports image and / or video capture may be equipped with multiple cameras, each using different specifications to collectively meet different image capture needs. A smartphone may integrate multiple camera types with different focal lengths to display and / or capture objects at different distances and scenes in different fields of view (FOV).

[0013] For example, a phone might be configured with a medium-focus main camera to meet normal photo / video capture needs, a longer-focal-length telescopic camera to capture distant objects, and an ultra-wide-angle camera with a shorter focal length to capture larger FOVs. During photo / video capture, the main camera might switch to the telescopic camera when a user zooms in further to focus on a distant object, and the main camera might switch to the ultra-wide-angle camera when the user zooms out further to capture a wider field of view. Multi-camera systems provide a much wider range of focus distances than a single camera. However, an abrupt camera change while zooming can cause a viewing deviation (known as binocular disparity).

[0014] To circumvent binocular disparity, a distortion transformation can be estimated from the available geometric metadata and image features to distort the image from one camera so that it is nearly aligned with the image from the other camera, making changes during camera changes less noticeable. Distortion transformations can involve scaling, rotation, reflection, identity mapping, shearing, or various combinations thereof. Furthermore, translations, similarities, affine mappings, and / or projective mappings can be used as distortion transformations. In general, two planar images can be related to each other using a distortion transformation, such as a homography. For example, a computer vision approach can be used to compute a homography, which can be used to distort the image from one camera to another.As described here, a homographic calculation can be determined to reduce the variation in view when changing cameras during zooming. The homography calculation can use geometric information (excluding image features), including metadata such as camera calibration data, focus distance, and so on. Although a geometry-based solution can be used, the presence of electrical and / or mechanical parts, such as voice coil motors (VCMs), optical image stabilization (OIS) adjustment, and / or thermal effects of a device can result in dynamic calibrations of the camera and changes in focus distances, which can lead to errors in determining a precise distortion transformation for a smooth viewing experience, creating abrupt transitions.

[0015] Some existing approaches attempt to address this problem. For example, views from physical cameras may be distorted to the same coordinate, and the distortion transformation may depend on camera calibration and focal length. However, image-based features are not used, so errors resulting from VCM / OIS adjustment and / or thermal effects may not be corrected. Another approach could be to blend multiple camera views and apply a fade-style animation to achieve smooth switching. However, this approach assumes that images from different cameras are displayed simultaneously. A theoretical model for using binocular disparity and motion parallax for depth estimation has been proposed, but it lacks practical implementations to address the technical issues associated with image acquisition devices.

[0016] This application refers to an "image-based" approach (sometimes referred to here as ContiZoom) to better support distortion quality and overcome the adverse effects of VCM / OIS adjustment and / or thermal effects. In contrast to the "geometry-based" approach, the new "image-based" approach is designed to utilize image information and / or features as additional input to enhance the distortion transformation used in the geometry-based approach.

[0017] The approach described here directly uses image features to provide a more precise metric for calculating the distortion transformation. This reduces spatial differences between the two cameras' frames and mitigates the impact of many inaccurate sensor metadata from the geometry-based approach.

[0018] From a geometric perspective, thermal changes in the fixture affect the principal points, which can cause the entire FOV to shift, which in turn causes the camera parameter interpolation (CPI) output to be unreliable. Thermal changes affect the focusing distance, and therefore, the already inaccurate focusing limit can become even more unstable with each successive frame and with continued use of the fixture due to additional thermal effects.

[0019] These factors can be largely mitigated by leveraging image information (features) to adjust inaccurate geometric metadata. For example, image feature matching can be performed between two frames from two different physical cameras. The existing geometric metadata can be corrected based on the image features. The geometry-based distortion transformation can be recalculated based on the geometric metadata that has been corrected based on the image features.

[0020] As described here, dual images from the pair of switching cameras are used for the image-based smooth zooming described here. In the event that continuous zoom quality is compromised by thermal changes or inaccurate estimation of focus distances, image-based visual information can effectively address the resulting problems. Bundle adjustment can be applied to the camera calibrations and the world points from the visual feature comparisons, so that the optimized parameters produce a more reliable homography for image warping. Scene depth can be estimated from both image-based visual features and phase differences, resulting in improved zoom smoothness during camera changes.

[0021] In some embodiments, the image-based algorithmic processes can be performed at up to 30 frames per second (fps) and can be configured to work seamlessly with other camera features such as image distortion correction, video stabilization, and so on. Computationally intensive steps, such as visual feature extraction, can be rendered less intensively using multithreading and DSP solutions.

[0022] The use of image-based visual features offers several advantages, including the ability to conveniently make images (e.g., in regular RGB format) available via a device's camera system. Since image alignment during camera switching is a desired result of continuous zooming, warping estimated from the image itself is more reliable and appropriate. Such warping effectively combines the image-based visual features with geometry-based calibrations and focus distances, improving the smoothness and stability of zooming during camera switching.

[0023] Thus, the techniques described here can enhance image capture devices equipped with multi-camera systems by reducing and / or eliminating visual variations in images and / or videos during camera transitions, thereby improving their actual and / or perceived quality. Improving the actual and / or perceived quality of photos or videos can provide benefits to the user experience. These techniques are flexible and can be applied to a wide variety of videos, both indoors and outdoors.

[0024] In the following, the term "homography" is used to refer to the implementation of a distortion transform. Furthermore, terms such as "distorted," "distort," etc., may be used in the context of applying a distortion transform. Smooth continuous zooming

[0025] Fig. Figure 1 illustrates binocular disparity in a multi-camera system according to the exemplary embodiments. For illustrative purposes, Fig.1 both cameras, t1 and t2, are directed at the objects (focused and unfocused). In some situations, camera t2 may be installed physically adjacent to camera t1 (e.g., right, left, and so on). For example, the camera positions can be designed to mimic human vision with the left and right eyes. In general, focused scene objects with the same depth (i.e., distance from the camera), such as the focused object 110, can be transferred almost perfectly from one camera, t1, to another camera, t2. For example, the focused object 110 in camera t1 is directed at the focused object 110A of camera t2, with no deviations. A planar object with a plane perpendicular to the camera's line of sight may exhibit such properties. Zooming may trigger a camera change (e.g., between wide-angle and ultra-wide-angle, wide-angle and telephoto, etc.).), resulting in a change in the FOV and a deviation known as binocular disparity. For objects that are out of focus, a disparity (a jump) between the images in two cameras is noticeable. For example, the distant object 105 and the near object 115 in camera t1 are out of focus. Accordingly, the distant object 105 in camera t1 is imaged onto the distant object 105A in camera t2, which is shifted from a real position 105B, when the cameras are switched. Similarly, the near object 115 in camera t1 is imaged onto the near object 115A in camera t2, which is shifted from the real position 115B.

[0026] As described here, a distortion transformation can be applied to reduce binocular disparity by distorting the image from one camera to the other. Objects with in-focus scenes (e.g., in-focus object 110) with the same depth can be distorted from one camera to the other without any perceptible disparities.

[0027] For out-of-focus objects (e.g., distant object 105, near object 115, etc.), aberrations may occur, or plane distortion may occur across multiple depths. Such aberrations for out-of-focus objects depend on the depth and the camera baseline. For example, for a focused planar object with a plane tilted toward the camera's view, aberrations of the "rotation" type may occur.

[0028] Fig.2 is a flow diagram of a workflow 200 for an image-based calculation of a distortion transform according to the exemplary embodiments.

[0029] At block 210, the workflow involves capturing frame-based data from a first and a second camera. A frame can be considered a unit of data processing. It contains the input data required for image-based continuous zooming, including the images, pre-crops of the images, camera calibrations including intrinsic and extrinsic parameters, autofocus distances, and other metadata.

[0030] At block 220, the workflow involves performing visual feature detection and matching to determine visual matches. Various feature detectors and descriptors may be used, such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Fast REtinA Keypoint (FREAK), or Features from Accelerated Segment Test (FAST) corner detection algorithm, and so on. Additional and / or alternative visual features may be used in the pipeline as long as the algorithm can achieve visual matches of sufficient quality.

[0031] Image feature matching can be performed in two steps, such as feature extraction and feature matching. Different approaches to feature extraction and matching can be used. For the purposes described here, existing feature matching methods such as ArCore features or ILK features can be used. The term "II,K" here generally refers to an inverse search version of the Lucas-Kanade algorithm for optical flow estimation.

[0032] At block 230, the workflow involves mapping visual features, camera calibrations, and autofocus distances for each frame through a two-view bundle. Camera calibrations can be used to estimate the depth of the points observed as feature matches. As described herein, the OIS / VCM system and / or thermal effects can cause the geometric metadata, such as camera calibrations and focus distances, to be updated frame by frame with significant errors. Accordingly, manipulation of specific calibration parameters can be performed so that the depth values of most points are close to the geometric focus distances.

[0033] Some embodiments may enable a smooth transition between cameras (e.g., wide-angle and telephoto cameras), but result in an FOV jitter issue at a distortion source camera. For example, visual features of downscaled images (e.g., 320 x 240) may not correspond to the same landmarks frame by frame. Additionally, for example, camera calibration is updated per frame due to OIS / VCM updates, and scene depth of focus is calculated and / or corrected per frame from the valid (e.g., inlier) feature matches between the dual cameras (e.g., wide-angle and telephoto). In some embodiments, when two cameras have different FOVs, the number of visual feature matches may be limited by the smaller FOV (e.g., telephoto), resulting in wasted visual information from the edges of the camera with the larger FOV (e.g., wide-angle).

[0034] Some approaches to mitigating such jitter may involve damping control of scene focus distance changes, and frame-based continuous zooming can then be triggered during the zoom process. Further, to address the jitter problem and utilize as much visual information as possible provided by the larger FOV camera, visual feature matching can be performed temporally between neighboring frames t - 1 and frame t, aiming for temporal consistency, such that each frame considers the geometric metadata of the previous frames when determining a distortion mesh.

[0035] At block 240, the workflow involves performing a bundle adjustment based on the two-view bundle relationships to obtain a set of optimized camera calibrations, focus distances, and other involved parameters. For example, image-based visual information can be leveraged to correct the geometric metadata of upstream modules so that it is more compatible with the images to be displayed as continuous zoom previews.

[0036] If bundle adjustment is performed frame by frame, optimized solutions for camera calibration and focus distances (e.g., with minimal pixel errors during reprojection) can be determined independently frame by frame. Some embodiments may exhibit misalignment between frames, resulting in a blurry preview when the distorted frames are displayed during sequential playback.

[0037] In such embodiments, the geometric bundle may be built over a window of individual images, and optimizing camera calibrations and other metadata may not always result in a smooth change in the individual image of the distortion transformation. Therefore, instead of using the homography of the most recently optimized data to distort the images, an attenuation process is given by: Ht=(1−λ)*Ht−1+λ*It is introduced for a step change and a smooth change of the distortion transformation. In equation 1, the term λ is a damping ratio with values between 0 and 1, H t is the homography to be applied to the frame t, and I t is the optimized image-based homography at single image t.

[0038] In some embodiments, prior to extracting the image-based visual features, a homography G is determined based on the geometric metadata (e.g., camera calibrations and focus distances) from upstream modules. However, this may include errors from OIS / VCM updating, thermal effects, and other sources, as previously described. This can be corrected with image-based data as follows:

[0039] Generally, two camera calibration models are available. One calibration model has been updated through OIS / VCM correction, which directly corresponds to the visual features of the images, and the other calibration model is kept in a neutral state and used as consistent initial values for further geometric optimization.

[0040] The previously calculated geometry-based homography G can be used to perform a coarse pre-warping of the image features and associated calibrations, followed by the previously described image-based process to correct the remaining errors. Such an approach effectively combines geometric and visual information to solve the engineering problem.

[0041] At block 250, the workflow involves determining a pre-distortion transform of the first camera image based on the bundle adjustment such that the distorted image has no more than a slight disparity with the second camera image.

[0042] Subsequently, the workflow at block 260 involves modifying the distortion transform based on the image features to finely distort the first camera image and further reduce the low disparity from the distortion transform.

[0043] Some embodiments involve optimizing the detection of the one or more visual features and generating the visual match by performing asynchronous multi-threaded processing, comprising receiving one or more images and associated metadata as inputs and sending visual feature matches and associated metadata as outputs. For example, image-based continuous zooming may involve computationally intensive steps, such as visual feature detection and matching. In order for the solution to execute in real time (i.e., at least 30 frames per second (fps)) on a consumer device, the computationally intensive steps may be processed asynchronously by a dedicated thread that receives images and the associated metadata as inputs and sends visual feature matches and the associated metadata as outputs.

[0044] The term "sparse features" used here generally refers to detected features that are sparsely distributed throughout the image. A feature point can be detected when a pixel and its surroundings reach a detection threshold. This can include, for example, the ArCore function, the Portrait Mode function, and the AutoCal function.

[0045] The term "dense features" used herein generally refers to the detected feature that could occlude an entire image. A feature point can be detected based on a predefined image region, and a match can be found for each region. In some embodiments, ILK can be used for dense feature detection. For example, an ILK algorithm can be used to extract "dense" feature points and matches from images.

[0046] The dense or sparse features may generally have different configurations in the ContiZoom conduit, as further described in detail below. Recalibration (sparse feature flow)

[0047] In contrast to planar target features typically used in factory calibration, sparse feature calibration uses natural features to recalibrate the geometric information received from the camera sensors. For example, image features are used to update the geometric metadata, and the existing geometrically based computation is used to calculate the modified homography.

[0048] Fig.Figure 3 is an example sparse feature workflow 300 for smooth continuous zooming in a multi-camera system according to the example embodiments. The general flow with sparse features is illustrated.

[0049] At block 305, input images, including dual 320 × 240 images, are acquired. The input images are acquired from the two target cameras. In some examples, the image resolution can be up to 640 × 480. The quality of the feature matches depends heavily on the quality of the scene's texture.

[0050] At block 310, sparse feature detection may be performed as previously described. In some embodiments, an ArCore function may be used. In some embodiments, the FAST function may be used. Generally, scale-invariant features are not required because the dual image can be rescaled to provide the focal length relatively precisely.

[0051] Natural feature calibration may be performed at block 315. This process recalibrates the camera parameters 320 from a factory calibration, provided, for example, by a camera manufacturer 325. Some embodiments may use a DualCameraCalibrator or AutoCal, each based on the FAST feature detector. Additionally, an optical flow-based detector such as ILK may be used, for example. However, natural feature calibration generally differs from factory calibration because the natural features do not originate from planar objects. Accordingly, a bundle adjustment (BA)-based approach (e.g., block 240 of Fig. 2) may be preferable to perform natural feature calibration.

[0052] In some embodiments, natural feature calibration may not optimize all parameters and instead focus on "principal points" and "extrinsic rotation" for wide-angle and telephoto. Table 1 below summarizes the parameters that can be optimized. Table 1 is for illustrative purposes only and may vary from device to device, depending on the types and / or characteristics of the cameras involved in the transition process. Table 1 Wide angle Tele Ultra-wide angle Key points Yes Yes No oblique No No No Focal length Perhaps Perhaps No translation No Perhaps No rotation No Yes No

[0053] At block 330, delta camera metadata may be obtained. CPI-based calibrations are relative to an active array coordinate, while image-based algorithms require calibrations relative to image coordinates. Accordingly, a transformation between the active array and the image may be determined. After the camera metadata is corrected, the difference between the factory calibrations and the corrected metadata may be stored as delta metadata and stored separately from the CPI calibration metadata. In some embodiments, the delta may be a constant offset during a transition period when transitioning from one camera to another.

[0054] Image-based correction can utilize the same camera metadata structure, representing the delta between the CPI outputs and the recalibrated camera metadata. Some embodiments can utilize the delta focusing distance (e.g., depth).

[0055] At block 335, features within a region of interest (ROI) may be detected. An ROI, as used herein, is a subregion within a single image that is deemed important to a user and is used in camera applications as a pilot region for many features, such as autofocusing, enabling focus distance used for geometry-based methods. In some embodiments, the ROI may be obtained as an ROI rectangle 340 from an algorithm, such as a face detection algorithm, a dropout detection algorithm, and so on. In general, the ROI may be processed differently for sparse features and dense features. Features within the ROI may be based on the sparse features detected at block 310.

[0056] A median depth in the ROI is determined at block 345. For example, a median of the depths of (inlier) image feature points extracted by the sparse / dense feature methods described above may be determined. Temporal feature matching and tracking

[0057] In some embodiments, temporal feature matching and tracking may be performed. Generally, the same landmark or ROI may appear in multiple frames (e.g., three or more frames), resulting in feature gradients. In some embodiments, temporal feature tracking may be applied only to the camera with a larger FOV (e.g., wide angle), which is sufficient for the temporal consistency of ContiZoom networks.

[0058] Fig. 4A illustrates temporal feature tuning and tracking according to the exemplary embodiments. Referring to Fig.4A illustrates a plurality of consecutively captured image frames for a wide-angle camera 405 and a telephoto camera 410. For the wide-angle camera 405, the first image frame 415 is illustrated at time t-1, and a second image frame 420 is illustrated at time t as two consecutive image frames. For the telephoto camera 410, the third image frame 425 is illustrated at a time t-1, and a fourth image frame 430 is illustrated at a time t as two consecutive image frames. Intra-frame feature matching is illustrated where, at a time t-1, the first feature A in frame 415 matches a corresponding feature A' in frame 425. Similarly, intra-frame feature matching is illustrated where, at a time t, the second feature B in the second frame 420 matches a corresponding feature B' in the fourth frame 430.Temporal feature matching and tracking is illustrated where the first feature A in the first frame 415 at time t-1 matches the second feature B in the second frame 420 at time t. For illustrative purposes, temporal feature tracking is shown for the wide-angle camera 405. In general, it may be desirable to perform temporal feature tracking in the camera with a larger FOV to capture the relevant feature trajectories.

[0059] Fig. 4B illustrates example images for temporal feature matching and tracking according to example embodiments. Referring to Fig. 4B illustrates two images. The first image 435 illustrates feature tuning without attenuating the focus distance. The second image 440 illustrates temporal feature tuning to reduce jitter.

[0060] In general, feature matching and tracking may involve two tasks: 1) determining feature tracking information from the images; and 2) applying feature tracking to the existing line to improve ContiZoom quality.

[0061] In some embodiments, the first task may involve providing interface functions for feature extraction and feature matching. As described with respect to Fig.4A, intra-frame feature matching may be performed on the dual images from the feed and follower cameras (e.g., wide-angle camera 405 to telephoto camera 410) to create intra-frame feature matchings within a frame. In some embodiments, temporal feature tracking may be performed by enabling feature matching between adjacent frames t-1 and t, with refactoring of the interface functions and the cache of extracted features from the previous frame.

[0062] In some embodiments, the second task may involve creating an indexing manager to handle the indices of visual feature points and matches between multiple images. This indexing manager enables the use of temporal feature tracking along the intra-frame feature matches. For example, the indexing manager manages the visual feature indices, including the feature point index and the feature match index, along their mutual matches. Some embodiments may support retrieving the feature point index from the feature match index and retrieving the feature match index from the index of a first feature point and a second feature point of that match.

[0063] Some embodiments may involve one or more mappings. For example, a first mapping from the index of a feature point from a first image to the index of the matching pair may involve the feature point. As another example, a second mapping from the index of a feature point from a second image to the index of the matching pair may involve the feature point. Additionally, for example, a vector of feature matches may be determined. For example, each feature match may involve two feature points from the first image and the respective second image.

[0064] Experimental evidence indicates that the jitter in the wide-as-lead camera may be primarily caused by jitter in the scene focus distance estimated per frame. Accordingly, temporal feature tracking can be used to estimate the scene focus distance at frames t. Leveraging feature tracking across multiple frames allows for quality improvement while maintaining temporal consistency.

[0065] In some situations, it may be reasonable to assume that a user performing a zoom operation with the camera is unlikely to exhibit large movements (e.g., panning, running, walking, or rapid changes of salient objects / ROI). In the case that the user exhibits large movements, small jitter or FOV transitions from zooming are likely to be unnoticeable. In the case that the user does not exhibit large movements, the change in focus distance between adjacent frames t - 1 and t must be managed to avoid perceptible jitter. One approach to achieving this is to keep the focus distance unchanged if frame t - 1 and frame t have a sufficient number of matching inliner tunings.

[0066] Fig.4C illustrates an example application 400 for temporal feature matching and tracking according to example embodiments. Fig. Figure 4C illustrates a plurality of consecutive image frames for a wide-angle camera 405 and a telephoto camera 410. In general, each image frame may estimate scene depth from a set of feature matches between the two cameras (e.g., wide-angle camera 405 and telephoto lens 410) of the same image frame. In some embodiments, an inlier feature may be selected to represent the scene for depth estimation. For illustrative purposes, the present example involves four (4) focus distances, namely d1, d2, d3, and d4. Temporal feature tracking between frame t-1 and frame t may be performed as follows.

[0067] In the event that a selected feature of the inlier has a temporally matched counterpart, the scene focus distance d2 can be set to be the same as d1. For example, feature A in frame t1 445 has a corresponding feature A' in frame t1' 465. Furthermore, feature A in frame t1 445 has a temporally matched counterpart feature B in frame t2 450. Accordingly, the scene focus distance d2 can be set to be the same as d1.

[0068] In the event that a selected inlier feature has no timing, but another feature with a similar depth to the originally selected inlier has timing and alignment with an intra-frame feature, the scene focus distance d3 can be set to be the same as d1. For example, feature E in frame t3 455 has no timing. However, feature D in frame t3 455 is at a similar depth to feature E in frame t3 455. In addition, feature D in frame t3 455 has a temporally matched counterpart feature C in frame t2 450. Furthermore, feature D in frame t3 455 has a corresponding feature D' in frame t3' 475. Accordingly, the scene focus distance d3 can be set to be the same as d1.

[0069] In the event that a selected inlier feature and its siblings with similar depths do not exhibit both temporal alignment and intra-frame feature alignment, d4 is recalculated. For example, feature H in frame t4 460 has a corresponding feature H' in frame t4' 480. However, feature H in frame t4 460 does not temporally align with another feature. Features G and I in frame t4 460 appear to have a similar depth to feature F. Feature G is temporally aligned with feature F in frame t3 455; however, feature G does not exhibit intra-frame feature alignment. Feature I also does not exhibit intra-frame feature alignment or temporal alignment. Accordingly, d4 is recalculated.

[0070] In the approach described above, scene depth estimation (i.e., focus distance) is based on per-frame intra-frame (i.e., between cameras) feature matching, and temporal tracking is generally used as post-verification to determine whether to change scene focus distances in a subsequent frame. However, this approach does not effectively utilize the information from temporal feature tracking.

[0071] Accordingly, an alternative approach to reducing and / or removing jitter by adjusting scene focus distances may involve the direct use of temporal feature tracking information, particularly when such information is available in sufficiently high quality. Factors that may determine the quality of the temporal feature tracking information may include one or more of (1) a sufficient number of timings between adjacent frames at one time t - 1 and another time t; (2) a sufficient number of timings that allow temporal tracking across multiple frames in the absence of large pan and / or rotation motion; or (3) affordable performance and latency when operating on a mobile device.

[0072] One approach uses temporal feature tracking along gyroscopic measurements to triangulate and select the "at-scale" 3D points as good inlier landmarks. Subsequently, the two-camera observations of these inlier landmarks can be used to estimate scene depth per frame.

[0073] In another approach, temporal feature tracking can be used along with gyroscopic and accelerometer measurements to directly estimate device positions and 3D points as inlier landmarks. The scene depths per frame can then be based on these inlier landmarks. Considering the performance and latency aspects of the device estimating the pose, this approach may be better suited for offline processing.

[0074] With further reference to Fig.3, the delta camera metadata from block 330 and the median depth from block 345 are used at block 350 to update the geometric homography (to determine an updated distortion transform) based on image features. For example, matching feature points can be used to correct inaccurate geometric metadata. This can involve two approaches: a recalibration-based approach with a sparse feature flow and an image homography-based approach with a dense feature flow.

[0075] Homography H is a 3 × 3 matrix that maps pixels in a plane in a first coordinate system for a first camera to corresponding pixels in the same plane in a second coordinate system for a second camera. The homography can be decomposed as follows: H=K2⋅(R−t*nTd)⋅K1−1 where K1 and K2 are intrinsic matrices corresponding to the first camera and the second camera and containing focal lengths and principal points, and where [R 3×3 | t 3×1 ] is the extrinsic that can transform a point in the first coordinate system for the first camera into the second coordinate system for the second camera, d and n define the focused plane for this homography in the coordinates of the first camera, so that the points X in the plane satisfy n T X + d = 0. In general, n = [0,0,-1] T used.

[0076] Given camera calibrations and the distance of the focused object, there are at least two ways to determine homography. For example, in a first approach, homography can be determined by inputting these parameters into the formula as shown in Equation 2. Furthermore, in a second approach, homography can be determined from a set of pixel pairs (e.g., at least four sample point pairs). For example, the homography matrix transforms a plane (with a certain depth) on the telephoto camera into a corresponding plane on the wide-angle camera. In some embodiments, for a pair of cameras (e.g., telephoto and wide-angle camera models), a "four-point" approach can be used to calculate the homography matrix between telephoto and wide-angle cameras. The input can include the telephoto and wide-angle camera models (intrinsic and extrinsic) and the target plane distance to the telephoto camera (object distance).

[0077] The “four-point approach” may involve randomly selecting four two-dimensional (2D) points on the telecamera, denoted as P tele . The 2D points can then be unprojected as 3D rays, denoted by Ray, using camera-intrinsic parameters. Ray can then be intersected with the given plane located at a distance Obj_dist from the telephoto camera. This creates four three-dimensional (3D) points in real space. The 3D points can be projected onto the wide-angle camera to obtain P Weitwinkel The homography matrix can then be determined as follows: [PWide Angle|1]=[H*PTelephoto|1] where [P|1] represents the homogeneous coordinate of P.

[0078] The second approach to determining homography from the set of pixel pairs may introduce distortions of the intrinsic camera data into the estimation process. When no image-based visual information is available, camera calibrations are taken from the CPI library and focus distances are taken from the autofocus process.

[0079] At block 355, a head start handler performs the head start based on the homography of block 350.

[0080] A mesh conversion function is applied to block 360. Image-based homography (dense feature flow)

[0081] The image-based approach directly computes an updated homography based on matching image features and combines the image-based homography with the geometric-based homography. Since the goal is to link two images (from two cameras), the image features can be used directly to compute the distortion homography without the geometric camera metadata.

[0082] Fig. Figure 5 is an example dense feature workflow 500 for smooth continuous zooming in a multi-camera system according to the example embodiments. The general flow with dense features is illustrated.

[0083] At block 505, input images, including dual 320 × 240 images, are acquired. The input images are acquired from the two target cameras. In some examples, the image resolution may be 640 × 480 or higher, depending on the processing power of the computing device and a latency requirement for different use cases. The quality of the feature matches may depend more heavily on the quality of the scene's texture.

[0084] The aligned ROI region is calculated at block 510. Some embodiments may involve cropping the original image frame to the ROI region. An ROI rectangle 525 may be obtained (e.g., as previously described with respect to ROI rectangle 340). Some embodiments involve aligning the two ROI regions with the calculated geometry-based homography so that dense feature detection may have better initial placement. To conserve computational resources, in some embodiments, the translational components may be extracted from the geometry-based homography, and these translational components may be applied to the ROI. In some embodiments, this translation may be performed by cropping.

[0085] Dense feature detection may be performed at block 520. Such feature detection / tuning was described in previous sections, and the ILK algorithm may be used. This process recalibrates the camera parameters 530 from a factory calibration, provided, for example, by a camera manufacturer 535. Some embodiments may use a DualCameraCalibrator or AutoCal, each based on the FAST feature detector. Additionally, an optical flow-based detector such as ILK may be used, for example. However, natural feature calibration generally differs from factory calibration because the natural features do not originate from planar objects. Accordingly, a bundle adjustment (BA)-based approach (e.g., block 240 of Fig. 2) may be preferable to perform natural feature calibration.

[0086] At block 540, a separate delta homography may be determined. In some embodiments, this may be based on camera parameters 530 from a factory calibration, for example, provided by a camera manufacturer 535. In some embodiments, the dense features from the foreground may be separated from those in the background. For example, while feature detection may focus on features within the ROI, background features may also be used. In some embodiments, a pure translation homography may be applied to confirm that the plane of the homography is perpendicular to the camera's z-axis. Additionally, for example, a two-stage k-means cluster may be used, where the features are the disparity and the positions of the features are chosen to be the center of the ROI.

[0087] A similar k-means process can be used for the sparse feature flow counterpart, however, the sparse feature flow may not have enough feature points to apply these approaches. Accordingly, a less optimal solution based on determining the median depth for the sparse feature flow (e.g., at block 345) may be used.

[0088] In general, the foreground homography is a delta homography in addition to the geometry-based homography because the original ROI has already been shifted using the geometry-based homography.

[0089] At block 545, an updated distortion transformation (or combined homography) may be determined as a combination of the image-based delta homography and the geometry-based homography. Some embodiments of the updated distortion transformation are a concatenation of the two homography mappings (image-based homography applied after the geometry-based homography) with corresponding coordinate transformations.

[0090] Existing components can be leveraged for both dense and sparse feature flow. For example, the term "camera manufacturer" (e.g., Camera Manufacturer 320, Camera Manufacturer 535) refers to the module that reads the factory calibration file and uses CPI to generate a camera parameter corresponding to each OIS / VCM metadata.

[0091] The term “CPI camera parameters” (e.g., camera parameters 325, camera parameters 530) refers to intrinsic and / or extrinsic camera parameters calculated by CPI.

[0092] At block 550, a head start handler performs the head start based on the combined homography of block 545.

[0093] A network conversion function is applied to block 555.

[0094] Sparse features may generally be more precise, providing feature points with accurate (x, y) image coordinates. However, sparse features may be relatively slow, and the number of detected features may depend on the complexity of the scene. Accordingly, the lack of a sufficient number of feature detections may negatively impact the performance of calibration optimization.

[0095] Dense features may be less precise because these "feature points" are essentially image patches. However, dense feature detection is generally fast and less dependent on scene complexity.

[0096] In some embodiments, a combination of sparse and dense features may be used. For example, sparse features may be detected first, and if the number of detected features is below a threshold, dense feature detection may be performed.

[0097] In addition, for example, sparse features can be used to determine a more precise depth, and a better homography can be determined as an initial estimate for dense feature matching.

[0098] As another example, dense features can be used to determine a region patch of foreground objects, and sparse features can be used to detect precise feature points on the foreground region patch.

[0099] Delta data is stored for both sparse and dense features. In the case of sparse features, the delta data is the camera metadata, and in the case of dense features, the delta data is the homography. Generally, both cameras may not be available during a zoom operation.

[0100] Fig. 6 illustrates exemplary handling of delta data during camera transitions according to the exemplary embodiments. The example in Fig.6 is based on a transition between wide-angle and telephoto lenses. However, a similar approach can be applied to transition between different pairs of cameras. The examples of zoom ratios, states, number of states, transition points, and so on are for illustrative purposes only. Such values may vary depending on the type of device, type of cameras, distance of the camera from the objects, and so on. For example, in Fig. 6, the mark "4.2x" is used to indicate the zoom factor at which the transition from wide-angle to telephoto lenses can be made. However, this value can vary depending on the type of device, the cameras involved in the transition, the distance of the camera from the objects, and so on. In addition, for example, the zoom factors shown in the example in Fig.6 Zoom ratios used (e.g. 2x, 4x, 4.2x, 4.4x) are example values and may vary depending on the device and / or system configurations.

[0101] For example, a transition from wide angle to telephoto is illustrated at 605, and a reverse transition from telephoto to wide angle is illustrated at 610. The legend 615 indicates the leading and following cameras.

[0102] State 1 (initially open at 1.0x) corresponds to the time of first camera activation. The initial camera can be the wide-angle camera, and the homography is the identity process. The geometric data of the wide-angle camera's CPI can be updated because it does not affect the homography.

[0103] State 2 (from 2x scale) corresponds to when the homography gradually transitions from the identity process to a target homography. However, since the dual cameras are not available in State 2, this homography is geometry-based and does not consider image features. No delta data is available. To avoid causing visual distortions during the transition between State 1 and State 2, in some embodiments, the CPI update can be attenuated.

[0104] State 3 (from 4.0x scale to 4.2x scale) corresponds to when both cameras are active simultaneously. In this case, the wide-angle camera is the main or feeder camera, and the telephoto camera is the secondary or follower camera. The image-based approach described previously can be used to calculate the delta data, and the updated metadata can be used to calculate the homography.

[0105] State 4 (from 4.2-scale to 4.4-scale magnification) corresponds to the state after switching from the wide-angle camera to the telephoto camera. In this case, the telephoto camera is the main or feeder camera, but the wide-angle camera remains active. No warping is applied, and homography will therefore be the identity operation. The last calculated delta is saved. And the geometric metadata of the telephoto and wide-angle cameras is updated.

[0106] In state 5 (beyond 4.4x), the telephoto camera is the main or feeder camera, and the wide-angle camera is inactive or closed. The remaining operations remain the same as in state 4.

[0107] State 6 (convolved) is similar to state 2, but now contains delta data. Accordingly, the delta data is stored and the homography is calculated with the additional delta.

[0108] State 7 (folded back) is similar to state 1, but now contains delta data. Accordingly, the delta data is stored.

[0109] During successive transitions, the process can be repeated between states 3 to 7.

[0110] The transition zone is a defined smooth transition zone where the two cameras are simultaneously active within a specific range of the zoom scale, allowing metadata (e.g., from OIS / VCM) to be streamed simultaneously for both cameras. For the image-based approach described here, it may be preferable to make this transition period as large as possible to reduce abrupt changes between camera metadata and / or between image-based and geometry-based results.

[0111] In some embodiments, it may be impractical to constantly stream to two cameras and / or to enlarge the transition zone to an optimal level due to hardware limitations. In such embodiments, occasional dual streaming may also be used. For example, occasional dual streaming means that the two cameras are occasionally active simultaneously, not due to the zoom scale, but due to a timer. For example, after opening a camera application, the timer may be set to 10 seconds, and the two cameras may be active simultaneously every 10 seconds. Smooth continuous zooming may occur periodically based on such a timer.

[0112] Fig.7 is a table 700 illustrating various cases for switching between a telephoto camera and a wide-angle camera according to the exemplary embodiments. Column C1 lists the states that are referenced to Fig. 6; Column C2 lists the state related to geometric metadata for the wide-angle camera; Column C3 lists the state related to geometric metadata for the telephoto camera; Column C4 lists the state related to delta data; and Column C5 lists the state related to homography. Each row, rows R1-R7, provides the status for each state, states 1-7 respectively. Table 700 summarizes the information provided with respect to Fig.6. For example, line R2 indicates that for state 2, a damp update is applied to the geometric metadata for the wide-angle camera, that a canonical mapping is used for the geometric metadata for the telephoto camera, that no delta data is present, and that the homography is a combination of ratio delta and geometric homography. Other lines present similar information for the respective states.

[0113] The term “delta data” refers to the results of the image-based solution described previously, where the delta consists of the camera’s geometric metadata in the case of sparse data and the delta homography in the case of dense data.

[0114] The "Update" state generally indicates near-real-time updating according to the OIS / VCM. The "Keep" state indicates that the state is the same as the previous state. The term "attenuation update" refers to an incremental update that has an attenuation ratio between the data from a previous frame and the data from a current frame. The term "Geo" refers to geometry-based homography (without image features). The term "delta+geo" refers to a combined homography of an image-based solution and a geometry-based solution. The term "ratio homography" indicates that the strength of the homography can depend on the zoom scale (e.g., for state 2 in row R2), the homography is identical at 2.0x, and the homography is at full strength when it is at 4.2x.As an interpolation between the identity process and the full-strength homography, other scales between 2.0x and 4.2x can be determined.

[0115] Delta damping is generally an operation to smooth out sharp changes in geometric data, such as abrupt changes in OIS / VCM. The damping ratio can be based on a change in zoom scale between two consecutively acquired frames. Similar damping can be applied to the triangle.

[0116] Fig.8A depicts an exemplary geometric relationship 800A at each pair of matching pixels according to the exemplary embodiments. A first plane 805 in the XYZ plane is depicted to include a point w with coordinates relative to an origin, O. The second plane 810 corresponds to the first plane 805 in the X'Y'Z' plane. The first coordinate system representing the XYZ plane can be mapped to the second coordinate system representing the X'Y'Z' plane with coordinates relative to an origin O' by a mapping X' = RX + T, where R is a rotation and T is a translation. For example, the point w in the first plane 805 is mapped to the point w' in the second plane 810.In some embodiments, a geometric two-view relationship may be established at each pair of matched pixels using intrinsic and extrinsic camera data, triangulated points from visual matching, and the autofocus distance as the initial scene depth.

[0117] Fig. 8B depicts a workflow 800B for determining a geometric relationship at each pair of matching pixels according to the exemplary embodiments. At block 815, a point w (e.g., in the first plane 805) is selected. At block 820, intrinsic features related to the first camera (e.g., the wide-angle camera with reference to Fig. 5 and Fig.6). At block 825, the 2D point w may be unprojected as a 3D ray, denoted as Ray, using the intrinsic properties related to the first camera. Depth information may be received at block 830. At block 835, a 3D point for the first camera is determined based on Ray and the depth data. At block 840, extrinsics of the first camera are mapped to the second camera (e.g., the telephoto camera with respect to Fig. 5 and Fig.6). At block 845, a 3D point for the second camera (corresponding to the 3D point determined at block 835) is determined. At block 850, intrinsics related to the second camera are determined. At block 855, a reprojection of point w is determined based on the intrinsic properties related to the second camera. Based on an actual position of point w' in the second camera, as obtained at block 860, and the reprojection of point w, one or more reprojection errors are determined at block 865.

[0118] Accordingly, a visual correction of geometric data in individual images is provided. Workflow 800B enables minimized reprojection errors of the visual correspondences. Based on workflow 800B, the geometry-based homography can be re-estimated using the partially corrected camera calibrations and the focused object distance to achieve uniformity across the individual images.

[0119] Fig. Figure 9 depicts an example workflow 900 for smooth continuous zooming in a multi-camera system according to exemplary embodiments. A continuous zoom frame 905 may include calibration name file 910, OIS / VCM pairs 915 from the two cameras, and distortion mesh configuration 920.

[0120] The text herein with reference to at least Fig.1-8 may be controlled by the continuous zoom manager 925. In some embodiments, the continuous zoom manager 925 may include the data trimmer 930, the calibration provider 945, and the homography provider 955. The data trimmer 930 may perform data validation 935 and data damping 940. The calibration provider 945 may provide CPI parameters 950 as obtained from a factory calibration file 980.

[0121] Homography provider 955 may determine geometry-based homography 960 and image-based homography 965 as described herein. Homography provider 955 may then determine homography compensation 970 and protrusion treatment 975.

[0122] Legend 985 indicates the different classes of components involved, such as container class, member functions, member variables and the function class. Example data network

[0123] Fig. 10 illustrates a distributed computing architecture 1000, according to example embodiments. The distributed computing architecture 1000 includes server devices 1008, 1010 configured to communicate with programmable devices 1004a, 1004b, 1004c, 1004d, 1004e over the network 1006. The network 1006 may be a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public internet, or any other type of network configured to provide a communication path between networked computing devices. The network 1006 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public internet.

[0124] Although Fig.10 shows only five programmable devices, distributed application architectures may serve dozens, hundreds, or thousands of programmable devices. Furthermore, the programmable devices 1004a, 1004b, 1004c, 1004d, 1004e (or additional programmable devices) may be any type of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mountable device (HMD), a network terminal, a mobile computing device, and so on. In some examples, such as illustrated by the programmable devices 1004a, 1004b, 1004c, 1004e, programmable devices may be directly connected to the network 1006. In other examples, such as illustrated by the programmable devices 1004a, 1004b, 1004c, 1004d, 1004e, programmable devices may be directly connected to the network 1006. B. the programmable device 1004d, programmable devices may be indirectly connected to the network 1006 via an associated computing device, such as.B. the programmable device 1004c. In this example, the programmable device 1004c may function as an associated computing device to pass electronic communication between the programmable device 1004d and the network 1006. In other examples, as illustrated by the programmable device 1004e, a computing device may be part of a vehicle and / or located on the interior of a vehicle, such as a car, a truck, a bus, a boat or ship, an aircraft, etc. In other examples, as illustrated in . Fig. 10, a programmable device may be connected to the network 1006 both directly and indirectly.

[0125] Server devices 1008, 1010 may be configured to perform one or more services as requested by programmable devices 1004a-1004e. For example, server device 1008 and / or 1010 may provide content to programmable devices 1004a-1004e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are also possible.

[0126] As another example, server device 1008 and / or 1010 may provide programmable devices 1004a-1004e with access to software for database, search, computing, graphics, audio, video, World Wide Web / Internet usage, and / or other functions. Many other examples of server devices are also possible. Computing device architecture

[0127] Fig. 11 is a block diagram of an exemplary computing device 1100 according to embodiments. In particular, the Fig. 11 may be configured to perform at least one function of and / or in connection with method 1200.

[0128] Computing device 1100 may include a user interface module 1101, a network communications module 1102, one or more processors 1103, a data store 1104, one or more cameras 1118, one or more sensors 1120, and a power system 1122, all of which may be interconnected via a system bus, network, or other interconnection mechanism 1105.

[0129] The user interface module 1101 may be operable to send and / or receive data to and / or from external user input / output devices. For example, the user interface module 1101 may be configured to send and / or receive data to and / or from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a speech recognition module, and / or other similar devices. The user interface module 1101 may be configured to send outputs to user display devices such asone or more cathode ray tubes (CRTs), liquid crystal displays, light-emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, either now known or later developed. The user interface module 1101 may be configured to generate audible outputs using devices such as a speaker, a speaker jack, an audio output port, an audio output device, headphones, and / or other similar devices. The user interface module 1101 may further be configured with one or more haptic devices capable of generating haptic outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with the computing device 1100.In some examples, the user interface module 1101 may be used to provide a graphical user interface (GUI) for use of the computing device 1100.

[0130] The network communication module 1102 may include one or more devices providing one or more wireless interfaces 1107 and / or one or more wired interfaces 1108 configured to communicate over a network. One or more wireless interfaces 1107 may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, an LTE™ transceiver, a WiMAX™ transceiver, and / or other types of wireless transceivers configurable to communicate over a wireless network.One or more wireless interfaces 1108 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or a similar transceiver configurable to communicate over a twisted pair cable, a coaxial cable, a fiber optic connection, or a similar physical connection to a wired network.

[0131] In some examples, the network communication module 1102 may be configured to enable reliable, secure, and / or authenticated communication. For each communication described herein, information to facilitate reliable communications (e.g., guaranteed message delivery) may be provided, possibly as part of a message header and / or footer (e.g., packet / message sequencing information, encapsulation headers and / or footers, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). Communications may be secured using one or more cryptographic protocols and / or algorithms, including, but not limited to, the following (e.g.,encoded or encrypted) and / or decrypted / decoded using: a Data Encryption Standard (DES), Advanced Encryption Standard (AES), Rivest-Shamir-Adelman (RSA) algorithm, Diffie-Hellman algorithm, secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms may be used to secure (and subsequently decrypt / decode) communications, in addition to those listed here or in addition to these.

[0132] One or more processors 1103 may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application-specific integrated circuits, etc.). One or more processors 1103 may be configured to execute computer-readable instructions 1106 contained in data storage 1104 and / or other instructions as described herein.

[0133] Data storage 1104 may include one or more non-transitory computer-readable media that can be read and / or accessed by at least one of one or more processors 1103. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage, which may be integrated, in whole or in part, with at least one of the one or more processors 1103. In some examples, data storage 1104 may be implemented using a single physical device (e.g., an optical, magnetic, organic, or other memory or disk storage device), while in other examples, data storage 1104 may be implemented using two or more physical devices.

[0134] Data storage 1104 may include computer-readable instructions 1106 and possibly additional data. In some examples, data storage 1104 may include storage required to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functionality of the devices and networks described herein. In some examples, data storage 1104 may include storage for a distortion transformation module 1112 (e.g., a module that computes geometry-based homography, image-based homography, and so on). In particular, these examples may include computer-readable instructions 1106, instructions that, when executed by one or more processors 1103, enable computing device 1100 to provide some or all of the functionality of distortion transformation module 1112.

[0135] In some examples, computing device 1100 may include one or more cameras 1118. The camera(s) 1118 may include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, the camera(s) 1118 may create the image(s) from captured light. The one or more images may be one or more still images and / or one or more images incorporated into video images. The camera(s) 1118 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or one or more other frequencies. Camera(s) 1118 may include a wide-angle camera, a telephoto camera, an ultra-wide-angle camera, and so on.Also, for example, the camera(s) 1118 may be front-facing or rear-facing cameras with respect to the computing device 1100.

[0136] In some examples, computing device 1100 may include one or more sensors 1120. Sensors 1120 may be configured to measure conditions within computing device 1100 and / or conditions in an environment of computing device 1100 and to provide data about those conditions. For example, sensors 1120 may include one or more of the following: (i) sensors for obtaining data about computing device 1100, e.g., but not limited to, a thermometer for measuring a temperature of computing device 1100, a battery sensor for measuring the performance of one or more batteries of power system 1122, and / or other sensors for measuring the state of computing device 1100; (ii) an identification sensor for identifying other objects and / or devices, such as, but not limited to, a radio frequency identification (RFID) reader, a proximity sensor, a one-dimensional barcode reader, a two-dimensional barcode (e.g., quick response (QR) code) reader, and a laser tracker, wherein the identifier sensors may be configured to read identifiers, e.g., RFID tags, barcodes, QR codes, and / or other devices and / or object configured to be read, and to provide at least identifying information; (iii) sensors for measuring locations and / or movements of the computing device 1100, such as, but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser displacement sensor, and a compass; (iv) an environmental sensor for obtaining data indicative of an environment of the computing device 1100, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (v) a force sensor for measuring one or more forces (e.g., inertial forces and / or G-forces) acting on the computing device 1100, such as(but not limited to) one or more sensors that measure the following forces: forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies ZMPs and / or ZMP locations. Many other examples of sensors 1120 are also possible.

[0137] The power system 1122 may include one or more batteries 1124 and / or one or more external interfaces 1126 for providing electrical power to the computing device 1100. Each of the one or more batteries 1124, when electrically coupled to the computing device 1100, may function as a source of stored electrical power for the computing device 1100. One or more batteries 1124 of the power system 1122 may be configured to be portable. Some or all of the one or more batteries 1124 may be readily removable from the computing device 1100. In other examples, some or all of one or more batteries 1124 may be located within the computing device 1100 and thus may not be readily removed from the computing device 1100. Some or all of the batteries 1124 may be rechargeable.For example, a rechargeable battery may be charged via a wired connection between the battery and another power supply, such as one or more power supplies located external to the computing device 1100 and connected to the computing device 1100 via the one or more external power interfaces. In other examples, some or all of one or more of the batteries 1124 may be non-rechargeable batteries.

[0138] One or more external power interfaces 1126 of the power system 1122 may include one or more wired power interfaces, such as a USB cable and / or a power cable, enabling wired electrical power connections to one or more power supplies located external to the computing device 1100. One or more external power interfaces 1126 may include one or more wireless power interfaces, such as a Qi wireless charger, enabling a wireless electrical power connection, such as via a Qi wireless charger, to one or more external power supplies. Once an electrical power connection to an external power source is established via one or more external power interfaces 1126, the computing device 1100 may receive electrical power from the external power source via the established power connection.In some examples, the power system 1122 may include related sensors, such as battery sensors associated with the one or more batteries or other types of electrical current sensors. Example procedure of the process

[0139] Fig. 12 illustrates a method 1200 according to example embodiments. The method 1200 may include various blocks or steps. The blocks or steps may be performed individually or in combination. The blocks or steps may be performed in any order and / or serially or in parallel. Furthermore, the blocks or steps may be omitted or added to the method 1200.

[0140] The blocks of method 1200 may be performed by various elements of computing device 1100, as described with respect to Fig. 11 shown and described.

[0141] Block 1210 includes displaying, through a screen of a computing device, an initial preview of a scene captured by a first image capture device of the computing device, wherein the first image capture device operates within a first focal length range.

[0142] Block 1220 further includes detecting, by the computing device, a zoom operation that is expected to cause the first image capture device to reach a limit of the first focal length range.

[0143] Block 1230 further comprises, in response to detecting, activating a second image capture device of the computing device to capture a magnified preview of the scene, the second image capture device configured to operate within a second focal length range.

[0144] Block 1240 includes updating a geometry-based warp transformation based on a comparison of the respective image features from the initial preview and the enlarged preview.

[0145] Block 1250 includes aligning the enlarged preview with the initial preview by applying the updated distortion transform, wherein the updated distortion transform reduces one or more viewing artifacts caused by a change in a field of view when transitioning from the initial preview to the enlarged preview.

[0146] Block 1260 includes displaying the aligned magnified preview of the image captured by the second image capture device while operating within the second focal length range through the screen of the computing device.

[0147] In some embodiments, comparing the respective image features includes detecting one or more visual features in the initial preview and the enlarged preview. Such embodiments further include generating a visual match between the initial preview and the enlarged preview based on one or more visual features.

[0148] Some embodiments include optimizing detecting the one or more visual features and generating the visual match by performing asynchronous multi-threaded processing, including receiving one or more images and associated metadata as inputs and sending visual feature matches and associated metadata as outputs.

[0149] In some embodiments, updating the geometry-based warp transformation includes correcting frame-based geometric metadata based on the visual match.

[0150] In some embodiments, updating the geometry-based distortion transform includes estimating a homography from the corrected geometric metadata, and wherein the homography maps a pixel in a plane of a first coordinate system associated with the first image capture device to a corresponding pixel in the same plane of a second coordinate system associated with the second image capture device.

[0151] In some embodiments, updating the geometry-based warp transformation utilizes frame-based data, including one or more images, a pre-crop of the image, a scene depth, or a calibration parameter associated with the first image capture device and the second image capture device, respectively. In some embodiments, the calibration parameter includes an autofocus distance.

[0152] In some embodiments, applying the updated distortion transform to each frame of the initial preview and a corresponding frame of the enlarged preview is performed in a side-by-side comparison.

[0153] In some embodiments, aligning the magnified preview with the initial preview includes aligning a depth value of a point in image space with a geometric focus distance of the point on each frame of the initial preview and a corresponding frame of the magnified preview.

[0154] Some embodiments include generating a bundle adjustment to be applied to one or more camera calibrations and one or more focal lengths for each frame of the initial preview and a corresponding frame of the magnified preview.

[0155] Some embodiments include generating a modified bundle adjustment based on respective bundle adjustments of the consecutive frames for a collection of consecutive frames.

[0156] Some embodiments include transitioning from the first image capture device to the second image capture device by the computing device and based on the updated distortion transform.

[0157] In some embodiments, the second focal length range could be larger or smaller than the first focal length range, according to zoom-in or zoom-out operations on the computing device.

[0158] In some embodiments, the one or more viewing artifacts include binocular disparity.

[0159] In some embodiments, updating the geometry-based distortion transform includes reducing jitter by applying temporal feature tuning and tracking.

[0160] The specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of the elements shown in a given figure. Furthermore, some of the illustrated elements may be combined or omitted. Furthermore, an exemplary embodiment may include elements not illustrated in the figures.

[0161] A step or block representing information processing may correspond to circuitry that can be configured to perform the specific logical functions of a method or technique described herein. Alternatively or additionally, a step or block representing information processing may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more processor-executable instructions for implementing specific logical functions or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, for example, on a storage device such as a floppy disk, hard disk, or other storage medium.

[0162] The computer-readable medium may also include non-transitory computer-readable media, such as computer-readable media that stores data for short periods of time, such as register memory, processor cache, and random access memory (RAM). The computer-readable media may also include non-transitory computer-readable media that stores program code and / or data for longer periods of time. The computer-readable media may therefore include secondary or persistent long-term storage, such as read-only memory (ROM), optical or magnetic disks, or compact disc read-only memory (CD-ROM). The computer-readable media may also be any other volatile or non-transitory storage systems. A computer-readable medium may, for example, be considered a computer-readable storage medium or a tangible storage device.

[0163] Although various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various disclosed examples and embodiments are intended to be illustrative and not limiting, with the true scope being indicated by the following claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] WO 63 / 377,581

[0001]

Claims

[1] Computer-implemented method comprising: Displaying an initial preview of a scene captured by a first image capture device of the computing device through a screen of a computing device, the first image capture device operating within a first focal length range; detecting, by the computing device, a zoom operation that is expected to cause the first image capturing device to reach a limit of the first focal length range; in response to detecting, activating a second image capture device of the computing device to capture an enlarged preview of the scene, the second image capture device configured to operate within a second focal length range; Updating a geometry-based warp transformation based on a comparison of the respective image features from the initial preview and the enlarged preview; Aligning the enlarged preview with the initial preview by applying the updated distortion transformation, wherein the updated distortion transformation reduces one or more viewing artifacts caused by a change in a field of view when transitioning from the initial preview to the enlarged preview; and Displaying the aligned magnified preview of the image captured by the second image capturing device while operating within the second focal length range through the screen of the computing device. [2] The method of claim 1, wherein the comparison of the respective image features further comprises: Detecting one or more visual features in the initial preview and the enlarged preview; and Creating a visual match between the initial preview and the enlarged preview based on one or more visual features. [3] The method of claim 2, further comprising: Optimizing detecting the one or more visual features and generating the visual match by performing asynchronous multi-threaded processing, comprising receiving one or more images and associated metadata as inputs and sending visual feature matches and associated metadata as outputs. [4] The method of claim 2, wherein updating the geometry-based distortion transformation comprises: Correcting image-based geometric metadata based on visual matching. [5] The method of claim 4, wherein updating the geometry-based distortion transform comprises estimating a homography from the corrected geometric metadata, and wherein the homography maps a pixel in a plane of a first coordinate system associated with the first image capture device to a corresponding pixel in the same plane of a second coordinate system associated with the second image capture device. [6] The method of claim 1, wherein updating the geometry-based warp transform utilizes frame-based data comprising one or more images, a pre-crop of the image, a scene depth, or a calibration parameter associated with the first image capture device and the second image capture device, respectively. [7] The method of claim 6, wherein the calibration parameter comprises an autofocus distance. [8] The method of claim 1, wherein applying the updated distortion transform to each frame of the initial preview and a corresponding frame of the enlarged preview is performed in a side-by-side comparison. [9] The method of claim 1, wherein aligning the enlarged preview with the initial preview comprises: Aligning a depth value of a point in image space with a geometric focus distance of the point on each frame of the initial preview and a corresponding frame of the enlarged preview. [10] The method of claim 1, further comprising: Generating a bundle adjustment to be applied to one or more camera calibrations and one or more focal lengths for each frame of the initial preview and a corresponding frame of the enlarged preview. [11] The method of claim 10, further comprising: Generating a modified bundle adjustment based on respective bundle adjustments of the consecutive frames for a collection of consecutive frames. [12] The method of claim 1, further comprising: Transitioning from the first image capture device to the second image capture device by the computing device and based on the updated distortion transform. [13] The method of claim 1, wherein updating the geometry-based distortion transformation further comprises: Reducing jitter by applying temporal feature matching and tracking. [14] Computing device comprising: a display screen; a first image capture device configured to operate within a first focal length range; a second image capture device configured to operate within a second focal length range; one or more processors; and Data storage, wherein the data storage has computer-executable instructions stored thereon that, when executed by the one or more processors, cause the mobile device to perform functions including: Displaying an initial preview of a scene captured by the first image capture device on the display screen; detecting, by the computing device, a zooming operation that is likely to cause the first image capturing device to reach a limit of the first focal length range; in response to detecting, activating a second image capture device to capture an enlarged preview of the scene; Updating a geometry-based warp transformation based on a comparison of the respective image features from the initial preview and the enlarged preview; Aligning the enlarged preview with the initial preview by applying the updated distortion transformation, wherein the updated distortion transformation reduces one or more viewing artifacts caused by a change in a field of view when transitioning from the initial preview to the enlarged preview; and Displaying the aligned magnified preview of the image captured by the second image capturing device while operating within the second focal length range through the screen of the computing device. [15] The computing device of claim 14, wherein the functions for comparing the respective image features further comprise: Detecting one or more visual features in the initial preview and the enlarged preview; and Creating a visual match between the initial preview and the enlarged preview based on one or more visual features. [16] The computing device of claim 15, wherein the functions for updating the geometry-based distortion transformation further comprise: Correcting image-based geometric metadata based on visual matching. [17] The computing device of claim 16, wherein the functions for updating the geometry-based distortion transform comprise estimating a homography from the corrected geometric metadata, and wherein the homography maps a pixel in a plane of a first coordinate system associated with the first image capture device to a corresponding pixel in the same plane of a second coordinate system associated with the second image capture device. [18] The computing device of claim 14, wherein updating the geometry-based warp transform utilizes frame-based data comprising one or more images, a pre-crop of the image, a scene depth, or a calibration parameter associated with the first image capture device and the second image capture device, respectively. [19] The computing device of claim 14, wherein the functions for applying the updated distortion transform to each frame of the initial preview and a corresponding frame of the enlarged preview are performed in a side-by-side comparison. [20] The computing device of claim 19, wherein the functions for aligning the enlarged preview with the initial preview include: Aligning a depth value of a point in image space with a geometric focus distance of the point on each frame of the initial preview and a corresponding frame of the enlarged preview. [21] The computing device of claim 14, wherein the functions for updating the geometry-based distortion transformation further comprise: Reducing jitter by applying temporal feature matching and tracking. [22] A non-transitory computer-readable medium comprising program instructions executable by one or more processors to cause the one or more processors to perform operations comprising: Displaying an initial preview of a scene captured by the first image capture device on the display screen; detecting, by the computing device, a zooming operation that is likely to cause the first image capturing device to reach a limit of the first focal length range; in response to detecting, activating a second image capture device to capture an enlarged preview of the scene; Updating a geometry-based warp transformation based on a comparison of the respective image features from the initial preview and the enlarged preview; Aligning the enlarged preview with the initial preview by applying the updated distortion transformation, wherein the updated distortion transformation reduces one or more viewing artifacts caused by a change in a field of view when transitioning from the initial preview to the enlarged preview; and Displaying the aligned magnified preview of the image captured by the second image capturing device while operating within the second focal length range through the screen of the computing device.

Images