Spatial alignment transform without fov loss

Abstract

The present disclosure provides devices, methods, computer-readable media, and units for spatial alignment transformation. In some aspects, a device can perform operations including: capturing, using a first sensor, a first image of a scene at a first zoom ratio; capturing, using a second sensor, a second image of the scene at a second zoom ratio, the first image having a different field of view (FOV) than the second image; determining a transformation matrix based on one or more spatial misalignments between the first image and the second image; determining a confidence associated with the transformation matrix; in response to the confidence being greater than a confidence threshold, determining a weighting factor based on the first zoom ratio, the second zoom ratio, and a current zoom ratio of the device; applying the weighting factor to the transformation matrix; and warping the first image to the second image using the weighted transformation matrix.

Description

[0001] Cross-references to related applications

[0002] This patent application claims priority to U.S. Patent Application No. 17 / 094,052, filed November 10, 2020, entitled “SPATIAL ALIGNMENT TRANSFORM WITHOUT FOV LOSS,” which has been assigned to the assignee of this application. All disclosures of the prior applications are considered part of this patent application and are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to image or video capture devices, including image frames from multiple image sensors processed by an image signal processor, for example, for spatial alignment transformation of the image frames. Background Technology

[0004] Many devices include multiple image sensors that can be used to capture one or more image frames. For example, a smartphone or tablet computer includes multiple image sensors for generating images or videos for different imaging applications. One or more image signal processors can process image frames from multiple image sensors. The processed image frames can be used by imaging applications, such as for generating preview images, user photos, recording videos, performing augmented reality operations, etc. Summary of the Invention

[0005] The systems, methods, and apparatus disclosed herein each have several innovative aspects, none of which are solely responsible for the desired properties disclosed herein.

[0006] One innovative aspect of the subject matter described in this disclosure can be implemented in a device. The device may include a processor and a memory coupled to the processor and storing instructions that, when executed by the processor, cause the device to perform operations. The operations may include: capturing a first image of a scene at a first zoom ratio using a first sensor; capturing a second image of the scene at a second zoom ratio using a second sensor, the first image having a different field of view (FOV) than the second image; determining a transformation matrix based on one or more spatial misalignments between the first and second images; determining a confidence level associated with the transformation matrix; determining a weighting factor based on the first zoom ratio, the second zoom ratio, and a current zoom ratio of the device in response to a confidence level greater than a confidence threshold; applying the weighting factor to the transformation matrix; and distorting the first image to the second image using the weighted transformation matrix.

[0007] In some implementations, the first and second images are captured simultaneously when the current zoom ratio lies within an overlapping region spanning between the first and second zoom ratios. In some instances, the weighting factor is 0% when the current zoom ratio equals the first zoom ratio, and 100% when the current zoom ratio equals the second zoom ratio. In some other instances, the weighting factor corresponds to the identity matrix when the confidence level is no greater than a confidence threshold. In some implementations, the first image has a wider field of view (FOV) than the second image. In some instances, the first sensor is a wide-angle image sensor, and the second sensor is a long-range image sensor.

[0008] In some implementations, the execution of the instructions causes the device to perform the following operations: gradually increasing the weighting factor as the current zoom ratio increases from a first zoom ratio to a second zoom ratio; and gradually decreasing the weighting factor as the current zoom ratio decreases towards the first zoom ratio within the overlapping region. In some other implementations, the execution of the instructions causes the device to perform the following operations: generating a preview image based on the distorted first and second images; and displaying the preview image to the user of the device.

[0009] In some implementations, the execution of the instructions causes the device to perform operations including: scaling the first image to match the FOV of the second image; and identifying one or more spatial misalignments between the scaled first and second images. In some other implementations, the execution of the instructions causes the device to perform operations including: avoiding the determination of an additional transformation matrix when the confidence level is greater than a confidence threshold.

[0010] Another innovative aspect of the subject matter described in this disclosure can be implemented as a method for spatial alignment transformation. The method can be performed by a device and may include: capturing a first image of a scene at a first zoom ratio using a first sensor of the device; capturing a second image of the scene at a second zoom ratio using a second sensor of the device, the first image having a different field of view (FOV) than the second image; determining a transformation matrix based on one or more spatial misalignments between the first and second images; determining a confidence level associated with the transformation matrix; determining a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device in response to a confidence level greater than a confidence threshold; applying the weighting factor to the transformation matrix; and distorting the first image to the second image using the weighted transformation matrix.

[0011] In some implementations, the first and second images are captured simultaneously when the current zoom ratio lies within an overlapping region spanning between the first and second zoom ratios. In some instances, the weighting factor is 0% when the current zoom ratio equals the first zoom ratio, and 100% when the current zoom ratio equals the second zoom ratio. In some other instances, the weighting factor corresponds to the identity matrix when the confidence level is no greater than a confidence threshold. In some implementations, the first image has a wider field of view (FOV) than the second image. In some instances, the first sensor is a wide-angle image sensor, and the second sensor is a long-range image sensor.

[0012] In some implementations, the method further includes: gradually increasing the weighting factor as the current zoom ratio increases from a first zoom ratio to a second zoom ratio; and gradually decreasing the weighting factor as the current zoom ratio decreases towards the first zoom ratio within the overlapping region. In some other implementations, the method further includes: generating a preview image based on a distorted first image and a second image; and displaying the preview image to a user of the device.

[0013] In some implementations, the method further includes: scaling the first image to match the FOV of the second image; and identifying one or more spatial misalignments between the scaled first and second images. In some other implementations, the method further includes: avoiding determining an additional transformation matrix when the confidence level is greater than a confidence threshold.

[0014] Another innovative aspect of the subject matter described in this disclosure can be implemented in a non-transitory computer-readable medium. This non-transitory computer-readable medium can store instructions that, when executed by a processor of the device, cause the device to perform operations. In some implementations, the operations include: capturing a first image of a scene using a first sensor at a first zoom ratio; capturing a second image of the scene using a second sensor at a second zoom ratio, the first image having a different field of view (FOV) than the second image; determining a transformation matrix based on one or more spatial misalignments between the first and second images; determining a confidence level associated with the transformation matrix; determining a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device in response to a confidence level greater than a confidence threshold; applying the weighting factor to the transformation matrix; and distorting the first image to the second image using the weighted transformation matrix.

[0015] In some implementations, the first and second images are captured simultaneously when the current zoom ratio lies within an overlapping region spanning between the first and second zoom ratios. In some instances, the weighting factor is 0% when the current zoom ratio equals the first zoom ratio, and 100% when the current zoom ratio equals the second zoom ratio. In some other instances, the weighting factor corresponds to the identity matrix when the confidence level is no greater than a confidence threshold. In some implementations, the first image has a wider field of view (FOV) than the second image. In some instances, the first sensor is a wide-angle image sensor, and the second sensor is a long-range image sensor.

[0016] In some implementations, the operation further includes: gradually increasing the weighting factor as the current zoom ratio increases from a first zoom ratio to a second zoom ratio; and gradually decreasing the weighting factor as the current zoom ratio decreases towards the first zoom ratio within the overlapping region. In some other implementations, the operation further includes: generating a preview image based on the distorted first and second images; and displaying the preview image to a user of the device.

[0017] In some implementations, the operation further includes: scaling the first image to match the FOV of the second image; and identifying one or more spatial misalignments between the scaled first and second images. In some other implementations, the operation further includes: avoiding determining an additional transformation matrix when the confidence level is greater than a confidence threshold.

[0018] Another innovative aspect of the subject matter described in this disclosure can be implemented in a device. The device may include: a unit for capturing a first image of a scene using a first sensor at a first zoom ratio; a unit for capturing a second image of the scene using a second sensor at a second zoom ratio, the first image having a different field of view (FOV) than the second image; a unit for determining a transformation matrix based on one or more spatial misalignments between the first and second images; a unit for determining a confidence level associated with the transformation matrix; a unit for determining a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device in response to a confidence level greater than a confidence threshold; a unit for applying the weighting factor to the transformation matrix; and a unit for distorting the first image to the second image using the weighted transformation matrix.

[0019] In some implementations, the first and second images are captured simultaneously when the current zoom ratio lies within an overlapping region spanning between the first and second zoom ratios. In some instances, the weighting factor is 0% when the current zoom ratio equals the first zoom ratio, and 100% when the current zoom ratio equals the second zoom ratio. In some other instances, the weighting factor corresponds to the identity matrix when the confidence level is no greater than a confidence threshold. In some implementations, the first image has a wider field of view (FOV) than the second image. In some instances, the first sensor and the second sensor are a wide-angle image sensor and a long-range image sensor, respectively.

[0020] In some implementations, the device further includes units for: gradually increasing a weighting factor as the current zoom ratio increases from a first zoom ratio to a second zoom ratio; and gradually decreasing the weighting factor as the current zoom ratio decreases towards the first zoom ratio within the overlapping region. In some other implementations, the device further includes units for: generating a preview image based on a distorted first image and a second image; and displaying the preview image to a user of the device.

[0021] In some implementations, the device further includes units for: scaling the first image to match the FOV of the second image; and identifying one or more spatial misalignments between the scaled first and second images. In some other implementations, the operation further includes: avoiding determining an additional transformation matrix when the confidence level is greater than a confidence threshold.

[0022] Details of one or more implementations of the subject matter described in this disclosure are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the specification, drawings, and claims. Note that the relative dimensions in the following drawings may not be drawn to scale. Attached Figure Description

[0023] Figure 1 A block diagram of an example device for performing spatial alignment transformation of image frames is shown.

[0024] Figure 2 A timing diagram is shown of an example device performing spatial alignment transformation of an image frame.

[0025] Figure 3 An example data stream is shown where the device performs a spatial alignment transformation on one or more image frames.

[0026] Figure 4A This illustrates an example of an overlapping area spanning between the device's first and second zoom ratios as the device's current zoom ratio increases.

[0027] Figure 4BA block diagram is shown depicting an example operation of the device generating a transformation matrix as the device's current zoom ratio increases.

[0028] Figure 5A An example overlapping region is shown as the device's current zoom ratio decreases, spanning between the device's first zoom ratio and second zoom ratio.

[0029] Figure 5B A block diagram is shown depicting an example operation of the device generating a transformation matrix as the device's current zoom ratio decreases.

[0030] Figure 6 A flowchart depicting an example operation of spatial alignment transformation is shown.

[0031] Figure 7 A flowchart depicting another example operation of spatial alignment transformation is shown.

[0032] Figure 8 A flowchart depicting another example operation of spatial alignment transformation is shown.

[0033] Figure 9 A flowchart depicting another example operation of spatial alignment transformation is shown.

[0034] Figure 10 A flowchart depicting another example operation of spatial alignment transformation is shown.

[0035] Similar reference numerals and names in different figures denote similar elements. Detailed Implementation

[0036] Various aspects of this disclosure can be used to capture image frames using multiple image sensors of a device, such as a combination of ultra-wide (high field of view (FOV), wide, long-range, and ultra-long-range (low FOV) sensors. Some aspects include processing the captured image frames, for example by performing a spatial alignment transformation on one or more captured image frames as the device transitions from capturing an image of a scene using a first image sensor among the multiple image sensors to capturing an image of the scene using a second image sensor among the multiple image sensors.

[0037] Example devices (e.g., smartphones) may include a configuration of two, three, four, or more cameras on the back of the device. Devices with multiple image sensors include one or more image signal processors, computer vision processors (CVPs), or other suitable circuitry for processing images captured by the image sensors. One or more image signal processors may provide processed image frames to memory and / or a processor (e.g., an application processor, image front-end (IFE), image processing engine (IPE), or other suitable processing circuitry) for further processing, such as encoding or other manipulation.

[0038] As used herein, an image sensor can refer to the image sensor itself and any other suitable component coupled to the image sensor. For example, an image sensor can also refer to other components of a camera, including the shutter, buffer, or other readout circuitry. An image sensor can also refer to an analog front-end or other circuitry used to convert analog signals into a digital representation of frames. Therefore, the term "image sensor" as used herein can refer to any suitable component used to capture image frames and read them out to an image signal processor.

[0039] An example multi-sensor device can use a wide sensor to capture a "wide" image of the scene when the device's current zoom ratio is a first value, and can switch to using a remote sensor to capture a "remote" image of the scene when the current zoom ratio is a second, higher value. When the current zoom ratio is within a defined range of values, the device can use two sensors to capture images of the scene simultaneously or nearly simultaneously; this range may be referred to herein as the "overlapping region." The device can use image data from one or more sensors to generate, for example, a preview image of the scene for display to the device user.

[0040] However, due to manufacturing defects, multiple sensors and / or captured images may be spatially misaligned, which can lead to misalignment errors associated with the generated image when the device's current zoom ratio is within the overlapping area. Misalignment errors can cause visual defects in the corresponding preview image. Some devices can use a transformation matrix to warp (e.g., shift and / or rotate) the image to spatial alignment, which can resolve some misalignment errors. Unfortunately, in doing so, conventional devices crop out portions of the image with a higher FOV and generate a corresponding image with a reduced FOV and lower quality (i.e., fewer pixels).

[0041] Various aspects of this disclosure provide a multi-sensor device that can smoothly transition from an image of a scene captured using a first sensor to an image of a scene captured using a second sensor by distorting one or more captured images, without reducing FOV or quality when generating the corresponding image.

[0042] In the following description, numerous specific details, such as examples of specific components, circuits, and processes, are set forth to provide a thorough understanding of this disclosure. The term "coupling" as used herein means a direct connection or a connection via one or more intermediate components or circuits. Furthermore, specific terms are set forth in the following description and for illustrative purposes to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that these specific details may not be necessary to practice the teachings of this disclosure. In other instances, well-known circuits and devices are illustrated in block diagram form to avoid obscuring the teachings of this disclosure. Some portions of the following detailed description are presented in the form of other symbolic representations of processes, logic blocks, operations, and operations on data bits in computer memory. In this disclosure, processes, logic blocks, procedures, etc., are considered as a self-consistent sequence of steps or instructions that lead to a desired result. These steps are those that require physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated in a computer system.

[0043] However, it should be remembered that all these and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise clearly stated in the following discussion, it should be understood that throughout this application, the use of terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” “solve,” “generate,” etc., refers to the operation and process of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the registers and memories of the computer system and transforms it into other data similarly represented as physical quantities in the registers, memories, or other such information storage, transmission, or display devices of the computer system.

[0044] In the figures, a single box can be described as performing one or more functions. However, in practice, the one or more functions performed by that box may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are described below generally according to their functions. Whether such a function is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functions in different ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure. Furthermore, the example device may include components other than those shown, including well-known components such as processors, memory, etc.

[0045] The aspects of this disclosure are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or “frames”). Furthermore, the aspects of this disclosure can be implemented in image sensors or devices coupled to image sensors having the same or different capabilities and characteristics (e.g., resolution, shutter speed, sensor type, etc.).

[0046] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that implement at least some portions of this disclosure. Although the following description and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus may include a device or part of a device for performing the operations described herein.

[0047] Figure 1 A block diagram of an example device 100 for performing spatial alignment transformation of image frames is shown. Device 100 may include or be otherwise coupled to an image signal processor 112 for processing image frames from multiple image sensors (e.g., a first image sensor 101 and a second image sensor 102). In some implementations, device 100 may also include or be coupled to a processor 104 and a memory 106 for storing instructions 108. Device 100 may also include or be coupled to a display 114 and multiple input / output (I / O) components 116. Device 100 may also include or be coupled to a power supply 118 for device 100, such as a battery or a component that couples device 100 to an energy source. Device 100 may also include or be coupled to additional features or components not shown. In one example, a wireless interface for a wireless communication device may be included, which may include multiple transceivers and a baseband processor. In another example, one or more other sensors (e.g., a gyroscope or a Global Positioning System (GPS) receiver) may be included in or coupled to the device. In another example, an analog front-end for converting analog image frame data into digital image frame data may be coupled between image sensors 101 and 102 and image signal processor 112.

[0048] First image sensor 101 and second image sensor 102 are configured to capture one or more image frames. For example, first image sensor 101 and second image sensor 102 may be included in a multi-camera configuration, or in a single-camera or single-camera configuration (e.g., a dual-camera configuration, a triple-camera configuration, etc., for a smartphone or other suitable device). Image sensors 101 and 102 may also include or be coupled to one or more lenses for focusing light, one or more apertures for receiving light, one or more shutters for blocking light outside an exposure window, one or more color filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front-ends for converting analog measurements into digital information, or other suitable components for imaging. Device 100 may also include a flash, a depth sensor, GPS, or other suitable components for imaging.

[0049] Image signal processor 112 processes image frames captured by image sensors 101 and 102. Although Figure 1 The device 100 is shown to include two image sensors 101 and 102 coupled to an image signal processor 112, but any number of image sensors can be coupled to the image signal processor 112. Furthermore, any number of additional image sensors or image signal processors can be present in the device 100.

[0050] In some aspects, the image signal processor 112 may execute instructions from memory, such as instructions 108 from memory 106, instructions stored in a separate memory coupled to or contained in the image signal processor 112, or instructions provided by processor 104. Alternatively or additionally, the image signal processor 112 may execute software and / or may include specific hardware (e.g., one or more integrated circuits (ICs)) to perform one or more operations described in this disclosure.

[0051] In some implementations, memory 106 may include a non-transient or non-temporary computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more of the operations described in this disclosure. In some implementations, instructions 108 include a camera application (or other suitable application) to be executed by device 100 for generating images or videos. Instructions 108 may also include other applications or programs executed by device 100, such as an operating system and specific applications other than those for image or video generation. Execution of the camera application (e.g., by processor 104) may cause device 100 to generate images using an image sensor and image signal processor 112. Memory 106 may also be accessed by image signal processor 112 to store processed frames, or may be accessed by processor 104 to obtain processed frames. In some other implementations, device 100 does not include memory 106. For example, device 100 may be circuitry including image signal processor 112, and the memory may be external to device 100. Device 100 may be coupled to and configured to access the memory to write processed frames.

[0052] In some implementations, processor 104 may include one or more general-purpose processors capable of executing scripts or instructions (e.g., instructions 108 stored in memory 106) of one or more software programs. For example, processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in memory 106. When executing the camera application, processor 104 may be configured to instruct image signal processor 112 to perform one or more operations with reference to image sensor 101 or 102. Execution of instructions 108 by processor 104 outside the camera application may also cause device 100 to perform any number of functions or operations. In some implementations, in addition to the ability to execute software to cause device 100 to perform multiple functions or operations (e.g., the operations described herein), processor 104 may also include ICS or other hardware. In some other implementations, device 100 does not include processor 104.

[0053] In some implementations, at least one of the image signal processor 112 or processor 104 can execute instructions to perform the various operations described herein. For example, the execution of the instructions can determine a transformation matrix based on one or more spatial misalignments between a first image of a scene captured using the first image sensor 101 at a first zoom ratio and a second image of a scene captured using the second image sensor 102 at a second zoom ratio. In some aspects, the first image may have a different FOV than the second image. The execution of the instructions can also determine a confidence level associated with the transformation matrix. The execution of the instructions can also determine a weighting factor based on the device's first zoom ratio, second zoom ratio, and current zoom ratio. Thereafter, the execution of the instructions can apply the weighting factor to the transformation matrix and use the weighted transformation matrix to distort the first image to the second image.

[0054] In some implementations, display 114 may include one or more suitable displays or screens that allow user interaction and / or present items to the user (e.g., previews of image frames captured by image sensors 101 and 102). In some aspects, display 114 is a touch-sensitive display. I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (e.g., commands) from the user and provide output to the user. For example, I / O component 116 may include (but is not limited to) a graphical user interface (GUI), keyboard, mouse, microphone, speaker, squeezable bezel, one or more buttons (e.g., power button), slider, switch, etc.

[0055] Although shown coupled to each other via processor 104, processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. Although image signal processor 112 is shown separate from processor 104, image signal processor 112 may be the core of processor 104, which is an application processor unit (APU) included in a system-on-a-chip (SoC), or otherwise included within processor 104. While device 100 is referenced in the examples herein to perform aspects of this disclosure, some device components may not be included. Figure 1 The details are shown to prevent obscuring aspects of this disclosure. Additionally, other components, quantities of components, or combinations of components may be included in suitable devices for performing aspects of this disclosure. Therefore, this disclosure is not limited to the configuration of a particular device or component, including device 100.

[0056] Figure 2 A timing diagram 200 is shown illustrating the spatial alignment transformation of an image frame performed by an example device. The example device could be... Figure 1 The example implementation of device 100 shown is illustrated. (See reference...) Figure 1 The devices and components shown in timing diagram 200 are described, but any suitable device or device component can perform the operations described in timing diagram 200. The operations shown in timing diagram 200 can be performed when the current zoom ratio of device 100 is within an overlapping region spanning between a first zoom ratio and a second zoom ratio, as described in more detail below. In some implementations, the first image sensor 101 is a wide-angle image sensor, and the second image sensor 102 is a remote image sensor.

[0057] Image signal processor 112 triggers first image sensor 101 and second image sensor 102 to capture a first image and a second image, respectively (202). In some implementations, the first image may have a wider field of view (FOV) than the second image. Although timing diagram 200 shows triggers being provided to image sensors 101 and 102 simultaneously, the triggers may be provided at different times or at any suitable time. Upon receiving a trigger, first image sensor 101 may capture the first image at a first zoom ratio (204), and second image sensor 102 may capture the second image at a second zoom ratio (206). In some implementations, the image sensors may capture images simultaneously or approximately simultaneously.

[0058] The second image sensor 102 can send a second image to the image signal processor 112 (208), and the first image sensor 101 can send a first image to the image signal processor 112 (210). In some cases, the image sensors can send the captured images to the processor simultaneously or approximately simultaneously. The image signal processor 112 receives the first image and the second image from the first image sensor 101 and the second image sensor 102 (212), respectively. In some cases, the image signal processor 112 can receive the images simultaneously or approximately simultaneously.

[0059] Image signal processor 112 may determine a transformation matrix (214) based on one or more spatial misalignments between a first image and a second image. In some implementations not shown in timing diagram 200, image signal processor 112 scales the first image to match the FOV of the second image and identifies one or more spatial misalignments between the scaled first and second images. Image signal processor 112 may determine a confidence level (216) associated with the transformation matrix and compare the confidence level to a confidence threshold (218). In some implementations, the confidence threshold may be a predetermined value and / or an adjustable value.

[0060] In response to a confidence level greater than a confidence threshold, image signal processor 112 may determine a weighting factor (220) based on a first zoom ratio, a second zoom ratio, and a current zoom ratio of device 100. In some implementations, the weighting factor may be 0% when the current zoom ratio equals the first zoom ratio, and 100% when the current zoom ratio equals the second zoom ratio. In some cases, image signal processor 112 may avoid determining an additional transformation matrix when the confidence level is greater than the confidence threshold. In some other implementations, the weighting factor may be an identity matrix when the confidence level is not greater than the confidence threshold.

[0061] Image signal processor 112 can then apply a weighting factor to the transformation matrix (222). In some implementations, image signal processor 112 can increase the weighting factor as the current zoom ratio increases towards the second zoom ratio in the overlapping region, and decrease the weighting factor as the current zoom ratio decreases towards the first zoom ratio in the overlapping region. Image signal processor 112 can send the weighted transformation matrix to processor 104 (224), and processor 104 can then receive the weighted transformation matrix (226). Processor 104 can then use the weighted transformation matrix to distort the first image to the second image (228). In some cases, device 100 can generate a preview image based on the distorted first and second images and display the preview image to the user of device 100.

[0062] The operation described with reference to timing diagram 200 can be extended to any number of image sensors coupled to the image signal processor and any number of frames to be provided to the image signal processor. Therefore, timing diagram 200 shows only two image sensors for the purpose of clearly explaining various aspects of this disclosure; that is, this disclosure is not limited to a specific number of image sensors. For example, in some implementations, device 100 may have some combination of ultra-wide sensors, wide sensors, long-range sensors, ultra-long-range sensors, and any other suitable type of sensor.

[0063] Figure 3 An example data stream 300 is shown where a device performs a spatial alignment transformation on one or more image frames. The device may be... Figure 1 The illustrated device 100 is an example implementation. While data flow 300 is described with reference to the devices, components, and operations shown in the preceding figures, any suitable device or device component can perform the operations described with respect to data flow 300. In some implementations, the operations shown in data flow 300 can be performed when the current zoom ratio of device 100 is within the overlapping region.

[0064] The first image 302 and the second image 312 are captured by the first image sensor and the second image sensor of the device 100, respectively, and are not shown for simplicity. In some implementations, the first image sensor and the second image sensor can be respectively Figure 1 Example implementations of the first image sensor 101 and the second image sensor 102. As a non-limiting example, the first image 302 may be a wide-angle image captured at a first zoom ratio (e.g., 4.9X), and the second image 312 may be a long-range image captured at a second zoom ratio (e.g., 5.0X). In some implementations, the first image 302 and the second image 312 may be captured at an initial resolution (e.g., 2304×1728) higher than the final output resolution of the image (e.g., 1920×1440). Figure 3 As shown, device 100 can crop the image to a smaller resolution (e.g., 480×360) for further processing at image signal processor 112.

[0065] In some implementations, the image signal processor 112 scales the first image to match the field of view (FOV) of the second image and identifies one or more spatial misalignments between the scaled first and second images. For example... Figure 3 As shown, the image signal processor 112 determines a transformation matrix based on one or more spatial misalignments, determines a confidence level associated with the transformation matrix, compares the confidence level with a confidence threshold, determines a weighting factor in response to a confidence level greater than the confidence threshold, and applies the weighting factor to the transformation matrix, as shown regarding... Figure 2 The image signal processor 112 can then send the weighted transformation matrix to the processor 104 (324).

[0066] Processor 104 may receive a first image (326) at an initial resolution and distort the first image to a second image using a weighted transformation matrix from image signal processor 112. In some aspects, the final output resolution of the distorted first image (e.g., 1920 × 1440) may be smaller than the initial resolution of the first image 302 (e.g., 2304 × 1728). In this way, the first image 302 may have a pixel margin between the final output resolution and the initial resolution. Therefore, when processor 104 distorts (e.g., shifts and / or rotates) the first image 302 to the second image, only pixels within the pixel margin can be shifted or rotated out of frame, so processor 104 may, for example, output the distorted image (330) to the display of device 100 without any FOV or quality loss, since only pixels within the pixel margin can be shifted or rotated out of frame. In some implementations, device 100 may then generate a preview image based on the distorted first and second images 312.

[0067] Figure 4A An example overlap region 400 is shown, spanning between the device's first and second zoom ratios, as the device's current zoom ratio increases. The device can be... Figure 1 The illustrated device 100 is an example implementation. While the overlapping region 400 is described with reference to the devices, components, and operations shown in the preceding figures, any suitable device or device component can perform actions related to... Figure 4A The described operation. In some implementations, the first sensor is... Figure 1 An example implementation of the first image sensor 101, which can be a wide-angle image sensor (or "wide sensor"), and the second image sensor is... Figure 1 An example implementation of the second sensor 102, which may be a remote image sensor (or "remote sensor").

[0068] When the current zoom ratio of device 100 is within the first sensor region 402, device 100 can capture only a wide image using only the wide sensor. In some implementations, the first sensor region 402 can be derived from... Figure 4A It begins at the far left (e.g., a zoom ratio of 1.0X) and ends at the "first zoom ratio" (e.g., a zoom ratio of 2.7X). Conversely, when the current zoom ratio of device 100 is within the second sensor region 404, device 100 can capture only remote images using only the remote sensor. In some implementations, the second sensor region 404 may begin at the second zoom ratio (e.g., a zoom ratio of 3.0X) and end at the far left. Figure 4A It ends at the far right (for example, zoom ratio 10.0X).

[0069] When the current zoom ratio of device 100 is within the overlapping region (e.g., a zoom ratio between 2.7X and 3.0X), device 100 can capture a wide image using a wide sensor while simultaneously capturing a remote image using a remote sensor, as per [the relevant information]. Figure 2 The device 100 can determine that at least one of the captured wide images is spatially misaligned with at least one of the captured remote images, for example, if the wide sensor and the remote sensor are spatially misaligned. Therefore, as described above, the device 100 can determine a transformation matrix M based on the spatial misalignment between the wide image and the remote image, for distorting the wide image to spatial alignment with the remote image, determine a confidence level associated with M, and compare the confidence level with a confidence threshold to determine whether M can be used to align the wide image and the remote image with a level of accuracy higher than a certain value.

[0070] In some implementations, when the confidence level is not greater than a confidence threshold, the weighting factor r applied to M can be an identity matrix. In such implementations, for example, when the current zoom ratio changes, device 100 can generate an additional transformation matrix based on identifying spatial misalignment between the additional wide image and the long-range image. In some implementations, when the confidence level is greater than a confidence threshold, device 100 can avoid determining the additional transformation matrix.

[0071] In response to a confidence level greater than a confidence threshold, device 100 can determine r based on a first zoom ratio, a second zoom ratio, and the current zoom ratio. Device 100 can then apply r to M to generate a weighted transformation matrix M′, and use M′ to distort the wide image to the remote image. In some implementations, device 100 can increase r as the current zoom ratio increases within the overlap region 400. For example, r could be 0% when the current zoom ratio equals the first zoom ratio, and 100% when the current zoom ratio equals the second zoom ratio. In this way, device 100 can distort the wide image throughout the entire overlap region 400 using the same (weighted) transformation matrix M.

[0072] In some implementations, device 100 can increase r proportionally to the increase in the current zoom ratio within the overlapping region 400. For illustration, Figure 4A Examples include two vertical lines between the first zoom ratio and the second zoom ratio to illustrate three equal portions within the overlapping region 400. As a non-limiting example, the first zoom ratio could be 2.7X, the left vertical line could represent a zoom ratio of 2.8X, the right vertical line could represent a zoom ratio of 2.9X, and the second zoom ratio could be 3.0X. Therefore, when the current zoom ratio is 2.7X, device 100 can calculate r as 0%; 33% when the current zoom ratio is 2.8X; 66% when the current zoom ratio is 2.9X; and 100% when the current zoom ratio is 3.0X. More specifically, device 100 can calculate r as:

[0073]

[0074] In some implementations, FirstZoomRatio can represent the zoom ratio at which device 100 begins to transition from capturing an image of the scene using a first sensor to capturing an image of the scene using a smaller second FOV sensor. In such an implementation, the first zoom ratio can be considered the beginning of the transition region. In this example, SecondZoomRatio can represent the zoom ratio at which device 100 completes the transition from capturing an image of the scene using the first sensor to capturing an image of the scene using the smaller second FOV sensor, and can be considered the end of the transition region (or "optical zoom ratio").

[0075] exist Figure 4A In the example, FirstZoomRatio can be 2.7X, and SecondZoomRatio can be 3.0X, which can represent a zoom ratio of 1.0X from the perspective of a smaller FOV (e.g., a remote) sensor. Therefore, if CurrentZoomRatio is 2.8X, device 100 can calculate r as follows:

[0076]

[0077] In some implementations, device 100 can then weight M according to the following formula, where I is the identity matrix. And M′ n M' represents the weighted transformation matrix M' at the nth position within the overlapping region:

[0078] M′ n =|r*M n +(1-r)*I|

[0079] In this example, where r = 33%, device 100 can deliver M′ n The calculation is as follows:

[0080]

[0081] Then, device 100 can use M′ n The wide image is distorted to a remote image. By dynamically increasing r as the current zoom ratio increases, device 100 can distort the wide image to a remote image using the same (dynamically weighted) transformation matrix without reducing its FOV (e.g., 2.7X) across the entire overlapping region 400. In this way, device 100 can convert from an image of a scene captured using a wide sensor to an image of a scene captured using a remote sensor without degrading, for example, the FOV or quality of a preview image generated based on the remote image and the distorted wide image.

[0082] As Figure 4A Another illustrative example is that if the current ZoomRatio is 2.7X, device 100 can calculate r and M′ as follows. n :

[0083]

[0084] M′ n =|0%*M n +(100%-0%)*I|=I

[0085] As Figure 4AAnother illustrative example is that if the current ZoomRatio is 3.0X, device 100 can calculate r and M′ as follows. n :

[0086]

[0087] M′ n =|100%*M n +(100%-100%)*I|=M n

[0088] In some implementations not shown, the device may have two or more sensors, such as an ultra-wide sensor, a wide sensor, a long-range sensor, and an ultra-long-range sensor. In such implementations, as the current zoom ratio increases by crossing a "first overlap region" from a specific zoom ratio within the ultra-wide sensor zoom range to a specific zoom ratio within the wide sensor zoom range, the device can utilize the above-described process to perform a smooth transition between sensors. Similarly, as the current zoom ratio increases by crossing a "second overlap region" from a specific higher zoom ratio within the wide sensor zoom range to a specific zoom ratio within the long-range sensor zoom range, the device can utilize the above-described process to perform a smooth transition between sensors. Similarly, as the current zoom ratio increases by crossing a "third overlap region" from a specific higher zoom ratio within the long-range sensor zoom range to a specific zoom ratio within the ultra-long-range sensor zoom range, the device can utilize the above-described process to perform a smooth transition between sensors.

[0089] Figure 4B A block diagram 450 illustrates an example operation of the device generating a transformation matrix as the device's current zoom ratio increases. The device can be... Figure 1 The illustrated device 100 is an example implementation. Although block diagram 450 is described with reference to the devices, components, and operations shown in the preceding figures, any suitable device or device component may perform the operations described with respect to block diagram 450.

[0090] like Figure 4BAs shown, the image signal processor 112 can receive a first image captured at a first zoom ratio (e.g., 2.7X) and a second image captured at a second zoom ratio (e.g., 3.0X). In some implementations, the first image may be a wide image captured using a wide sensor, and the second image may be a remote image captured using a remote sensor. In some implementations, the device 100 may adjust (e.g., crop and / or scale) the wide image to match the FOV of the remote image and generate a transformation matrix M based on the adjusted wide image (e.g., at a 3.0X FOV) and the remote image. In some cases, the device 100 may also adjust (e.g., crop and / or scale) the remote image to the second zoom ratio (e.g., 3.0X in this example).

[0091] Figure 5A An example overlap region 500 is shown spanning between the device's first and second zoom ratios as the device's current zoom ratio decreases. The device may be... Figure 1 The illustrated device 100 is an example implementation. While the overlapping region 500 is described with reference to the devices, components, and operations shown in the preceding figures, any suitable device or device component can perform actions related to... Figure 5A The described operation. In some implementations, the first sensor is... Figure 1 An example implementation of the first image sensor 101, which can be a wide-angle image sensor (or "wide sensor"), and the second image sensor is... Figure 1 An example implementation of the second sensor 102, which may be a remote image sensor (or "remote sensor").

[0092] Similar to Figure 4A When the current zoom ratio of device 100 is within the first sensor region 502, device 100 can capture only a wide image using only the wide sensor; when the current zoom ratio of device 100 is within the second sensor region 504, device 100 can capture only a remote image using only the remote sensor; and when the current zoom ratio of device 100 is within the overlap region 500 (2.7X-3.0X in this example), device 100 can capture a wide image using the wide sensor while simultaneously capturing a remote image using the remote sensor. (See also: Regarding...) Figure 4A The device 100 can determine a transformation matrix M for distorting a wide image to spatial alignment with a remote image, and can determine a weighting factor r for weighting M to generate a weighted transformation matrix M′.

[0093] and Figure 4AConversely, in some implementations, device 100 can decrease r as the current zoom ratio decreases within the overlapping region 500. As a non-limiting example, the first zoom ratio could be 2.7X, the left vertical line could represent a zoom ratio of 2.8X, the right vertical line could represent a zoom ratio of 2.9X, and the second zoom ratio could be 3.0X. Therefore, in conjunction with reference to... Figure 4A In the opposite manner described, device 100 can calculate r as 0% at the current zoom ratio of 3.0X, 33% at the current zoom ratio of 2.9X, 2.8X at the current zoom ratio of 66%, and 100% at the current zoom ratio of 2.7X, thereby generating M′ accordingly. n Then use M′ n Distort a wide image to a remote image.

[0094] Device 100 can then use M′ n The wide image is distorted to the remote image. By dynamically decreasing r as the current zoom ratio increases, device 100 can distort the wide image to the remote image using the same (dynamically weighted) transformation matrix without reducing its FOV (e.g., 2.7X) across the entire overlapping region 500. In this way, device 100 can smoothly transform from an image of a scene captured using a wide sensor to an image of a scene captured using a remote sensor without degrading, for example, the FOV or quality of a preview image generated based on the remote image and the distorted wide image.

[0095] Figure 5B A block diagram 550 is shown depicting an example operation of the device generating a conversion matrix when the device's current zoom ratio decreases. The device can be... Figure 1 The illustrated device 100 is an example implementation. Although block diagram 550 is described with reference to the devices, components, and operations shown in the preceding figures, any suitable device or device component can perform the operations described with respect to block diagram 550.

[0096] like Figure 5B As shown, the image signal processor 112 can receive a first image captured at a first zoom ratio (e.g., 2.7X) and a second image captured at a second zoom ratio (e.g., 3.0X). In some implementations, the first image may be a wide image captured using a wide sensor, and the second image may be a remote image captured using a remote sensor. In some implementations, the device 100 may adjust (e.g., crop and / or scale) the wide image to match the FOV of the remote image and generate a transformation matrix M based on the adjusted wide image (e.g., at a 3.0X FOV) and the remote image.

[0097] Figure 6 A flowchart depicting an example operation 600 for spatial alignment transformation is shown. Operation 600 can be performed by, for example... Figure 1The device 100 shown performs the following actions: At block 602, the device uses a first sensor to capture a first image of the scene at a first zoom ratio. At block 604, the device uses a second sensor to capture a second image of the scene at a second zoom ratio, the first image having a different field of view (FOV) than the second image. At block 606, the device determines a transformation matrix based on one or more spatial misalignments between the first and second images. At block 608, the device determines a confidence level associated with the transformation matrix. At block 610, in response to a confidence level greater than a confidence level threshold, the device determines a weighting factor based on the device's first zoom ratio, second zoom ratio, and current zoom ratio. At block 612, the device applies the weighting factor to the transformation matrix. At block 614, the device uses the weighted transformation matrix to distort the first image to the second image.

[0098] In some implementations, the first and second images are captured simultaneously when the current zoom ratio lies within an overlapping region spanning the first and second zoom ratios. In some cases, the weighting factor is 0% when the current zoom ratio equals the first zoom ratio, and 100% when the current zoom ratio equals the second zoom ratio. In other cases, the weighting factor corresponds to the identity matrix when the confidence level is no greater than a confidence threshold. In some implementations, the first image has a wider field of view (FOV) than the second image. In some cases, the first sensor is a wide-angle image sensor, and the second sensor is a long-range image sensor.

[0099] Figure 7 A flowchart depicting another example operation 700 for spatial alignment transformation is shown. Operation 700 can be performed by, for example... Figure 1 The device 100 shown is used for device execution. In various aspects, operation 700 can be used to determine Figure 6 This is one implementation of the weighting factor in box 610 of operation 600. In other aspects, operation 700 can... Figure 6 Operation 600 begins after the first image is distorted to the second image in box 614 of operation 600. For example, after distorting the first image to the second image in box 614 of operation 600, operation 700 can begin in box 702. In box 702, the device gradually increases the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio. In box 704, the device gradually decreases the weighting factor as the current zoom ratio decreases towards the first zoom ratio in the overlapping region spanning between the first zoom ratio and the second zoom ratio.

[0100] Figure 8 A flowchart depicting another example operation 800 for spatial alignment transformation is shown. Operation 800 can be performed by, for example... Figure 1 The device 100 shown is used for device execution. In various aspects, operation 800 can be performed... Figure 6The operation begins after operation 600. For example, after distorting the first image to the second image in box 614 of operation 600, operation 800 can begin in box 802. At box 802, the device generates a preview image based on the distorted first and second images. At box 804, the device displays the preview image to the user of the device.

[0101] Figure 9 A flowchart depicting another example operation 900 for spatial alignment transformation is shown. Operation 900 can be performed by, for example... Figure 1 The device 100 shown is used for device execution. In various aspects, in Figure 6 After capturing a second image of the scene in box 604 of operation 600, operation 900 can begin in box 902. At box 902, the device scales the first image to match the field of view (FOV) of the second image. At box 904, the device identifies one or more misalignments in the space between the scaled first and second images.

[0102] Figure 10 A flowchart depicting another example operation 1000 for spatial alignment transformation is shown. Operation 1000 can be performed by, for example... Figure 1 The device 100 shown is used for device execution. In various aspects, in Figure 6 After determining the confidence level associated with the transformation matrix in box 608 of operation 600, operation 1000 can begin in box 1002. At box 1002, the device avoids determining an additional transformation matrix when the confidence level is greater than a confidence threshold.

[0103] As used in this article, the phrase “at least one” in a list of items refers to any combination of these items, including a single member. For example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc.

[0104] The various illustrative logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the implementation methods disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether this functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0105] Hardware and data processing apparatuses for implementing the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or performed by: general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof for performing the functions described herein. A general-purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor), multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some implementations, specific processes and methods may be performed by circuits specific to a given function.

[0106] In one or more aspects, the described functionality may be implemented in hardware, digital electronic circuits, computer software, firmware (including the structures disclosed in this specification and their equivalents), or any combination thereof. Implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a computer storage medium for execution by or control of the operation of a data processing apparatus.

[0107] If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium as one or more instructions or code. The processes of the methods or algorithms disclosed herein can be implemented in a processor-executable software module residing on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that supports the transfer of a computer program from one place to another. Storage media can be any available medium accessible by a computer. By way of example, and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection can be appropriately referred to as a computer-readable medium. As used herein, disks and discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically copy data, while optical discs optically copy data using lasers. The combinations above should also be included within the scope of protection of computer-readable media. Furthermore, the operation of a method or algorithm may reside as one or any combination or set of code and instructions on machine-readable and computer-readable media, which may be incorporated into a computer program product.

[0108] Various modifications to the implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations without departing from the spirit and scope of this disclosure. Therefore, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the broadest scope consistent with this disclosure and the principles and novel features disclosed herein.

Claims

1. An apparatus comprising: processor; as well as A memory coupled to the processor and storing instructions that, when executed by the processor, cause the device to perform operations including: A first image of the scene is captured using a first sensor at a first zoom ratio; A second image of the scene is captured using a second sensor at a second zoom ratio, wherein the first image has a different field of view (FOV) than the second image. The transformation matrix is ​​determined based on one or more spatial misalignments between the first image and the second image; Determine the confidence level associated with the transformation matrix; In response to the confidence level being greater than a confidence threshold, a weighting factor is determined based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device; A weighted transformation matrix is ​​generated, at least in part, by applying the weighting factors to the transformation matrix; as well as The first image is distorted into the second image using the weighted transformation matrix.

2. The device according to claim 1, wherein, When the confidence level is not greater than the confidence level threshold, the weighting factor includes the identity matrix.

3. The device according to claim 1, wherein, When the current zoom ratio is in the overlapping area spanning between the first zoom ratio and the second zoom ratio, the first image and the second image are captured simultaneously.

4. The device according to claim 3, wherein, When the current zoom ratio is equal to the first zoom ratio, the weighting factor is 0%, and when the current zoom ratio is equal to the second zoom ratio, the weighting factor is 100%.

5. The device according to claim 3, wherein, The execution of the instruction causes the device to perform further operations including the following: As the current zoom ratio increases from the first zoom ratio to the second zoom ratio, the weighting factor is gradually increased; as well as As the current zoom ratio decreases toward the first zoom ratio within the overlapping region, the weighting factor is gradually reduced.

6. The device according to claim 1, wherein, The execution of the instruction causes the device to perform further operations including the following: Generate a preview image based on a distorted first and second image; and The preview image is displayed to the user of the device.

7. The device according to claim 1, wherein, The first image has a wider field of view (FOV) than the second image.

8. The device according to claim 1, wherein, The first sensor includes a wide-angle image sensor, and the second sensor includes a remote image sensor.

9. The device according to claim 1, wherein, The execution of the instruction causes the device to perform further operations including the following: Scale the first image to match the FOV of the second image; as well as One or more spaces between the scaled first and second images are identified as misaligned.

10. The device according to claim 1, wherein, The execution of the instruction causes the device to perform further operations including the following: When the confidence level is greater than the confidence level threshold, avoid determining an additional transformation matrix.

11. A method comprising: The device's first sensor captures a first image of the scene at a first zoom ratio; A second image of the scene is captured using a second sensor of the device at a second zoom ratio, the first image having a different field of view (FOV) than the second image. The transformation matrix is ​​determined based on one or more spatial misalignments between the first image and the second image; Determine the confidence level associated with the transformation matrix; In response to the confidence level being greater than a confidence threshold, a weighting factor is determined based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device; A weighted transformation matrix is ​​generated, at least in part, by applying the weighting factors to the transformation matrix; as well as The first image is distorted into the second image using the weighted transformation matrix.

12. The method according to claim 11, wherein, When the confidence level is not greater than the confidence level threshold, the weighting factor includes the identity matrix.

13. The method according to claim 11, wherein, When the current zoom ratio is in the overlapping area spanning between the first zoom ratio and the second zoom ratio, the first image and the second image are captured simultaneously.

14. The method according to claim 13, wherein, When the current zoom ratio is equal to the first zoom ratio, the weighting factor is 0%, and when the current zoom ratio is equal to the second zoom ratio, the weighting factor is 100%.

15. The method of claim 13, further comprising: As the current zoom ratio increases from the first zoom ratio to the second zoom ratio, the weighting factor is gradually increased; as well as As the current zoom ratio decreases toward the first zoom ratio within the overlapping region, the weighting factor is gradually reduced.

16. The method of claim 11, further comprising: A preview image is generated based on the distorted first and second images; as well as The preview image is displayed to the user of the device.

17. The method according to claim 11, wherein, The first image has a wider field of view (FOV) than the second image.

18. The method according to claim 11, wherein, The first sensor includes a wide-angle image sensor, and the second sensor includes a remote image sensor.

19. The method of claim 11, further comprising: Scale the first image to match the FOV of the second image; as well as One or more spaces between the scaled first and second images are identified as misaligned.

20. The method of claim 11, further comprising: When the confidence level is greater than the confidence level threshold, avoid determining an additional transformation matrix.

21. A non-transitory computer-readable medium storing instructions, which, when executed by a processor of a device, cause the device to perform operations including: A first image of the scene is captured using a first sensor at a first zoom ratio; A second image of the scene is captured using a second sensor at a second zoom ratio, wherein the first image has a different field of view (FOV) than the second image. The transformation matrix is ​​determined based on one or more spatial misalignments between the first image and the second image; Determine the confidence level associated with the transformation matrix; In response to the confidence level being greater than a confidence threshold, a weighting factor is determined based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device; A weighted transformation matrix is ​​generated, at least in part, by applying the weighting factors to the transformation matrix; as well as The first image is distorted into the second image using the weighted transformation matrix.

22. The computer-readable medium of claim 21, wherein, When the confidence level is not greater than the confidence level threshold, the weighting factor includes the identity matrix.

23. The computer-readable medium of claim 21, wherein, When the current zoom ratio is in the overlapping area spanning between the first zoom ratio and the second zoom ratio, the first image and the second image are captured simultaneously.

24. The computer-readable medium of claim 23, wherein, When the current zoom ratio is equal to the first zoom ratio, the weighting factor is 0%, and when the current zoom ratio is equal to the second zoom ratio, the weighting factor is 100%.

25. The computer-readable medium according to claim 23, wherein, The execution of the instruction causes the device to perform further operations including the following: As the current zoom ratio increases from the first zoom ratio to the second zoom ratio, the weighting factor is gradually increased; as well as As the current zoom ratio decreases toward the first zoom ratio within the overlapping region, the weighting factor is gradually reduced.

26. The computer-readable medium of claim 21, wherein, The execution of the instruction causes the device to perform further operations including the following: Generate a preview image based on a distorted first and second image; and The preview image is displayed to the user of the device.

27. The computer-readable medium of claim 21, wherein, The first image has a wider field of view (FOV) than the second image.

28. The computer-readable medium of claim 21, wherein, The first sensor includes a wide-angle image sensor, and the second sensor includes a remote image sensor.

29. The computer-readable medium of claim 21, wherein, The execution of the instruction causes the device to perform further operations including the following: Scale the first image to match the FOV of the second image; as well as One or more spaces between the scaled first and second images are identified as misaligned.

30. An apparatus comprising: A unit for capturing a first image of a scene using a first sensor at a first zoom ratio; A unit for capturing a second image of the scene using a second sensor at a second zoom ratio, wherein the first image has a different field of view (FOV) than the second image. A unit for determining the transformation matrix based on one or more spatial misalignments between the first image and the second image; A unit used to determine the confidence level associated with the transformation matrix; A unit for determining a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device in response to the confidence level being greater than a confidence threshold; A unit for generating a weighted transformation matrix by at least partially applying the weighting factor to the transformation matrix; as well as A unit for distorting the first image to the second image using the weighted transformation matrix.

31. The device according to claim 30, wherein, When the confidence level is not greater than the confidence level threshold, the weighting factor includes the identity matrix.

32. The device according to claim 30, wherein, When the current zoom ratio is in the overlapping area spanning between the first zoom ratio and the second zoom ratio, the first image and the second image are captured simultaneously.

33. The device according to claim 32, wherein, When the current zoom ratio is equal to the first zoom ratio, the weighting factor is 0%, and when the current zoom ratio is equal to the second zoom ratio, the weighting factor is 100%.

34. The device according to claim 32, further comprising: A unit for gradually increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio; as well as A unit for gradually reducing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlapping region.

35. The device according to claim 30, further comprising: A unit for generating a preview image based on a distorted first image and a second image; as well as A unit for displaying the preview image to the user of the device.

36. The device according to claim 30, wherein, The first image has a wider field of view (FOV) than the second image.

37. The device according to claim 30, wherein, The first sensor includes a wide-angle image sensor, and the second sensor includes a remote image sensor.

38. The apparatus of claim 30, further comprising: Units used to scale the first image to match the FOV of the second image; as well as Used to identify one or more misaligned cells between the scaled first and second images.

39. The apparatus of claim 30, further comprising: A unit used to avoid determining an additional transformation matrix when the confidence level is greater than the confidence level threshold.

Images