Camera sensor changes in multi-sensor devices

Abstract

By controlled timing of switching of image sensors according to predefined image sensor configurations, discontinuity in the continuity of image frames output from the image capture device due to switching from one image sensor to another of the device can be reduced or eliminated. Operation of the multi-sensor image device according to predefined image sensor configurations can include appropriately selecting a source of image adjustment during a zoom level transition. The predefined image sensor configurations can define transition parameters for a particular zoom range of the image capture device.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Patent Application No. 17 / 133,221, filed December 23, 2020, entitled “CAMERA SENSOR CHANGES INMULTI-SENSOR DEVICE,” the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to image processing. Some features of this disclosure enable and provide improvements in the processing of output from multi-sensor image capture devices by image signal processors. Background Technology

[0004] An image capture device is a device capable of capturing one or more digital images (whether still images, photographs, or video sequences) and can be incorporated into various devices. For example, an image capture device can include a standalone digital camera or digital camcorder, a wireless communication device with a camera such as a mobile phone, cellular phone, or satellite phone, a personal digital assistant (PDA), a panel or tablet computer, a gaming device, computer equipment (such as a webcam or video surveillance camera), or other devices with digital imaging or video capabilities.

[0005] Some image capture devices include multiple image sensors that capture image data through one or more lenses; these may be referred to as multi-sensor image capture devices. The multiple image sensors can be configured with different lenses to provide multiple fields of view and / or different zoom levels for a scene. Example lens types include wide-angle lenses, ultra-wide-angle lenses, telephoto lenses, telescope lenses, periscope zoom lenses, fisheye lenses, macro lenses, prime lenses, or various combinations thereof. In one example, a dual-camera configuration may include a wide-angle lens and a telephoto lens.

[0006] However, the use of multiple image sensors increases the complexity of image processing within the device, as users are typically not interested in multiple images of a scene, but rather in capturing and displaying a single image. Therefore, multiple frames captured from multiple image sensors can be processed to generate a single image for the user. Furthermore, due to the different physical characteristics between image sensors, frames obtained from each image sensor may flow together in such a way that the transition from one image sensor to another is perceptible to the human eye, such as scene shifts appearing in the resulting video or preview display. For example, zooming in or out on the device may involve switching from one image sensor to another, resulting in a significant change in the field of view during sensor switching, which is perceptible to a user viewing a preview image on a display or watching a video recorded during the zoom-in or zoom-out process. This artifact in the output of a multi-sensor image capture device is undesirable.

[0007] The disadvantages mentioned herein are merely representative and are included to highlight the problems that the inventors have identified with respect to existing devices and have attempted to improve. The various aspects of the device described below can address some or all of the aforementioned disadvantages, as well as other disadvantages known in the art. The improved aspects of the device described below can offer additional advantages beyond those described above and can be used in applications other than those described above. Summary of the Invention

[0008] By controlling the timing of image sensor switching according to a predefined image sensor configuration, and by appropriately selecting the source and / or combination thereof for image adjustment according to the predefined image sensor configuration, the continuity interruption of image frames output from the image capture device due to switching from one image sensor to another can be reduced or eliminated. The predefined image sensor configuration can define conversion parameters for a specific zoom range of the image capture device. For example, the predefined image sensor configuration can define a first zoom range and a second zoom range for the image capture device. The first zoom range can be specified such that image sensor switching to the target sensor is performed by deforming the source sensor to align the image with the target sensor. Then, when the target sensor can better match the field of view of the source sensor, sensor switching occurs at a subsequent zoom level after a certain duration to reduce noticeable offsets in the output of the multi-sensor imaging device. The second zoom range can be specified such that the image sensor is outside the desired range, wherein the image capture device immediately switches from the source sensor to the target sensor. Following an immediate switch, the output of the target sensor is adjusted to align with the output of the source sensor from the previous frame to reduce noticeable offsets in the output of the multisensor imaging device. Within the defined scope of these two examples, the image signal processor determines whether to adjust the output from the source sensor to match the target sensor or vice versa. Furthermore, the use of predefined image sensor configurations allows the multisensor imaging device to respond to explicit requests for lens changes by providing information for switching between image sensors.

[0009] The following summarizes some aspects of this disclosure to provide a basic understanding of the techniques discussed. This overview is not a comprehensive summary of all anticipated features of this disclosure, and is neither intended to identify all important or key elements of all aspects of this disclosure, nor to depict the scope of any or all aspects of this disclosure. Its sole purpose is to present some concepts of one or more aspects of this disclosure in summary form as a prelude to the more detailed description that follows.

[0010] In general, this disclosure describes image processing techniques relating to a digital camera having an image sensor and an image signal processor (ISP). To achieve various zoom levels, in some cases, the image signal processor can initiate a conversion between one image sensor and another based on a zoom level command. This conversion can be performed based on predetermined criteria in the image signal processor (e.g., a defined range in a predefined image sensor configuration). Alternatively, the conversion can be performed based on a request from another component in the device, such as by the CPU in response to an image processing function executed on the CPU. Different image sensors can have different lenses coupled to different image sensors and be packaged as separate cameras, such as a first camera with a wide-angle lens coupled to a first image sensor and a second camera with a telephoto lens coupled to a second image sensor.

[0011] An image signal processor (ISP) can be configured to control the capture of image frames from one or more image sensors and process these frames to generate a view of the scene in an output frame. The configuration for controlling capture may include instructions for processing image sensor change requests, and in some examples, processing these requests during zoom level transitions. In response to an image sensor change request, the ISP can determine the appropriate time to perform the change based on a predefined image sensor configuration. When an image sensor change request is received during a zoom transition, the ISP can use information about the image sensors to determine when an image signal change can be performed to reduce visual impact in the output frame stream or when there is no visual impact in the output frame stream. In some embodiments, the ISP can access a predefined image sensor configuration that describes a defined range of zoom levels for the image sensors. This defined range can specify when image adjustments can be made to match the field of view between two or more image sensors to reduce the occurrence of image sensor changes in the output frame. In some embodiments, the range defines the characteristics of the image adjustments to be performed at various zoom levels. For example, configuration information may describe whether to adjust the output of the first sensor or the output of the second sensor to obtain the output frame when switching from a first sensor to a second sensor. In some embodiments, adjustments to the output image from the image sensor may include blending and / or geometric deformation. Further details regarding image sensor changes and image adjustments using blending or geometric deformation are provided below.

[0012] In the example, the image signal processor can receive instructions to switch from a first image sensor to a second image sensor in response to software executing on the CPU determining that certain features are detected in the output image frames. Example criteria include detecting a specific brightness or contrast level and / or detecting motion in the scene. When the indicated camera switch is performed (such as during zoom operation), the image signal processor can effectively obtain a first set of input frames from the first sensor before the sensor switch and a second set of input frames from the second sensor after the sensor switch. The image signal processor can be configured to generate a single output frame stream based on the corresponding output images from the image sensors. The single output frame stream can include image frames containing image data from the image sensors, which have been adjusted, such as through blending or geometric deformation, to match the image frames with other image frames in the output stream (e.g., frames previously captured by different image sensors). Whether the frames in the output stream are from the source image sensor or the target image sensor based on the sensor switch before and / or after the sensor switch can be determined by a predefined image sensor configuration. In some embodiments, the amount of geometric deformation or blending of the image frames captured by the image sensors can be determined by a predefined image sensor configuration.

[0013] After the image signal processor generates output frames representing the scene, the view of the scene can be displayed on a device display, saved as an image or as a sequence of images as video to a storage device, transmitted over a network, and / or printed to an output medium. For example, the image signal processor can be configured to acquire input frames of image data (e.g., pixel values) from different image sensors and then generate corresponding output frames of image data (e.g., preview display frames, still image captures, video frames, etc.). In other examples, the image signal processor can output frames of image data to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization, generating video files from output frames, configuring frames for display, configuring frames for storage, etc. That is, the image signal processor can acquire input frames from one or more image sensors, each coupled to one or more camera lenses, and then generate and output a stream of output frames to various output destinations. In such an example, the image signal processor can be configured to generate an output stream of frames that dynamically represents changing zoom levels (e.g., increasing or decreasing zoom levels). In the example, the image signal processor can receive input for changing the zoom level based on pinch zoom operations, gesture detection, or other user input to a device that includes an image sensor or a user device coupled to a device that includes an image sensor.

[0014] In some examples, the image signal processor can perform geometric deformation on the last frame obtained from the first image sensor when an image sensor change is anticipated. That is, the image signal processor can deform the frame obtained from the first image sensor before the image sensor change, thereby aligning the pixels of the deformed frame with the pixel coordinates of the first anticipated input frame obtained from the second image sensor after the sensor change. When blending the last frame obtained from the first (or "source") image sensor (e.g., the geometrically deformed frame) with subsequent frames obtained from the second (or "destination") image sensor, the image signal processor can blend the pixels of the last frame with the pixels of the subsequent frames obtained after the sensor change. By initiating the blending process in response to an image sensor change, the image signal processor can advantageously minimize the amount of memory used during the conversion blending process while providing various quality improvements resulting from pixel blending after image sensor conversion.

[0015] In one aspect of this disclosure, a method includes: receiving a request to change from a first sensor to a second sensor during image capture from a multi-sensor capture device; determining a current zoom level corresponding to the received request to change from the first sensor to the second sensor; determining whether the current zoom level is within a first defined range; and / or generating an output frame from the multi-sensor capture device by adjusting an image from one of the first or second sensors based on whether the current zoom level is determined to be within the first defined range. The method may be performed according to predefined control criteria or at the request of a user. For example, the method may be performed when the request to change from the first sensor to the second sensor corresponds to a change request received or scheduled to occur during a transition from the first zoom level to the second zoom level. The method further includes: adjusting an image from a first sensor when the zoom level is determined to be within a first defined range, for example by geometrically deforming a first image output from the first sensor to align it with the field of view of a second sensor, wherein the adjustment may be a hybrid weighting parameter, which is changed to the second sensor after the hybrid weighting parameter reaches a threshold; generating an output frame by adjusting an image from the second sensor after the conversion crosses the first defined range; adjusting an image from the second sensor when the zoom level is determined to be within a second defined range; and / or determining whether the current zoom level is within the first defined range by determining whether the margin on the first sensor is higher than a threshold amount.

[0016] When a request to switch from a second sensor to a third sensor is received, similar or identical methods for handling such requests can be performed. In some embodiments, a single message may include instructions to switch from a first sensor to a second sensor and from a second sensor to a third sensor at different times during zoom transitions. The single message may request multiple changes, which are interpreted as different requests to switch sensors. Such a method may include: receiving a request to switch from a second sensor to a third sensor during image capture from a multi-sensor capture device; determining a second current zoom level corresponding to the received request to switch from the second sensor to the third sensor; determining whether the second current zoom level is within a second defined range; and / or, based on whether the second current zoom level is determined to be within the second defined range, generating an output frame from the multi-sensor capture device by adjusting the image from one of the second or third sensors.

[0017] For example, the method can be performed when the received request to switch from the first sensor to the second sensor is based on lighting conditions during image capture, when the received request to switch from the first sensor to the second sensor is based on movement detected during image capture, and / or when the received request to switch from the first sensor to the second sensor is based on user input for the tracked object during image capture. The received request may be generated by the imaging device, for example, on one or more processors of the imaging device, and / or may be generated by an external device and transmitted, for example, wirelessly to the imaging device performing the image capture method according to the embodiments described herein.

[0018] In another aspect of this disclosure, an apparatus is disclosed comprising at least one processor and a memory coupled to said at least one processor. The at least one processor is configured to perform any of the methods or techniques described herein. For example, the at least one processor may be configured to perform the following steps: receiving a request to change from a first sensor to a second sensor during image capture from a multi-sensor capture device; determining a current zoom level corresponding to the received request to change from the first sensor to the second sensor; determining whether the current zoom level is within a first defined range; and / or, based on whether the current zoom level is determined to be within the first defined range, generating an output frame from said multi-sensor capture device by adjusting an image from one of the first or second sensors. The at least one processor may include an image signal processor, or a processor containing specific functions for camera control and / or processing. The at least one processor may also, or alternatively, include an application processor. The methods and techniques described herein may be performed entirely by an image signal processor or an application processor, or various operations may be divided between an image signal processor and an application processor, and between additional processors in some embodiments.

[0019] The device may include at least two image sensors, including a first image sensor and a second image sensor, wherein the first image sensor has a larger field of view (FOV) than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, while the second image sensor may be a telephoto image sensor. In another example, the first sensor is configured to acquire an image through a first lens having a first optical axis, and the second sensor is configured to acquire an image through a second lens having a second optical axis different from the first optical axis. Alternatively or additionally, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. This configuration can be implemented using a lens array on a mobile device, for example, where multiple image sensors and associated lenses are located at offset positions on the front or back of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The device may switch to output frames based on the output of other image sensors, and / or generate output frames using image frames from multiple image sensors, the output frames being adjustable based on the methods and techniques described herein.

[0020] In another aspect of this disclosure, an apparatus configured for image capture is disclosed. The apparatus includes: a unit for receiving a request to change from a first sensor to a second sensor during image capture from a multi-sensor capture device; a unit for determining a current zoom level corresponding to the received request to change from the first sensor to the second sensor; a unit for determining whether the current zoom level is within a first defined range; and / or a unit for generating an output frame from the multi-sensor capture device by adjusting an image from one of the first or second sensors based on whether the current zoom level is determined to be within the first defined range. The apparatus also includes one or more units for capturing data representative of a scene, such as image sensors (including charge-coupled device (CCD), Bayer filter sensors, infrared (IR) detectors, ultraviolet (UV) detectors, complementary metal-oxide-semiconductor (CMOS) sensors), and time-of-flight detectors. The apparatus may also include units for accumulating and / or focusing light onto one or more of the image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses).

[0021] In another aspect of this disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including those described in the methods and techniques described herein. For example, the operations may include: receiving a request to change from a first sensor to a second sensor during image capture from a multi-sensor capture device; determining a current zoom level corresponding to the received request to change from the first sensor to the second sensor; determining whether the current zoom level is within a first defined range; and / or, based on whether the current zoom level is determined to be within the first defined range, generating an output frame from the multi-sensor capture device by adjusting an image from one of the first or second sensors.

[0022] Other aspects, features, and implementations will become apparent to those skilled in the art after reading the following description of specific exemplary aspects in conjunction with the accompanying drawings. While features may be discussed in conjunction with certain aspects and figures below, each aspect may include one or more of the advantageous features discussed herein. That is, while one or more aspects may be discussed as having certain advantageous features, one or more such features may also be used according to each aspect. Similarly, while exemplary aspects may be discussed below as aspects of an apparatus, system, or method, exemplary aspects may be implemented in various apparatuses, systems, and methods.

[0023] The method can be embedded as computer program code in a computer-readable medium, the computer program code including instructions to cause a processor to perform the steps of the method. In some embodiments, the processor may be part of an information processing system including a first network adapter configured to transmit data via a first network connection of a plurality of network connections; a processor coupled to the first network adapter; and memory. In some embodiments, the network connection may couple the information processing system to external components, such as wired or wireless docking stations.

[0024] Certain features and technical advantages of embodiments of the present invention have been broadly outlined above to facilitate a better understanding of the detailed embodiments described below. Additional features and advantages that form the subject matter of the claims of the present invention will be described below. Those skilled in the art will understand that the disclosed concepts and specific embodiments can be readily used as the basis for modifications or the design of other structures for achieving the same or similar purposes. Those skilled in the art should also recognize that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims. The additional features will be better understood from the following description when considered in conjunction with the accompanying drawings. However, it should be clearly understood that each drawing is provided for illustrative and descriptive purposes only and is not intended to limit the invention. Attached Figure Description

[0025] A further understanding of the nature and advantages of this disclosure can be achieved by referring to the following accompanying drawings. In the drawings, similar components or features may have the same reference numerals. Furthermore, various components of the same type can be distinguished by adding a dash after the reference numerals and a second reference numeral to distinguish similar components. If only the first reference numeral is used in the specification, the description applies to any similar component having the same first reference numeral, regardless of the second reference numeral.

[0026] Figure 1 It is a block diagram of a computing device configured to perform one or more of the example technologies described in this disclosure.

[0027] Figure 2 This is a schematic diagram illustrating the operation of multiple image sensors in the example techniques described in this disclosure.

[0028] Figure 3 This is a schematic diagram of a defined zoom level range for flexible zoom conversion in the example techniques described in this disclosure.

[0029] Figure 4 This is a flowchart illustrating the operation of multiple image sensors in response to a sensor change request in the example technology described in this disclosure.

[0030] Figure 5 This is a call flowchart illustrating the operation of multiple image sensors with sensor change requests in the example technology described in this disclosure.

[0031] Figure 6 This is a block diagram illustrating the use of geometric deformation to adjust an image in an example technique described in this disclosure.

[0032] The same reference numerals and names in the various figures denote the same elements. Detailed Implementation

[0033] The specific embodiments described below with reference to the accompanying drawings are intended as descriptions of various configurations and are not intended to limit the scope of this disclosure. Rather, the specific embodiments include particular details for providing a thorough understanding of the subject matter of the invention. It will be apparent to those skilled in the art that these specific details are not necessary in every case, and in some cases, well-known structures and components are shown in block diagram form for clarity.

[0034] This disclosure provides systems, apparatus, methods, and computer-readable media that support flexible zoom switching in multi-sensor image capture devices. The flexible zoom switching allows changing the image sensor upon request from a user or an application running on a computing device. Specific implementations of the subject matter described in this disclosure can be implemented to achieve potential advantages or benefits, such as improving image quality by switching to an image sensor with better image quality for current scene conditions and / or applied image capture settings, and / or improving the user experience by reducing noticeable shifts in the display when changing the image sensor.

[0035] Various aspects of this disclosure can be used to capture image frames using multiple image sensors in an image capture device. The multiple image sensors may include combinations of ultra-wide-angle (high field of view (FOV)), wide-angle, telephoto, and ultra-telephoto (low FOV) sensors. That is, each image sensor can be configured through hardware configuration and / or software settings to obtain different but overlapping fields of view. In one configuration, the image sensors are configured with different lenses with different magnifications, thereby producing different fields of view. The sensors can be configured such that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for wide-angle FOV can capture a field of view in the range of 64-84 degrees, a sensor configured for ultra-wide-angle FOV can capture a field of view in the range of 100-140 degrees, a sensor configured for telephoto FOV can capture a field of view in the range of 10-30 degrees, and a sensor configured for ultra-telephoto FOV can capture a field of view in the range of 1-8 degrees. Some aspects of this disclosure include: processing captured image frames, for example by adjusting the spatial alignment of one or more image frames among the captured image frames when the device transitions from capturing an image of a scene using a first image sensor among the plurality of image sensors to capturing an image of the scene using a second image sensor among the plurality of image sensors. The adjustment can be performed on an image frame selected from one of the plurality of image sensors, based at least in part on a predefined image sensor configuration and the current zoom level in the zoom level conversion.

[0036] Example devices for capturing image frames using multiple image sensors, such as smartphones, may include a configuration of two, three, four, or more cameras on the back (e.g., the side opposite the user's display) or front (e.g., the same side as the user's display). Devices with multiple image sensors include one or more image signal processors, computer vision processors (CVPs), or other suitable circuitry for processing images captured by the image sensors. The one or more image signal processors may provide the processed image frames to a memory and / or a processor (e.g., an application processor, image front-end (IFE), image processing engine (IPE), or other suitable processing circuitry) for further processing, such as encoding or other operations.

[0037] As used herein, an image sensor can refer to the image sensor itself and any other suitable component coupled to the image sensor. For example, an image sensor can also refer to other components of a camera, including the shutter, buffer, or other readout circuitry. An image sensor can also refer to an analog front-end or other circuitry used to convert analog signals into a digital representation of frames. Therefore, the term "image sensor" as used herein can refer to any suitable component used to capture image frames and read them out to an image signal processor.

[0038] When the device's current zoom ratio is a first value, the example multi-sensor device can use a wide-angle sensor to capture a "wide-angle" image of the scene, and when the current zoom ratio is a higher second value, the device can switch to using a telephoto sensor to capture a "telephoto" image of the scene. When the current zoom ratio is within a defined range of values ​​(which may be referred to herein as the "overlap area"), the device can simultaneously or nearly simultaneously use two sensors to capture images of the scene. The device can use image data from one or more sensors to generate, for example, a preview image of the scene to display to the device's user. However, each image sensor may have a different FOV not only due to magnification but also due to its different location on the device. That is, each image sensor is displaced relative to the other image sensors in the xy plane corresponding to the device's plane, which results in an offset of the image frames captured from each image sensor. Furthermore, due to manufacturing defects, multiple sensors and / or the captured images may be spatially misaligned, which may lead to misalignment errors associated with the generated images at the device's current zoom ratio. Optical axis offset and / or misalignment errors from one image sensor to another may cause visual defects in the corresponding preview image or captured video. Various aspects of this disclosure provide a multi-sensor device that can switch from capturing scene images using a first sensor to capturing scene images using a second sensor by adjusting captured images from a first or second sensor according to a predefined image sensor configuration.

[0039] In the following description, numerous specific details, such as examples of specific components, circuits, and processes, are set forth to provide a thorough understanding of this disclosure. As used herein, the term “coupled” means a direct connection to or a connection via one or more intermediary components or circuits. Furthermore, specific nomenclature is set forth in the following description and for illustrative purposes to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that these specific details may not be necessary to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of this disclosure. Certain portions of the following detailed description are presented according to symbolic representations of processes, logic blocks, operations, and other operations relating to data bits within computer memory. In this disclosure, processes, logic blocks, operations, etc., are considered as a self-consistent sequence of steps or instructions that lead to a desired result. These steps are steps that require physical manipulation of physical quantities. Typically (though not required), these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated in a computer system.

[0040] However, it should be remembered that all these terms and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise clearly indicated in the following discussion, it should be understood that throughout this application, the use of terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” “stabilize,” and “generate” refers to the operation and processes of a computer system or similar electronic computing device, which manipulate and convert data represented as physical (electronic) quantities in the registers and memories of the computer system into other data similarly represented as physical quantities in the registers, memories, or other such information storage, transmission, or display devices of the computer system.

[0041] In the accompanying drawings, a single block may be described as performing one or more functions; however, in practice, the one or more functions performed by that block may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described below according to their functions. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure. Furthermore, the example device may include components other than those shown, including well-known components such as processors, memory, etc.

[0042] Various aspects of this disclosure are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or “frames”). Furthermore, various aspects of this disclosure can be implemented in devices having or being coupled to image sensors having the same or different capabilities and characteristics (e.g., resolution, shutter speed, sensor type, etc.).

[0043] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some of the contents of this disclosure. Although the term "device" is used in the following description and examples to describe various aspects of this disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus can include a device or part of a device for performing the described operations.

[0044] Figure 1 A block diagram of an example device 100 for performing image capture from multiple image sensors is shown. Device 100 may include or be otherwise coupled to an image signal processor 112 for processing image frames from multiple image sensors (e.g., a first image sensor 101 and a second image sensor 102). In some embodiments, device 100 also includes or is coupled to a processor 104 and a memory 106 for storing instructions 108. Device 100 may also include or be coupled to a display 114 and multiple input / output (I / O) components 116. Device 100 may also include or be coupled to a power supply 118, such as a battery or components for coupling device 100 to an energy source. Device 100 may also include or be coupled to additional features or components not shown. In one example, a wireless interface may be included for use with a wireless communication device; the wireless interface may include multiple transceivers and a baseband processor. In another example, one or more other sensors (e.g., a gyroscope or a Global Positioning System (GPS) receiver) may be included in or coupled to the device. In a further example, an analog front-end for converting analog image frame data into digital image frame data may be coupled between image sensors 101 and 102 and image signal processor 112.

[0045] Image signal processor 112 can receive image data from a local bus connection to an image sensor, or via other connections (e.g., a wired interface to an external image sensor or a wireless interface to a remote image sensor). In some embodiments, device 100 may include a first camera and a second camera, the first camera including a first image sensor 101 and a corresponding first lens 131, and the second camera including a second image sensor 102 and a corresponding second lens 132. In some embodiments, device 100 may include an interface for receiving image data from image sensors 101 and 102 located outside device 100. Device 100 can perform image processing on image data from a combination of image sensors located within or separate from device 100.

[0046] First image sensor 101 and second image sensor 102 are configured to capture one or more image frames. For example, first image sensor 101 and second image sensor 102 may be included in a multi-camera configuration or in separate single-camera or separate multi-camera configurations (e.g., dual-camera configuration, triple-camera configuration, etc., for smartphones or other suitable devices). Image sensors 101 and 102 may also include or be coupled to one or more lenses for focusing light, one or more apertures for receiving light, one or more shutters for blocking light outside an exposure window, one or more filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front-ends for converting analog measurements into digital information, or other suitable components for imaging. For example, first image sensor 101 may be coupled to first lens 131 and second image sensor 102 may be coupled to second lens 132. First lens 131 and second lens 132 may have different fields of view, for example, when first lens 131 is an ultra-wide-angle (UW) lens and second lens 132 is a wide-angle (W) lens. The device 100 may also include or be coupled to a flash, depth sensor, GPS or other suitable components for imaging.

[0047] Image signal processor 112 processes image frames captured by image sensors 101 and 102. Although Figure 1The device 100 is illustrated as including two image sensors 101 and 102 coupled to the image signal processor 112, but any number of image sensors may be coupled to the image signal processor 112. Furthermore, the device 100 may contain any number of additional image sensors or image signal processors. In some embodiments, the image signal processor 112 may execute instructions from memory, such as instructions 108 from memory 106, instructions stored in a separate memory coupled to or included in the image signal processor 112, or instructions provided by processor 104. Alternatively or additionally, the image signal processor 112 may execute software and / or may include specific hardware (e.g., one or more integrated circuits (ICs)) to perform one or more operations described in this disclosure.

[0048] In some embodiments, memory 106 may include a non-transient or non-temporary computer-readable medium storing computer-executable instructions 108 for performing all or part of one or more operations described in this disclosure. In some embodiments, instructions 108 include a camera application (or other suitable application) to be executed by device 100 to generate images or videos. Instructions 108 may also include other applications or programs executed by device 100, such as an operating system and other specific applications besides those for image or video generation. For example, execution of the camera application by processor 104 may cause device 100 to generate images using image sensors 101 and 102 and image signal processor 112. Memory 106 may also be accessed by image signal processor 112 to store processed frames, or may be accessed by processor 104 to obtain processed frames. In some embodiments, device 100 does not include memory 106. For example, device 100 may be circuitry including image signal processor 112, and the memory may be external to device 100. Device 100 may be coupled to memory and configured to access memory to write output frames for display or long-term storage.

[0049] In some embodiments, processor 104 may include one or more general-purpose processors capable of executing scripts or instructions of one or more software programs, such as instructions 108 stored in memory 106. For example, processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in memory 106. When executing the camera application, processor 104 may be configured to instruct image signal processor 112 to perform one or more operations against image sensor 101 or 102. For example, a camera application executing on processor 104 may receive a zoom level change request from a user and instruct image signal processor 112 to switch to the new zoom level. During the switch to the new zoom level, processor 104 may detect changes in the scene that satisfy specific criteria for changing the image sensor, and processor 104 may issue a sensor change request to image signal processor 112. Execution of instructions 108 by processor 104 outside of the camera application may also cause device 100 to perform any number of functions or operations. In some embodiments, processor 104 may include an IC or other hardware in addition to the ability to execute software to enable device 100 to perform multiple functions or operations (such as those described herein). In some other embodiments, device 100 does not include processor 104, for example when all the described functions are configured in image signal processor 112.

[0050] In some embodiments, at least one of the image signal processor 112 or processor 104 may execute instructions to perform various operations described herein. For example, the execution of instructions may instruct the image signal processor 112 to change from capturing a first image of a scene using a first image sensor 101 at a first zoom ratio to capturing a second image of a scene using a second image sensor 102 at a second zoom ratio. In some embodiments, the first image may have a different FOV than the second image because the first image sensor 101 has a different FOV than the second image sensor 102 or the first lens 131 has a different FOV than the second lens 132.

[0051] In some embodiments, display 114 may include one or more suitable displays or screens that allow the user to interact and / or present items (e.g., previews of image frames captured by image sensors 101 and 102) to the user. In some embodiments, display 114 is a touch-sensitive display. I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (e.g., commands) from the user and provide output to the user. For example, I / O component 116 may include (but is not limited to) a graphical user interface (GUI), keyboard, mouse, microphone, speaker, squeezable bezel, one or more buttons (e.g., power button), slider, switch, etc.

[0052] Although shown as coupled to each other via processor 104, processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled to each other in various other arrangements, such as via one or more local buses, not shown for simplicity. Although image signal processor 112 is shown as separate from processor 104, image signal processor 112 may be the core of processor 104 as an application processor unit (APU), included in a system-on-a-chip (SoC), or otherwise included in processor 104. While device 100 is referred to in the examples herein for the purpose of carrying out various aspects of this disclosure, some device components may not be shown in the examples. Figure 1 The details are shown to prevent any obscurity of the various aspects of this disclosure. Furthermore, other components, multiple components, or combinations of components may be included in suitable devices for performing the various aspects of this disclosure. Therefore, this disclosure is not limited to the configuration of a particular device or component, including device 100.

[0053] As described above, different first lenses 131 and second lenses 132 may result in image frames captured from the first image sensor 101 and the second image sensor 102 having different characteristics, such as different fields of view, but the image sensors may also produce similar image frames in the overlapping area. Figure 2 An example overlap region 200 is shown, spanning between the device's first and second zoom ratios as the device's current zoom ratio increases. The device could be... Figure 1 The example embodiment of device 100 shown is illustrated. Although the overlapping area 200 has been described with reference to the devices, components, and operations shown in the above figures, any suitable device or device component can perform operations for... Figure 2 The described operation. In some embodiments, the first sensor is Figure 1 An example embodiment of the first image sensor 101 may be a wide-angle image sensor (or "wide-angle sensor"), and the second image sensor is... Figure 1 An example embodiment of the second sensor 102 may be a telephoto image sensor (or "telephoto sensor").

[0054] When the current zoom ratio of device 200 is within the first sensor area 202, device 100 can capture wide-angle images using only the wide-angle sensor. In some embodiments, the first sensor area 202 can be... Figure 2It begins at the far left (e.g., a 1.0X zoom ratio) and ends with "first zoom ratio" (e.g., a 2.7X zoom ratio). Conversely, when the current zoom ratio of device 100 is within the second sensor area 204, device 100 can capture telephoto images using only the telephoto sensor. In some embodiments, the second sensor area 204 may begin with a second zoom ratio (e.g., a 3.0X zoom ratio) and ends with... Figure 2 It ends at the far right (e.g., a 10.0X zoom ratio).

[0055] Within the overlap region, a range of zoom levels can be captured from either the wide-angle or telephoto image. When the current zoom ratio of device 100 is within the overlap region (e.g., a zoom ratio between 2.7X and 3.0X), device 100 can capture a telephoto image using the telephoto sensor while simultaneously capturing a wide-angle image using the wide-angle sensor. By adjusting either the wide-angle or telephoto image to align with the other, the output frame from the image signal processor that processes the wide-angle and telephoto images can be based on either the wide-angle or telephoto image. This adjustment allows for a smooth transition from the wide-angle sensor to the telephoto sensor, for example, when a zoom level change occurs from the zoom level within the first sensor region 202 to the zoom level within the second sensor region 204. Without such an adjustment, a user might notice an optical axis offset from the wide-angle sensor to the telephoto sensor, resulting in a noticeable discontinuity, manifested as an object jumping from a first position to a second position in the output frame. An image signal processor can determine which image, either a wide-angle or telephoto image, to adjust based on a predefined image sensor configuration, which can include multiple defined zoom ranges. Figure 3 The example predefined image sensor configuration is shown.

[0056] Figure 3This is a diagram illustrating predefined image sensor configurations with potentially overlapping ranges between image sensors according to some embodiments of the present disclosure. First Figure 302 shows zoom levels available for a first image sensor corresponding to the ultra-wide-angle (UW) image sensor in this embodiment. Second Figure 304 shows zoom levels available for a second image sensor corresponding to the wide-angle (W) image sensor in this embodiment. Third Figure 306 shows zoom levels available for a third image sensor corresponding to the telephoto (T) image sensor in this embodiment. The scale of each UW, W, and T image sensor shown is scaled such that the zoom level of each image sensor has the same field of view for the other image sensors at the corresponding points on the axis. In this embodiment, the initial zoom range from approximately 1.0X to 1.2X for each sensor in Figures 302, 304, and 306 is shaded to indicate that at these zoom levels of these lenses, there is almost no margin for cropping at the edges of the corresponding image frames. Within these initial zoom ranges, image frames can be captured, but the data in the image frames used to perform image frame scaling (e.g., to match the desired field of view) is limited.

[0057] Each of the identified zoom ranges from multiple individual cameras can be combined to form a fourth figure 308 that defines the desired zoom range for a multi-sensor imaging device. The fourth figure 308 illustrates several defined ranges within a logical zoom that can be used to capture images using available UW, W, and T image sensors represented by figures 302, 304, and 306. The logical zoom of figure 308 can correspond to a zoom level presented to the user during operation of a camera application. When the user specifies a zoom level in the camera application, the image sensor used to capture images at that specific logical zoom level can be identified, and that image sensor can be controlled to adjust its actual zoom level to achieve the desired logical zoom level. For example, when the user requests a 5.0X logical zoom, the image signal processor can control the telephoto image sensor to acquire an image frame at 1.3X zoom or control the wide-angle image sensor to acquire an image frame at 5.0X zoom.

[0058] Figure 308 illustrates a predefined image sensor configuration with defined ranges 312, 314, 316, 318, 320, and 322. These defined ranges can be used to determine which image sensor output to adjust to generate the output frame and / or when to switch from one image sensor in the UW, W, and T image sensors to another. These defined ranges may include a first set of defined ranges 312, 314, and 320, which may be referred to as the “green zone” and indicated by crosshairs. The ranges in the first set of defined ranges specify zoom levels for which the image from the second sensor has a field of view that overlaps with that of the first sensor, but the margins in the second sensor are insufficient for processing to match the first sensor. Determining whether sufficient margins exist may involve comparing the available margins of the image sensor with a threshold. The threshold may be based on, and equal to, the margins that the sensor can provide due to the field of view being smaller than the frame size. For example, this determination can be made by comparing the field of view of the image sensor at a specific zoom level with a predefined offset value, which indicates the amount of offset used to align the image sensor output with another image sensor. If the field of view of the image sensor is larger than the desired frame size at the specific zoom level by at least the offset value, the determination can be that there is sufficient margin to match the image sensor output to another image sensor.

[0059] When a zoom level change is requested, the device adjusts from a first zoom level to a second zoom level on Figure 308. An image sensor change request can be received simultaneously with the transition from the first zoom level to the second zoom level. The image sensor change request can indicate both a source image sensor and a target image sensor. When the logical zoom level transition crosses one of the first set of defined ranges 314 or 320 and an image sensor change request is received, the output of the source image sensor can be adjusted to align with the output of the target image sensor to produce an output frame. The use of the source image sensor as the basis for the output frame can continue until the current zoom level reaches the end of one of the first set of defined ranges 314 or 320.

[0060] In some embodiments, image sensor changes may occur before reaching the end of one of the defined ranges 314 or 320, for example, when a parameter reaches or exceeds a threshold. For example, an image signal change may occur when the α parameter used in adjusting the image sensor output reaches or exceeds a threshold, which may happen before reaching the end of the defined range 314. In some embodiments, the adjustment may be a blending of the output image from a determined source or target sensor with a previous frame. For example, an image signal processor may blend one or more pixels from a previous frame output from a source sensor with one or more pixels from the output image from a determined source or target sensor by applying a blending function to determine the pixel contribution level as follows:

[0061] Y′1=∝(X M )+(1-∝)Y1, where:

[0062] Y′1 represents the output frame adjusted by the mixed weights ∝.

[0063] X M Indicates the previous frame,

[0064] Y1 represents the output of the determined source or target sensor, and

[0065] ∝ represents the following mixed weights: 0 ≤ ∝ ≤ 1.

[0066] In different embodiments, the image signal processor may determine the values ​​assigned to the blending weights (e.g., between 0 and 1, between 0 and 100%, etc.) automatically, based on manual input, or both. Regardless of how the blending weight parameters are determined, a threshold can be set to accomplish the requested sensor change, such as when α reaches 0.8 or 80%. In some embodiments, zoom transitions may occur faster than the parameters reach the threshold, with one of the second set of defined ranges occurring first, in which case the image sensor changes regardless of whether the parameters have not reached the threshold.

[0067] In some embodiments, the image signal processor can, within a defined range of one of the first defined ranges 312, 314, or 320, process the previous frame (X) captured from the first sensor before changing the sensor. M Geometric deformation can be performed on the previous frame (X). In some embodiments, the previous frame (X) can be subjected to geometric deformation. M Perform geometric transformations, for example, through the following reference. Figure 6Variations of the description. In some embodiments, the image signal processor may discard the mixing of one or more pixels (e.g., one or more edge pixels (e.g., corner pixels)) from a previous frame, and instead mix one or more other pixels from the previous frame with one or more corresponding pixels from the output image of the determined source or target image sensor. In such embodiments, based on mixing weights during the sensor change process or another equivalent method for the aforementioned mixing of pixels in a specific region of the frame, the mixed frame generated via pixels from the previous frame and pixels from the output image may include contributions from the previous frame and complementary contributions from a reduced portion of the output image relative to the output frame.

[0068] In one example involving a defined range 314, a zoom level change from 0.8X to 1.4X is requested, and a request to change the image sensor from UW to W is received when the zoom level is 1.1X. The image signal processor can determine that the image sensor change request was received within the defined range 314, which is one of a first set of defined ranges. Based on the defined range 314, the image signal processor can determine to adjust the output of the UW image sensor to align with the W image sensor, while the zoom level transition continues through 1.2X, which is the end of the defined range 314. At the zoom level of 1.2X, the image signal processor can switch to the W image sensor and complete the zoom level change by switching to the requested 1.4X zoom level using parameters controlling the W image sensor. The delay in changing the image sensor from a UW image sensor to a W image sensor allows for a zoom level at which the W image sensor has sufficient margin to match the UW image sensor output and reduce artifacts in the frame sequence obtained during the transition from 0.8X to 1.4X zoom level.

[0069] The defined ranges may include a second set of defined ranges 316, 318, and 322, which may be referred to as the “red zone” and indicated by shading. Within these zoom levels, if a sensor change request is received, a sensor change is performed immediately or as soon as possible, and the output image of the target image sensor is adjusted to match the previous output frame of the source image sensor. The adjustment continues after the change to the target image sensor to achieve the desired zoom level. The image sensor change can be performed at any zoom level within the defined ranges 316, 318, and 322 because there is sufficient margin on the target image sensor to perform the adjustment to align the output image of the target image sensor with the output image of the source image sensor.

[0070] In one example involving defined range 316, a zoom level change from 0.8X to 1.4X is requested, and an image sensor change request from UW to W is received when the zoom level is 1.3X. The image signal processor can determine that the image sensor change request was received within defined range 316, which is one of a second set of defined ranges. Based on defined range 316, the image signal processor can determine to adjust the output of the W image sensor to align with the UW image sensor and immediately change the output frame to the adjusted output image from the W image sensor. The image signal processor can perform the zoom level change by controlling the parameters of the W image sensor to switch to the requested 1.4X zoom level.

[0071] Such as Figure 3 The predefined image sensor configuration shown can be used to define a zoom level range and the behavior for handling image sensor changes within that zoom level range. In some embodiments, the defined range may correspond to physical characteristics of the image sensor, such as image sensor size and / or the zoom range of the associated lens. The configuration may be stored in a configuration file or in other memory coupled to an image signal processor and accessed to determine image sensor change behavior based on the zoom level. For example, the predefined image sensor configuration can be used to determine whether to adjust the output image of a first sensor or the output image of a second sensor to generate an output frame during zoom level transitions, such as... Figure 4 As described in the example methods.

[0072] Figure 4 This is a flowchart illustrating an example method for controlling an image capture device according to some embodiments of the present disclosure. Method 400 begins at block 402, where image capture from a first sensor of the multi-sensor capture device is performed. Block 402 may involve executing a camera application to generate a preview image for acquiring a picture and / or generating a sequence of frames for a video file. In other examples, the image capture at block 402 may involve the execution of a video conferencing application, a chat application, a mixed reality application, an augmented reality application, or another application involving image processing. The application may execute on an application processor directly coupled to the multi-sensor capture device, or indirectly coupled to the multi-sensor capture device via an image signal processor.

[0073] At box 404, a request to change the zoom level can be received. The zoom level change can be requested by a user of an application running on the application processor. Alternatively, the zoom level change can be automatically generated by the application running on the application processor, for example, when tracking objects in a scene is involved. In another example, the zoom level change can be automatically generated by an image signal processor in response to changing conditions in the scene. In some embodiments, the zoom level change request can specify a target zoom level for the multi-sensor capture device. In other embodiments, the zoom level change request can specify other parameters corresponding to the target zoom level, such as a defined image size or region of the scene to be captured by specifying the multi-sensor capture device.

[0074] At box 406, the multi-sensor capture device begins a transition from a first zoom level to a second zoom level while capturing data from its first sensor. The first zoom level may correspond to the current zoom level at the time the request at box 404 is received. The second zoom level may be the requested zoom level, such as the zoom level generated when a user requests a zoom-in or zoom-out operation via a camera application. For example, the second zoom level may represent a scene area specified by a pinch-to-zoom operation on a touchscreen. The transition at box 406 may include, for example, activating the position of a lens coupled to the first sensor, and / or adjusting cropping and / or scaling operations applied to image frames captured from the first sensor.

[0075] At box 408, during the transition to the second zoom level that begins at box 406, a request to switch from the first sensor to the second sensor is received. This request may be generated by the camera application or other application code (e.g., application code within the camera frame, application code executing on the application processor of the computing device). The sensor switching request may be generated based on one or more criteria. In one example, the brightness of a scene, object, or face may be determined to match or exceed a threshold level used to trigger the sensor change request. For example, the target sensor may have a lens with a larger aperture than the source sensor's lens, making a switch from the source sensor to the target sensor beneficial for improving image quality. In another example, the motion of an object or person in the scene may be determined to match or exceed a threshold level used to trigger the sensor change request.

[0076] At box 410, in response to a sensor change request at box 408, the computing device can determine whether the current zoom level is within the first defined range of the second sensor. For example, refer to... Figure 3 The example configuration can compare the current zoom level with ranges 312, 314, and 320 when a sensor change request is received. See also... Figure 3The current zoom level at the time of receiving a sensor change request can be compared with ranges 316, 318, and 322. Determining whether the current zoom level falls within one of ranges 312, 314, 316, 318, 320, or 322 can be used to determine when to respond to a sensor change request and perform a sensor change.

[0077] At box 412, an output frame is generated by adjusting the image output from the first sensor and / or the second sensor, wherein the first and / or second sensor is determined at least in part based on the identification of the defined range performed in box 410. For example, when the current zoom level is within the first defined range, the output image can be generated by adjusting the image from the first sensor until a later sensor change occurs. The sensor change can occur as soon as possible after receiving a sensor change request that the margin from the second sensor is sufficient to be adjusted for alignment with the first sensor. In another example, when the current zoom level is within the second defined range, the output image can be generated by immediately switching to the second sensor and adjusting the output from the second sensor to align with the output of the first sensor, which serves as the basis for the immediately preceding output frame. Adjusting the output image from either the first or second sensor can align the fields of view between two sensors with different fields of view, reducing fluctuations when switching from the first sensor to the second sensor. When the current zoom level is within an undefined range, the sensor change can occur immediately. Adjustment may not be performed on the output of the second sensor, as the image may not be able to be adjusted to match the field of view outside the defined range.

[0078] The operations in boxes 410 and 412 regarding determining the current zoom level and generating the adjusted image based on the current zoom level can be repeated until the zoom level conversion is complete. For example, as the current zoom level transitions from a first zoom level to a second zoom level, it can be determined in box 412 whether the current zoom level is within a first defined range. When the current zoom level reaches the standard defined for the first defined range, actions can be taken to change the method used to generate the adjusted image, for example, by changing the method from adjusting the image based on a first image sensor or a second image sensor, as described in the examples below.

[0079] Figure 5 The example shown illustrates the operation of a multi-sensor imaging device within a computing device using a predefined image sensor configuration. Figure 5This is a calling diagram illustrating image capture in an image capture apparatus according to some embodiments of the present disclosure. A first sensor 502 and a second sensor 504 may be coupled to an image signal processor (ISP) 506, for example, via a camera serial interface bus or via a shared bus. The ISP 506 may be coupled to an application processor (AP) 508, for example, via a peripheral component interface bus. The AP 508 may execute software to provide a user interface 510 as part of a camera application, for example, for receiving commands from the user (e.g., zoom level change) and / or displaying output frames or frame sequences to the user, such as generating preview images for the camera application.

[0080] Image capture can be performed at call 512, for example, to generate a preview image at user interface 510. At call 512, the first sensor 502 provides the captured image to ISP 506. ISP 506 processes the image and provides it to AP 508 at call 514. During operation of the camera application, AP 508 can receive a zoom level change request at call 516. The request at call 516 may be in response to a pinch-to-zoom gesture from the user at user interface 510, or it may be automatically generated. AP 508 can send a zoom level change request to ISP 506 at call 520. In response to receiving call 520, ISP 506 can begin a transition from the current zoom level to the requested zoom level. During this transition, ISP 506 can continue capturing images, for example, at call 522, and provide the images to AP 508, for example, at call 524. Continued image capture can allow, during zoom changes, for example, to present a preview image to the user via the camera application, or for the camera application to continue recording video.

[0081] During the transition to the zoom level requested by call 520, AP 508 may send a sensor change request to ISP 506 at call 526. In response to call 526, ISP 506 may determine a method for changing the lens during the transition to the zoom level requested by call 520. For example, ISP 506 may determine whether the current zoom level at the time of receiving call 526 is within a predefined range, and perform an operation based on whether the current zoom level is within the predefined range, such as in... Figure 4As described in blocks 408, 410, and 412. At call 528, ISP 506 receives output images from first sensor 502 and / or second sensor 504. ISP 506 may process one or both of the output images from first sensor 502 and second sensor 504 to generate an output frame to be provided to AP 508 at call 530. In some embodiments, a determination is made regarding which of sensors 502 and 504 to capture and adjust to form the output frame 508 to AP, based on whether the current zoom level matches a defined range. As the zoom level continues to change in response to call 520, ISP 506 may continue processing images from one or both of sensors 502 and 504 until the requested zoom level is achieved and the sensor change requested by call 526 is complete.

[0082] After completing the sensor change in response to call 526, ISP 506 can capture an image from the second sensor 504 at call 532. At call 534, ISP 506 provides those images or processed versions of those images to AP 508. If the current zoom level after the sensor change is not the requested zoom level, the zoom level change in call 520 can be completed after the sensor change at call 532.

[0083] The processing of sensor change requests, such as that performed by the ISP in response to a request from the AP, allows for flexible zoom transitions between lenses at any zoom level rather than at a predefined fixed point. During zoom level transitions, the resulting output frames have reduced artifacts (including reduced fluctuations or interruptions) between output frames of the multi-sensor capture device. This flexible zoom transition can be performed based on a predefined image sensor configuration and / or the characteristics of the sensors in the multi-sensor capture device. In some embodiments, this flexible zoom transition can provide a spatial alignment universal framework to allow transitions between two, three, four, or more image sensors with different fields of view. One embodiment of such a multi-sensor capture device includes a camera module having: a first image sensor coupled to a first lens to provide an ultra-wide-angle view, a second image sensor coupled to a second lens to provide a wide-angle view, and a third image sensor coupled to a third lens to provide a telephoto view. Additional lenses (e.g., infrared image sensors) may be part of the multi-sensor capture device and are used to capture depth information corresponding to the scene.

[0084] Despite the presence of multiple image sensors, providing users with a single viewing experience improves the user experience when using multi-sensor capture devices. By reducing the need to display separate images from multiple sensor outputs to the user, they can benefit from an improved capture experience based on the varying advantages of multiple sensor outputs without increasing user complexity. Furthermore, the framework for handling sensor change requests can lead to improved image quality by allowing the use of a better lens to capture images from the multi-sensor capture device based on scene conditions. Switching to a better lens during zoom transitions provides the user with a better image in response to changes in scene conditions, rather than waiting to change image sensors at a fixed zoom level.

[0085] Adjust the output of the image sensor (e.g., as mentioned above and) Figure 4 (As described in box 412) Aligning the field of view of one image sensor with the output of another image sensor can be performed via a geometric deformation operation. Reference Figure 6 An example describing the geometric transformation from one image to another. Figure 6 An example data stream 600 is shown for a device used to perform spatial alignment transformations on one or more image frames. The device may be... Figure 1 The example implementation of device 100 shown is illustrated. Although data flow 600 is described with reference to the devices, components, and operations shown in the preceding figures, any suitable device or device component can perform the operations described for data flow 600.

[0086] The first image 602 and the second image 612 are captured by the first and second image sensors of the device 100, respectively, which are not shown for simplicity. In some embodiments, the first and second image sensors may be respectively... Figure 1 An example implementation of the first image sensor 101 and the second image sensor 102. As a non-limiting example, the first image 602 may be a wide-angle image captured at a first zoom ratio (e.g., 4.9X), and the second image 612 may be a telephoto image captured at a second zoom ratio (e.g., 5.0X). In some implementations, the first image 602 and the second image 612 may be captured at an initial resolution (e.g., 2304×1728) higher than the final output resolution of the images (e.g., 1920×1440). Figure 6 As shown, device 100 can crop images to a smaller resolution (e.g., 480×360) for further processing at image signal processor 112.

[0087] In some embodiments, the image signal processor 112 scales the first image 602 to match the field of view (FOV) of the second image 612 and identifies one or more spatial misalignments between the scaled first image 602 and the second image 612. Figure 6 As shown, the image signal processor 112 determines a transformation matrix based on one or more spatial misalignments, determines a confidence level associated with the transformation matrix, compares the confidence level to a confidence threshold, determines a weighting factor in response to a confidence level greater than the confidence threshold, and applies the weighting factor to the transformation matrix. In response to a confidence level greater than the confidence threshold, the image signal processor 112 may determine the weighting factor based on a first zoom ratio, a second zoom ratio, and the current zoom ratio of the device 100. In some embodiments, the weighting factor may be 0% when the current zoom level is equal to a first zoom ratio corresponding to the current zoom level at the time the sensor change request is received, and may be 100% when the current zoom ratio is equal to the requested zoom level from the zoom level change request or the zoom level corresponding to the end of a defined range. In some embodiments, the image signal processor 112 may avoid determining an additional transformation matrix when the confidence level is greater than the confidence threshold. In other embodiments, the weighting factor may be an identity matrix when the confidence level is not greater than the confidence threshold. Subsequently, the image signal processor 112 can send the weighted transformation matrix to the processor 104 in process 624.

[0088] Processor 104 may receive a first image at an initial resolution at process 626 and deform the first image into a second image using a weighted transformation matrix from image signal processor 112. In some embodiments, the final output resolution of the deformed first image (e.g., 1920 x 1440) may be smaller than the initial resolution of the first image 602 (e.g., 2304 x 1728). In this way, the first image 602 may have pixel margins between the final output resolution and the initial resolution. Therefore, when processor 104 deforms (e.g., shifts and / or rotates) the first image 602 into the second image, only pixels within the pixel margins can be shifted out of the frame or rotated out of the frame, so processor 104 can output the deformed image to the display of device 100 at process 630 without any FOV or quality loss, because only pixels within the pixel margins may be shifted out of the frame or rotated out of the frame. In some embodiments, device 100 may then generate a preview image based on the deformed first image and the second image 612. In some embodiments, the ISP 112 may perform a transformation from a first image to a second image and send the output frame generated by the transformation to the processor 104 for output to a display in process 630.

[0089] The geometry of data stream 600 can be adjusted based on a defined zoom level range, such as... Figure 3 As shown. For example, in Figure 3Within the first defined range, based on determining the current zoom level within the first defined range, the first image 602 can correspond to the source image sensor output, such that an adjusted image is obtained from the source sensor. As shown in data stream 600, in this example, the first image 602 corresponding to the source image sensor output is provided to processor 104 via process 626 to transform the first image 602 into a second image 612. As another example, in Figure 3 Within the second defined range, based on determining the current zoom level within the second defined range, the first image 602 can correspond to the output of the target image sensor, such that an adjusted image is obtained from the target sensor. As shown in data stream 600, in this example, the first image 602 corresponding to the output of the target image sensor is provided to the processor 104 via process 626 to transform the first image 602 into the second image 612.

[0090] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and methods. For example, the data, instructions, commands, information, signals, bits, symbols, and chips mentioned in all of the foregoing descriptions can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0091] This article is aimed at Figure 1 The components, functional blocks, and modules described include processors, electronic devices, hardware devices, electronic components, logic circuits, memory, software code, firmware code, and other examples, or any combination thereof. Furthermore, the features discussed herein can be implemented via dedicated processor circuitry, via executable instructions, or via a combination thereof.

[0092] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure. Those skilled in the art will also readily recognize that the order or combination of components, methods, or interactions described herein are merely examples, and that components, methods, or interactions of various aspects of this disclosure can be combined or performed in ways other than those shown and described herein.

[0093] The various illustrative logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software in terms of functionality has been generally described and illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether this functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the system as a whole.

[0094] Hardware and data processing means for implementing the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the various aspects disclosed herein may be implemented or performed by a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some embodiments, the processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some embodiments, specific processes and methods may be performed by circuitry dedicated to a given function.

[0095] In one or more aspects, the described functionality can be implemented in hardware, digital electronic circuits, computer software, firmware, including the structures disclosed in this specification and their equivalents, or any combination thereof. Embodiments of the subject matter described in this specification can also be implemented as one or more computer programs, i.e., one or more computer program instruction modules, encoded on a computer storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus.

[0096] If implemented in software, the functionality can be stored or transmitted as one or more instructions or code on a computer-readable medium. The processes of the methods or algorithms disclosed herein can be implemented in processor-executable software modules that may reside on a computer-readable medium. Computer-readable media include computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium accessible to a computer. Exemplarily, and not limitingly, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium capable of storing desired program code modules in the form of instructions or data structures and accessible to a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically reproduce data, while optical discs optically reproduce data using lasers. Combinations of the above are also included within the scope of computer-readable media. Furthermore, the operation of a method or algorithm may reside as one or any combination or set of code and instructions on a machine-readable and computer-readable medium, which may be incorporated into a computer program product.

[0097] Various modifications to the embodiments described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but are to be given the widest scope consistent with this disclosure, the principles disclosed herein, and the novel features.

[0098] Furthermore, those skilled in the art will readily understand that the terms “upper” and “lower” are sometimes used to facilitate the description of the drawings and to indicate relative positions corresponding to the orientation of the figures on a properly oriented page, and do not reflect the correct orientation of any implemented device.

[0099] Some features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although the features described above may be described as functioning in certain combinations, or even initially claimed in this way, in some cases one or more features in the claimed combination may be removed from that combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0100] Similarly, although operations are shown in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or to perform all shown operations to achieve the desired result. Furthermore, the drawings may schematically illustrate one or more example processes in the form of flowcharts. However, other operations not shown may be incorporated into the schematically shown example processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any shown operations. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments; it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. In addition, some other embodiments are within the scope of the following claims. In some cases, the operations recited in the claims may be performed in a different order, but the desired result may still be achieved.

[0101] As used herein, including in the claims, the term "or" when used in a list of two or more items means that any one of the listed items may be used alone, or any combination of two or more of the listed items may be used. For example, if a composition is described as containing component A, B, or C, the composition may contain A alone; B alone; C alone; a combination of A and B; a combination of A and C; a combination of B and C; or a combination of A, B, and C. Furthermore, as used herein, including in the claims, "or" in a list of items beginning with "at least one" indicates a separate list, such that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination thereof. As will be understood by those skilled in the art, the term “substantially” is defined as being largely but not necessarily entirely what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees and substantially parallel includes parallel). In any disclosed implementation, the term “substantially” may be replaced with “within [a certain percentage] of the specified transaction,” where the percentage includes 0.1%, 1%, 5%, or 10%.

[0102] The prior description of this disclosure is provided to enable those skilled in the art to implement or use it. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be given the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image processing, comprising: Receive a first request for conversion from the first zoom level to the second zoom level; Receive a second request for switching from a first sensor to a second sensor during image capture from a multi-sensor capture device, wherein the second request is received based on the first request; Determine the current zoom level corresponding to the second request to change from the first sensor to the second sensor; and Upon receiving the second request to switch from the first sensor to the second sensor, output image data is generated from the multi-sensor capture device via the following: Based on the current zoom level being determined to be within a first defined range, the image from the first sensor is adjusted, wherein adjusting the image includes: geometrically deforming the image from the first sensor to align it with the field of view associated with the second sensor; and Based on the current zoom level being determined to be within a second defined range, the image from the second sensor is adjusted, wherein the image from the second sensor is adjusted to align with the field of view associated with the first sensor.

2. The method according to claim 1, wherein, The adjustment of the image from the first sensor is based on a hybrid weighting parameter, and the method further includes: changing the hybrid weighting parameter to the second sensor after the hybrid weighting parameter reaches a threshold.

3. The method according to claim 1, wherein, The generation of the output image data from the multi-sensor capture device by adjusting the image from the first sensor is performed when the margin on the first sensor is higher than a threshold amount at the second zoom level.

4. The method according to claim 1, wherein, The second request to switch from the first sensor to the second sensor is based on the lighting conditions during the image capture.

5. The method according to claim 1, wherein, The second request to switch from the first sensor to the second sensor is based on movement detected during the image capture.

6. The method according to claim 1, wherein, The second request to switch from the first sensor to the second sensor is based on user input used to track the object during the image capture.

7. The method according to claim 1, wherein, The method further includes: Receive a request to switch from the second sensor to the third sensor during image capture from the multi-sensor capture device; Determine the second current zoom level corresponding to the received request to change from the second sensor to the third sensor; Determine whether the second current zoom level is within the range defined by the third; and Based on whether the second current zoom level is determined to be within the third defined range, the output image data is generated from the multi-sensor capture device by adjusting the image from one of the second or third sensors.

8. The method according to claim 1, wherein, The first sensor is configured to acquire an image through a first lens having a first optical axis, and wherein the second sensor is configured to acquire an image through a second lens having a second optical axis different from the first optical axis.

9. The method according to claim 8, wherein, The first lens has a first magnification, and the second lens has a second magnification different from the first magnification.

10. The method according to claim 1, wherein, The second request to change from the first sensor to the second sensor corresponds to the change during the transition from the first zoom level to the second zoom level.

11. The method according to claim 1, wherein, The second request is received simultaneously with the multi-sensor capture device switching from the current zoom level to the second zoom level.

12. An apparatus for image processing, comprising: processor; as well as A memory, coupled to the processor and storing instructions, which, when executed by the processor, cause the device to perform operations including: Receive a first request for conversion from the first zoom level to the second zoom level; Receive a second request for switching from a first sensor to a second sensor during image capture from a multi-sensor capture device, wherein the second request is received based on the first request; Determine the current zoom level corresponding to the second request to change from the first sensor to the second sensor; and Upon receiving the second request to switch from the first sensor to the second sensor, output image data is generated from the multi-sensor capture device via the following: Based on the current zoom level being determined to be within a first defined range, the image from the first sensor is adjusted, wherein adjusting the image includes: geometrically deforming the image from the first sensor to align it with the field of view associated with the second sensor; and Based on the current zoom level being determined to be within a second defined range, the image from the second sensor is adjusted, wherein the image from the second sensor is adjusted to align with the field of view associated with the first sensor.

13. The device according to claim 12, wherein, The adjustment of the image from the first sensor is based on a hybrid weighting parameter, and the execution of the instruction causes the device to perform an operation that further includes changing the hybrid weighting parameter to the second sensor after the hybrid weighting parameter reaches a threshold.

14. The device according to claim 12, wherein, The generation of the output image data from the multi-sensor capture device by adjusting the image from the first sensor is performed when the margin on the first sensor is higher than a threshold amount at the second zoom level.

15. The device according to claim 12, wherein, The second request to switch from the first sensor to the second sensor is based on the lighting conditions during the image capture.

16. The device according to claim 12, wherein, The second request to switch from the first sensor to the second sensor is based on movement detected during the image capture.

17. The device according to claim 12, wherein, The second request to switch from the first sensor to the second sensor is based on user input used to track the object during the image capture.

18. The device according to claim 12, wherein, The execution of the instruction causes the device to perform operations that also include the following: Receive a request to switch from the second sensor to the third sensor during image capture from the multi-sensor capture device; Determine the second current zoom level corresponding to the received request to change from the second sensor to the third sensor; Determine whether the second current zoom level is within the range defined by the third; as well as Based on whether the second current zoom level is determined to be within the third defined range, the output image data is generated from the multi-sensor capture device by adjusting the image from one of the second or third sensors.

19. The device according to claim 12, wherein, The first sensor is configured to acquire an image through a first lens having a first optical axis, and wherein the second sensor is configured to acquire an image through a second lens having a second optical axis different from the first optical axis.

20. The device according to claim 19, wherein, The first lens has a first magnification, and the second lens has a second magnification different from the first magnification.

21. The apparatus of claim 12, further comprising: The first sensor; as well as The second sensor, The first sensor has a larger field of view (FOV) than the second sensor.

22. The device according to claim 12, wherein, The first sensor includes a wide-angle image sensor, and the second sensor includes a telephoto image sensor.

23. The device according to claim 12, wherein, The second request to change from the first sensor to the second sensor corresponds to the change during the transition from the first zoom level to the second zoom level.

24. The device according to claim 12, wherein, The second request is received simultaneously with the multi-sensor capture device switching from the current zoom level to the second zoom level.

25. A non-transitory computer-readable medium storing instructions, which, when executed by a processor of a device, cause the device to perform operations including: Receive a first request for conversion from the first zoom level to the second zoom level; Receive a second request to switch from a first sensor to a second sensor during image capture from a multi-sensor capture device, wherein... The second request was received based on the first request; Determine the current zoom level corresponding to the second request to change from the first sensor to the second sensor; as well as Upon receiving the second request to switch from the first sensor to the second sensor, output image data is generated from the multi-sensor capture device via the following: Based on the current zoom level being determined to be within a first defined range, the image from the first sensor is adjusted, wherein adjusting the image includes: geometrically deforming the image from the first sensor to align it with the field of view associated with the second sensor; and Based on the current zoom level being determined to be within a second defined range, the image from the second sensor is adjusted, wherein the image from the second sensor is adjusted to align with the field of view associated with the first sensor.

26. The non-transitory computer-readable medium according to claim 25, wherein, The adjustment of the image from the first sensor is based on a hybrid weighting parameter, and the operation further includes: changing the hybrid weighting parameter to the second sensor after the hybrid weighting parameter reaches a threshold.

27. The non-transitory computer-readable medium according to claim 25, wherein, The generation of the output image data from the multi-sensor capture device by adjusting the image from the first sensor is performed when the margin on the first sensor is higher than a threshold amount at the second zoom level.

28. The non-transitory computer-readable medium according to claim 25, wherein, The second request to switch from the first sensor to the second sensor is based on the lighting conditions during the image capture.

29. The non-transitory computer-readable medium according to claim 25, wherein, The second request to switch from the first sensor to the second sensor is based on movement detected during the image capture.

30. The non-transitory computer-readable medium according to claim 25, wherein, The second request to switch from the first sensor to the second sensor is based on user input used to track the object during the image capture.

31. The non-transitory computer-readable medium according to claim 25, wherein, The execution of the instruction causes the device to perform operations that also include the following: Receive a request to switch from the second sensor to the third sensor during image capture from the multi-sensor capture device; Determine the second current zoom level corresponding to the received request to change from the second sensor to the third sensor; Determine whether the second current zoom level is within the range defined by the third; as well as Based on whether the second current zoom level is determined to be within the third defined range, the output image data is generated from the multi-sensor capture device by adjusting the image from one of the second or third sensors.

32. The non-transitory computer-readable medium according to claim 25, wherein, The first sensor is configured to acquire an image through a first lens having a first optical axis, and wherein the second sensor is configured to acquire an image through a second lens having a second optical axis different from the first optical axis.

33. The non-transitory computer-readable medium according to claim 32, wherein, The first lens has a first magnification, and the second lens has a second magnification different from the first magnification.

34. The non-transitory computer-readable medium according to claim 25, wherein, The second request to change from the first sensor to the second sensor corresponds to the change during the transition from the first zoom level to the second zoom level.

35. The non-transitory computer-readable medium according to claim 25, wherein, The second request is received simultaneously with the multi-sensor capture device switching from the current zoom level to the second zoom level.

Images