Camera conversion for image capture device with variable aperture capability

Abstract

This disclosure provides systems, methods, and devices for image capture that support image processing. In a first aspect, a method of image processing includes: receiving first image data from a first image sensor; receiving second image data from a second image sensor of a second camera, wherein at least one of the first image data or the second image data is received through a variable aperture (VA) having an adjusted aperture setting applied by the first camera or the second camera; and / or determining a first output frame based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data. Other aspects and features are also claimed and described.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to image processing, and more particularly to processing image data of an image capture device having multiple image sensors. Some features may enable and provide improved image processing, including improved consistency when switching between image sensors of an image capture device. Background Art

[0002] An image capture device is a device that can capture one or more digital images (whether still images for photographs or image sequences for videos). Capture devices can be incorporated into various devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computer devices (such as webcams, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0003] An image sensor on an image capture device is a component that includes an array of detectors that are configured to generate an electrical signal that is proportional to the amount of light received by those detectors. An image sensor may have a color filter array that matches the detector array so that each detector measures the intensity of a single color of light, which in some configurations may be the intensity of red, blue, and green. A lens associated with the image sensor directs the light to the image sensor. A lens is characterized by an aperture size, sometimes referred to as an f-stop, which defines the size of the diaphragm through which light passes to be directed to the image sensor. A larger aperture size, represented by a smaller f-stop, allows the image sensor to capture more light, which, for example, improves image quality in dark environments. A smaller aperture size, represented by a larger f-stop, allows less light to be captured, which, for example, increases the depth of focus (DOF), thereby bringing more of the scene content into focus. Summary of the invention

[0004] The following summarizes some aspects of the present disclosure to provide a basic understanding of the discussed technology. This summary is not an exhaustive overview of all expected features of the present disclosure, and is neither intended to identify key or important elements of all aspects of the present disclosure, nor to delineate the scope of any or all aspects of the present disclosure. Its sole purpose is to present some concepts of one or more aspects of the present disclosure in a summarized form as a prelude to more specific embodiments presented later.

[0005] The image capture device may include more than one image sensor to support different types of photography, such as portrait, night scene, landscape, and each of those image sensors may have a different associated lens system with different aperture sizes. In some aspects, one or more of the image sensors may have a variable aperture (VA) lens system that may be configured for use with the image sensor. The VA lens system allows the aperture size to be changed during operation of the image capture device, which provides the image capture device or user with control over the brightness of the captured photography and control over the depth of focus (DOF) of the captured photo.

[0006] A variable aperture camera module may provide improved quality in low light scenes and may provide a natural blur effect by allowing the user to configure the camera to different aperture sizes that receive different amounts of light with different amounts of blur. However, different aperture settings applied by the same camera will result in differences in texture / noise and DOF. Additionally, other cameras of the image capture device may have non-variable apertures (non-VA), which may result in abrupt changes in the form of significant DOF transitions when switching between cameras. The embodiments disclosed herein may apply image processing to switching cameras while providing DOF consistency when switching cameras by applying blur to background areas in non-VA image data when switching from a VA camera to a non-VA camera, and by setting the aperture of the VA camera to mirror the DOF of the non-VA camera when switching from a non-VA camera to a VA camera. By one or a combination of the techniques described in the embodiments herein, the DOF may appear more consistent when switching cameras.

[0007] In one aspect of the present disclosure, a method for image processing includes: receiving first image data from a first image sensor of a first camera; receiving second image data from a second image sensor of a second camera, wherein at least one of the first image data or the second image data is received through a variable aperture (VA) having an adjusted aperture setting applied by the first camera or the second camera; and determining a first output frame based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data.

[0008] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: receiving first image data from a first image sensor of a first camera; receiving second image data from a second image sensor of a second camera, wherein at least one of the first image data or the second image data is received through a variable aperture (VA) having an adjusted aperture setting applied by the first camera or the second camera; and / or determining a first output frame based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data.

[0009] In additional aspects of the present disclosure, an apparatus includes: a first camera, the first camera including a first image sensor; a second camera, the second camera including a second image sensor; a variable aperture, wherein the variable aperture is included in one of the first camera or the second camera; and at least one processor, the at least one processor coupled to the first camera and the second camera; and a memory, the memory coupled to the at least one processor, wherein the at least one processor is configured to perform operations including the following: receiving first image data from the first image sensor of the first camera; receiving second image data from the second image sensor of the second camera, wherein at least one of the first image data or the second image data is received through a variable aperture (VA) having an adjusted aperture setting applied by the first camera or the second camera; and / or determining a first output frame based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data.

[0010] In additional aspects of the present disclosure, an apparatus includes means for receiving first image data from a first image sensor of a first camera; means for receiving second image data from a second image sensor of a second camera, wherein at least one of the first image data or the second image data is received through a variable aperture (VA) having an adjusted aperture setting applied by the first camera or the second camera; and / or means for determining a first output frame based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data.

[0011] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations. The operations include: receiving first image data from a first image sensor of a first camera; receiving second image data from a second image sensor of a second camera, wherein at least one of the first image data or the second image data is received through a variable aperture (VA) having an adjusted aperture setting applied by the first camera or the second camera; and / or determining a first output frame based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data.

[0012] Image capture devices (devices that can capture one or more digital images, whether still images, photographs or video sequences of images) can be incorporated into a variety of devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computer devices (such as webcams, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0013] In general, the present disclosure describes image processing techniques for a digital camera having an image sensor and an image signal processor (ISP). The ISP may be configured to control the capture of image frames from one or more image sensors and process one or more image frames from the one or more image sensors to generate a view of a scene in a corrected image frame. The corrected image frame may be part of a sequence of image frames that forms a video sequence. The video sequence may include other image frames received from an image sensor or other image sensors and / or other corrected image frames based on input from an image sensor or another image sensor. In some embodiments, processing of one or more image frames may be performed within an image sensor, such as in a binning module. The image processing techniques described in the embodiments disclosed herein may be performed by circuits such as a binning module in an image sensor, in an image signal processor (ISP), in an application processor (AP), or in a combination of two or all of these components.

[0014] In one example, an image signal processor may receive instructions for capturing a sequence of image frames in response to loading of software (such as a camera application) to generate a preview display from an image capture device. The image signal processor may be configured to generate a single output frame stream based on image frames received from one or more image sensors. The single output frame stream may include raw image data from an image sensor, merged image data from an image sensor, or corrected image frames processed by one or more algorithms within the image signal processor (such as in a merging module). For example, image frames obtained from an image sensor (which may have some processing performed on the data before output to the image signal processor) may be processed in the image signal processor by processing the image frames through an image post-processing engine (IPE) and / or other image processing circuits for performing one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, etc.

[0015] After the image signal processor determines an output frame representing a scene using image correction (such as the merging described in various embodiments herein), the output frame may be displayed on a device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, sent over a network, and / or printed to an output medium. For example, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from different image sensors, and then generate corresponding output frames of image data (e.g., preview display frames, still image capture, frames for video, frames for object tracking, etc.). In other examples, the image signal processor may output frames of image data to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., auto focus (AF), automatic white balance (AWB), and automatic exposure control (AEC)), generate video files via output frames, configure frames for display, configure frames for storage, send frames over a network connection, etc. That is, an image signal processor may obtain incoming frames from one or more image sensors, each of which is coupled to one or more camera lenses, and may then generate and output an output frame stream to various output destinations.

[0016] In some aspects, a corrected image frame may be produced by combining aspects of the image correction of the present disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, a first image frame and a second image frame are captured using different exposure times, different apertures, different lenses, and / or other characteristics that may result in an improved dynamic range of the fused image when the two image frames are combined. In some aspects, the method may be performed for MFNR photography, wherein the first image frame and the second image frame are captured using the same or different exposure times, and the first image frame and the second image frame are fused to generate a corrected first image frame having reduced noise compared to the captured first image frame.

[0017] In some aspects, a device may include an image signal processor or processor (e.g., an application processor) that includes specific functionality for camera control and / or processing, such as enabling or disabling a merging module or otherwise controlling aspects of image correction. The methods and techniques described herein may be performed entirely by an image signal processor or processor, or various operations may be split between the image signal processor and the processor, and in some aspects split across additional processors.

[0018] The device may include one, two or more image sensors, such as including a first image sensor. When there are multiple image sensors, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a remote image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. The configuration may occur in a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located in offset positions on the front or back side of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image correction techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0019] In an additional aspect of the present disclosure, a device configured for image processing and / or image capture is disclosed. The device includes a component for capturing an image frame. The device also includes one or more components for capturing data representing a scene, such as an image sensor (including a charge coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor), a time-of-flight detector. The device may also include one or more components for gathering and / or focusing light into one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame input to the image processing techniques described herein.

[0020] For those of ordinary skill in the art, other aspects, features, and specific implementations will become apparent when reviewing the following description of specific exemplary aspects in conjunction with the accompanying drawings. Although each feature may be discussed below with respect to certain aspects and accompanying drawings, each aspect may include one or more of the advantageous features discussed herein. In other words, although one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used according to each aspect. In a similar manner, although the exemplary aspects may be discussed below as aspects of a device, system, or method, the exemplary aspects may be implemented in various devices, systems, and methods.

[0021] The method may be embedded in a computer readable medium as a computer program code including instructions for causing a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device including: a first network adapter configured to send data, such as an image or video as recorded data or as streaming data, over a first network connection of a plurality of network connections; and a processor coupled to the first network adapter and a memory. The processor may cause the corrected image frame described herein to be sent over a wireless communication network, such as a 5G NR communication network.

[0022] The features and technical advantages of examples according to the present disclosure have been outlined quite broadly above so that the following specific embodiments may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be easily used as a basis for modifying or designing other structures for the same purpose of achieving the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. When considered in conjunction with the accompanying drawings, the characteristics of the concepts disclosed herein (both their organization and methods of operation) and the associated advantages will be better understood from the following description. Each of the figures in the accompanying drawings is provided for the purpose of illustration and description and not as a definition of limitations to the claims.

[0023] Although various aspects and specific implementations are described in this application by illustrating some examples, it will be understood by those skilled in the art that additional specific implementations and use cases may occur in many other arrangements and scenarios. The innovations described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects and / or uses can be generated via integrated chip specific implementations and other devices based on non-module components (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail / purchase devices, medical devices, devices that enable artificial intelligence (AI), etc.). Although some examples may or may not specifically relate to use cases or applications, the innovations described may have various types of applicability. The scope of specific implementations may range from chip-level or module components to non-module, non-chip-level specific implementations, and further to aggregated, distributed, or original equipment manufacturer (OEM) devices or systems that combine one or more aspects of the described innovations. In some practical environments, the devices that combine the various aspects and features described may also necessarily include additional components and features for implementing and practicing the various aspects claimed and described. For example, the transmission and reception of wireless signals necessarily include multiple components for analog and digital purposes (e.g., hardware components including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.). The innovations described herein are intended to be practiced in a variety of devices, chip-level components, systems, distributed arrangements, end-user devices, etc., having different sizes, shapes, and configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] A further understanding of the nature and advantages of the present disclosure may be achieved by reference to the following drawings. In the drawings, similar components or features may have the same reference label. In addition, various components of the same type may be distinguished by adding a dash and a second label to distinguish between similar components after the reference label. If only the first reference label is used in the specification, the description applies to any one of the similar components having the same first reference label, regardless of the second reference label.

[0025] FIG. 1 shows a block diagram of an example device 100 for performing image capture from one or more image sensors.

[0026] 2A is an illustration of a process for switching from a VA camera to a non-VA camera, according to some embodiments of the present disclosure.

[0027] 2B is an illustration of a process for switching from a non-VA camera to a VA camera, according to some embodiments of the present disclosure.

[0028] 3 is a block diagram illustrating an image processing technique for converting between a non-VA camera and a VA camera according to some embodiments of the present disclosure.

[0029] 4 is a flow chart illustrating a method of processing image data during conversion between a non-VA camera and a VA camera according to some embodiments of the present disclosure.

[0030] 5 is a flow chart illustrating an example method of processing image data during a transition from a VA camera to a non-VA camera, according to some embodiments of the present disclosure.

[0031] 6 is a flow chart illustrating an example method of processing image data during a transition from a non-VA camera to a VA camera, according to some embodiments of the present disclosure.

[0032] The same reference numbers and designations in different drawings represent the same elements. DETAILED DESCRIPTION

[0033] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to limit the scope of the present disclosure. Instead, the specific embodiments include specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0034] The present disclosure provides systems, apparatus, methods, and computer-readable media that support image processing for camera conversion, including from a variable aperture (VA) camera to a non-VA camera and from a non-VA camera to a VA camera.

[0035] Specific implementations of the subject matter described in the present disclosure may be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, the present disclosure provides techniques for improving the user experience of operating an image capture device having multiple image sensors. Embodiments of the present disclosure may be applied to hide sudden changes in the appearance of an image during operation of an image capture device. The transition from one image sensor to another may be smoothed to reduce abrupt changes in a sequence of image frames generated by the image capture device. In embodiments where one or more of the image sensors are configured in a VA camera for variable aperture (VA) operation, the transition may be particularly advantageous because the VA camera may be adjusted as part of the transition. To smooth the transition, the aperture size of the VA camera may be predetermined based on a specific scene detected in the image data, and / or the aperture size of the VA camera may be changed before the camera switching time based on matching characteristics of the aperture size of a non-VA camera. Matching may refer to having characteristics that are more like the aperture size of a non-VA camera, which does not require a perfect match. When matching produces characteristics that are similar to those of a non-VA camera, such as having approximately the same image brightness and / or having image brightness similarities so that the image differences are not noticeable to the user, these characteristics may be considered matched.

[0036] An example device for capturing image frames using one or more image sensors, such as a smartphone, may include a configuration of two, three, four, or more cameras on the back side (e.g., the side opposite the user display) or the front side (e.g., the same side as the user display) of the device. One or more of the image sensors may be configured for variable aperture (VA) operation, such as where a variable aperture element is coupled to the image sensor as part of a camera module. A device with multiple image sensors includes one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., an AI engine), or other suitable circuitry for processing images captured by the image sensors. The one or more image signal processors may provide processed image frames to a memory and / or processor (such as an application processor, an image front end (IFE), an image processing engine (IPE), or other suitable processing circuitry) for further processing, such as for encoding, storage, transmission, or other manipulation.

[0037] As used herein, an image sensor may refer to the image sensor itself and any specific other components coupled to the image sensor for generating image frames for processing by an image signal processor or other logic circuitry or for storage in a memory (whether a short-term buffer or a long-term non-volatile memory). For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which is provided to a digital circuit coupled to the image sensor.

[0038] In the following description, a large number of specific details (such as examples of specific components, circuits and processes) are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of the present disclosure. However, it will be understood by those skilled in the art that these specific details may not be required to implement the teachings disclosed herein. In other examples, in order to avoid blurring the teachings of the present disclosure, known circuits and devices are shown in block diagram form.

[0039] Some portions of the following detailed description are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, programs, logic blocks, processes, etc. are conceived as a self-consistent sequence of steps or instructions leading to a desired result. These steps are steps requiring physical manipulations of physical quantities. Typically, although not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0040] In the accompanying drawings, a single box may be described as performing one or more functions. The one or more functions performed by the box may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, the following generally describes various exemplary components, blocks, modules, circuits, and steps in terms of their functionality. Whether such functionality is implemented as hardware or software depends on specific applications and the design constraints proposed for the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. Moreover, the example device may include components other than those shown, including known components such as processors, memories, etc.

[0041] Aspects of the present disclosure are applicable to any electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). In addition, aspects of the present disclosure may be implemented in image sensors or devices coupled to image sensors having the same or different capabilities and characteristics (e.g., resolution, shutter speed, sensor type, etc.). In addition, aspects of the present disclosure may be implemented in a device for processing image frames, whether or not the device includes or is coupled to an image sensor, such as a processing device that can retrieve stored images for processing, including a processing device present in a cloud computing system.

[0042] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "set," "generate," etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission, or display devices.

[0043] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more parts that can implement at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0044] FIG. 1 shows a block diagram of an example device 100 for performing image capture from one or more image sensors. The device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some implementations, the device 100 also includes or is coupled to a processor 104 and a memory 106 storing instructions 108. The device 100 may also include or be coupled to a display 114 and an input / output (I / O) component 116. The I / O component 116 may be used to interact with a user, such as a touch screen interface and / or physical buttons. The I / O component 116 may also include a network interface for communicating with other devices, including a wide area network (WAN) adapter 152, a local area network (LAN) adapter 153, and / or a personal area network (PAN) adapter 154. An example WAN adapter is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 153 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 154 is a Bluetooth wireless network adapter. Each of the adapters 152, 153, and / or 154 may be coupled to an antenna including multiple antennas configured for primary set reception and diversity reception and / or configured for receiving a particular frequency band. The device 100 may also include or be coupled to a power source 118 for the device 100, such as a battery or a component that couples the device 100 to an energy source. The device 100 may also include or be coupled to additional features or components not shown in FIG. 1 . In one example, a wireless interface that may include multiple transceivers and a baseband processor may be coupled to the WAN adapter 152 or included in the WAN adapter for use in a wireless communication device. In yet another example, an analog front end (AFE) for converting analog image frame data into digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

[0045] The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data about the movement of the device 100, data about the environment surrounding the device 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which may also be used to determine velocity and distance traveled by appropriately integrating the measured acceleration, and one or more of the acceleration, velocity, and / or distance may be included in the generated motion data. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub or directly to the image signal processor 112. In another example, the non-camera sensor may be a global positioning system (GPS) receiver.

[0046] The image signal processor 112 may receive image data such as for forming an image frame. In one embodiment, a local bus connection couples the image signal processor 112 to the image sensors 101 and 102 of the first camera and the second camera, respectively. In another embodiment, a wire interface couples the image signal processor 112 to an external image sensor. In yet another embodiment, a wireless interface couples the image signal processor 112 to the image sensors 101, 102.

[0047] The first camera may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may be controlled by an associated auto focus (AF) algorithm 133 executed in the ISP 112, which adjusts the lenses 131 and 132 to focus on a specific focal plane at a certain scene depth from the image sensors 101 and 102. The AF algorithm 133 may be assisted by a depth sensor 140.

[0048] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. The lenses 131 and 132 focus light at the image sensors 101 and 102, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, one or more color filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, and / or other suitable components for imaging. The first lens 131 and the second lens 132 may have different fields of view to capture different representations of the scene. For example, the first lens 131 may be an ultra-wide (UW) lens, and the second lens 132 may be a wide (W) lens. The plurality of image sensors may include a combination of ultra-wide (high field of view (FOV)) sensors, wide sensors, long-range sensors, and ultra-long-range (low FOV) sensors. That is, each image sensor may be configured by hardware configuration and / or software settings to obtain different but overlapping fields of view. In one configuration, the image sensor is configured with different lenses having different magnifications, which results in different fields of view. The sensors may be configured such that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV may capture a field of view in the range of 64 to 84 degrees, a sensor configured for an ultra-lateral FOV may capture a field of view in the range of 100 to 140 degrees, a sensor configured for a long-range FOV may capture a field of view in the range of 10 to 30 degrees, and a sensor configured for an ultra-long-range FOV may capture a field of view in the range of 1 to 8 degrees.

[0049] Image signal processor 112 processes image frames captured by image sensors 101 and 102. Although FIG. 1 illustrates device 100 including two image sensors 101 and 102 coupled to image signal processor 112, any number (e.g., one, two, three, four, five, six, etc.) of image sensors may be coupled to image signal processor 112. In some aspects, a depth sensor such as depth sensor 140 may be coupled to image signal processor 112, and the output from the depth sensor may be processed in a manner similar to that of image sensors 101 and 102. Furthermore, any number of additional image sensors or image signal processors may be present for device 100.

[0050] In some embodiments, the image signal processor 112 may execute instructions from a memory, such as instructions 108 from the memory 106, instructions stored in a separate memory coupled to or included in the image signal processor 112, or instructions provided by the processor 104. In addition or in the alternative, the image signal processor 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in the present disclosure. For example, the image signal processor 112 may include one or more image front ends (IFEs) 135, one or more image post-processing engines 136 (IPEs), and / or one or more automatic exposure compensation (AEC) 134 engines. The AF 133, AEC 134, AFE 135, APE 136 may each include dedicated circuitry embodied as software code executed by the ISP 112 and / or a combination of hardware within the ISP 112 and software code executed on the ISP 112.

[0051] In some implementations, the memory 106 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more operations described in the present disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 for generating images or videos. The instructions 108 may also include other applications or programs executed by the device 100, such as an operating system and specific applications other than for image or video generation. A camera application such as that executed by the processor 104 may cause the device 100 to generate images using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames, or may be accessed by the processor 104 to obtain processed frames. In some embodiments, the device 100 does not include the memory 106. For example, the device 100 may be a circuit including the image signal processor 112, and the memory may be external to the device 100. The device 100 may be coupled to an external memory and configured to access the memory to write output frames for display or long-term storage. In some embodiments, device 100 is a system on a chip (SoC) that combines image signal processor 112 , processor 104 , sensor hub 150 , memory 106 , and input / output components 116 into a single package.

[0052] In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform various operations described herein, including noise reduction operations. For example, execution of the instructions may instruct the image signal processor 112 to start or end capturing an image frame or a sequence of image frames, wherein the capture includes noise reduction as described in the embodiments herein. In some embodiments, the processor 104 may include one or more general-purpose processor cores 104A capable of executing scripts or instructions of one or more software programs (such as instructions 108 stored in the memory 106). For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 106.

[0053] When executing the camera application, the processor 104 can be configured to command the image signal processor 112 to perform one or more operations with reference to the image sensor 101 or 102. For example, the camera application can receive a command to start a video preview display, and upon receiving the command, capture and process a video including a sequence of image frames from one or more image sensors 101 or 102. Image correction, such as using cascaded IPEs, can be applied to one or more image frames in the sequence. Execution of instructions 108 by the processor 104 outside the camera application can also cause the device 100 to perform any number of functions or operations. In some embodiments, in addition to the ability to execute software to cause the device 100 to perform multiple functions or operations (such as the operations described herein), the processor 104 can also include an IC or other hardware (e.g., an artificial intelligence (AI) engine 124). In some other embodiments, the device 100 does not include the processor 104, such as when all of the described functionality is configured in the image signal processor 112.

[0054] In some embodiments, display 114 may include one or more suitable displays or screens that allow user interaction and / or present items to the user (such as previews of image frames captured by image sensors 101 and 102). In some embodiments, display 114 is a touch-sensitive display. I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from a user and provide output to the user through display 114. For example, I / O component 116 may include (but is not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (e.g., a power button), a slider, a switch, etc.

[0055] Although shown coupled to each other via processor 104, components such as processor 104, memory 106, image signal processor 112, display 114, and I / O component 116 may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. Although image signal processor 112 is illustrated as being separate from processor 104, image signal processor 112 may be a core of processor 104, which is an application processor unit (APU), included in a system on chip (SoC), or otherwise included in processor 104. Although device 100 is referenced in the examples herein to perform various aspects of the present disclosure, some device components may not be shown in FIG. 1 to prevent obscuring various aspects of the present disclosure. Additionally, other components, number of components, or combinations of components may be included in suitable devices for performing various aspects of the present disclosure. Therefore, the present disclosure is not limited to configurations of specific devices or components, including device 100.

[0056] In some embodiments of the image capture device, the first camera and / or the second camera may support variable aperture (VA) operation, allowing the aperture to change during the capture of image data. When one camera is configured for VA operation and the other camera does not support VA operation, there may be abrupt changes when switching image capture operation between a VA camera and a non-VA camera. For example, there may be brightness, texture, noise, and depth of focus (DOF) differences between the same image scene captured by a VA camera and a non-VA camera. Image processing may be performed to smooth the transition between VA and non-VA cameras, and vice versa, thereby reducing abrupt changes or artifacts in other outputs of user preview images, user recorded videos, user recorded photo sequences, and / or image frames from the image capture device from the image capture device. Figures 2A and 2B illustrate example image processing for transitioning between a VA camera to a non-VA camera and a non-VA camera to a VA camera, respectively.

[0057] FIG. 2A is an illustration of a process for switching from a VA camera to a non-VA camera according to some embodiments of the present disclosure. For example, the VA camera 220 may generate image data for forming an output image frame for display on a display of an image capture device. A transition to the non-VA camera 222 may be determined, after which image data from both the VA camera 220 and the non-VA camera 222 are used to form an output image frame for display to a user. During time period 202, a VA image frame including image data from the VA camera 220 is processed to match a non-VA image frame including image data from the non-VA camera 222. Next, during time period 204, a non-VA image frame including image data from the non-VA camera 222 is processed to match a VA image frame including image data from the VA camera 220. During time periods 202 and 204, processing may be performed to generate a smooth transition from displaying image data output from the VA camera 220 to displaying image data output from the non-VA camera 222. In some embodiments, the processing may include deformation, which refers to adjusting the image data of the source camera to match the field of view of the image data of the target camera. The amount of adjustment is determined over several image frames to smooth the transition between the different fields of view of VA camera 220 and non-VA camera 222. The process may also include other adjustments to smooth the transition between cameras, such as background blur, background sharpening, image brightness, texture, noise, and / or others.

[0058] In a similar manner, a process for switching from a non-VA camera to a VA camera may be performed. FIG. 2B is an illustration of a process for switching from a non-VA camera to a VA camera according to some embodiments of the present disclosure. For example, the non-VA camera 222 may generate image data for forming an output image frame for display on a display of an image capture device. A transition to the VA camera 220 may be determined, after which image data from both the non-VA camera 222 and the VA camera 220 are used to form an output image frame for display to a user. During time period 212, a non-VA image frame including image data from the non-VA camera 222 is processed to match a VA image frame including image data from the VA camera 220. Next, during time period 214, a VA image frame including image data from the VA camera 220 is processed to match a non-VA image frame including image data from the non-VA camera 222. During time periods 212 and 214, processing may be performed to generate a smooth transition from displaying image data output from the non-VA camera 222 to displaying image data output from the VA camera 220.

[0059] In some embodiments, transitioning from a non-VA camera to a VA camera or from a VA camera to a non-VA camera may include adjusting the aperture size of the VA camera. For example, when transitioning from a non-VA camera to a VA camera as in FIG. 2B , the aperture size of the VA camera may be adjusted during a portion of the time period 212 to reduce the difference in aperture size between the non-VA camera 222 and the VA camera 220. During and / or after the time period 214, the VA camera 220 may adjust the aperture size based on scene conditions, user input, or other criteria, rather than based on the aperture size of the non-VA camera 222. Similarly, when transitioning from a VA camera to a non-VA camera as in FIG. 2A , the aperture size of the VA camera may be adjusted during a portion of the time period 202 to reduce the difference in aperture size between the VA camera 220 and the non-VA camera 222.

[0060] Adjusting one image data captured from one camera to match the characteristics of other image data may be performed by the image capture device's processor 104. An example of such a configuration of a processor is shown in FIG3, but other logic circuits may be similarly configured to process image data, such as the IFE 135 of the ISP 112.

[0061] 3 is a block diagram illustrating an image processing technique for converting between a non-VA camera and a VA camera according to some embodiments of the present disclosure. The processor 104 may receive first image data (such as image data from a VA camera) and second image data (such as image data from a non-VA camera). After processing the first image data and the second image data, the processor 104 may generate an output image frame 310. In some embodiments, the output image frame 310 may be displayed on a display by a preview application executed on an image capture device. In some embodiments, the output image frame 310 may be output to a video file. In some embodiments, the output image frame 310 may be sent to a remote location via a network (e.g., a wireless communication link) to be displayed or stored at another location different from the location of the image capture device. In some embodiments, the output image frame 310 may involve multiple uses and / or locations and / or be further processed beyond the processing shown in FIG.

[0062] Processing of the first image data and the second image data may include processing by one or more modules 304A, 304B, 304C, and / or 304D. For example, processing may be performed in the background blur compensation module 304A. The background blur compensation module 304A may process the first image data to match the background blur of the second image data. When the first image data and the second image data are obtained from different cameras having different aperture sizes, the depth of focus (DOF) of the image data is different, resulting in different amounts of background blur in the scene as represented in the first image data and the second image data. Because a larger aperture size results in a shallower DOF in the background, when switching to a smaller aperture size, background blur may be added to the image data obtained from the smaller aperture size to match the DOF of the larger aperture size. The amount of background blur is then reduced until it is determined that the camera transition is complete, and an output image frame is generated based only on the image data from the camera having the smaller aperture size. Likewise, when switching from a camera with a smaller aperture size to a camera with a larger aperture size, background blur may gradually increase in subsequent frames of the output image frame 310 through the transition until the output image frame is generated based solely on image data from the camera with the larger aperture size.

[0063] As another example, processing may be performed in the image brightness alignment module 304B. The image brightness alignment module 304B may process the second image data to match the image brightness of the first image data. Because a larger aperture may produce a brighter image, the average brightness of the second image data may be increased when switching from a camera with a larger aperture size to a smaller aperture size. Then, the amount of image brightness increase may be gradually reduced to align the brightness with the smaller aperture of the first image data. For example, the brightness may be adjusted by increasing the gain of the second camera to match the first camera, and gradually reducing the gain. After the conversion is complete, the gain of the second camera may be automatically adjusted based on scene conditions without considering the brightness of the first camera. Similarly, when switching from a camera with a smaller aperture size to a camera with a larger aperture size, the gain may be reduced to reduce the image brightness in the second image data to match the image brightness of the first image data until an output image frame is generated based only on image data from a camera with a larger aperture size.

[0064] As another example, processing may be performed in the texture / noise alignment module 304C. The texture / noise alignment module 304C may process the second image data to match the texture or noise characteristics of the first image data. Because an image captured by a larger aperture may have higher sharpness but poorer noise levels, processing the second image data may include adjusting the sharpness on the first image data acquired from a camera with a smaller aperture size and / or adjusting impulse noise. The applied sharpness and / or noise level may then be adjusted over time to suit the camera corresponding to the first image data. Similarly, when switching from a smaller aperture size to a larger aperture size, the processing may gradually apply the sharpness and / or noise level to smooth the transition to the larger aperture size.

[0065] As another example, processing may be performed in the field of view deformation module 304D. The deformation module 304D may process the second image data to match the field of view of the first image data. Image data captured by two different cameras at different locations may have different fields of view. In addition, misalignment during manufacturing may further offset the field of view. Further, different zoom levels between the two cameras may cause the field of view to differ further. Processing of the second image data may include deforming the second image data by cropping, compositing, or otherwise adjusting the second image data to match the field of view of the first image data. The amount of such deformation may be reduced through the sequence of image frames until an output image frame is generated based only on the image data from the second camera, and the output image frame has a field of view corresponding to the position, zoom level, and other characteristics of the second camera.

[0066] In some embodiments, the processor 104 may have a configuration record that indicates the configuration of a first camera that acquired the first image data and the configuration of a second camera that acquired the second image data. For example, the configuration record may indicate an aperture size, an image sensor type, or other information that the processor 104 may use to determine an amount to adjust a characteristic of the second image data to match the same characteristic of the first image data. In the background blur compensation module 304A, the processor 104 may use the first aperture size of the first camera that acquired the first image data and the second aperture size of the second camera that acquired the second image data to determine an amount of background blur to add to the second image data to match the first image data.

[0067] In some embodiments, the processor 104 may receive metadata accompanying the first image data and the second image data, the metadata indicating the configuration of the first camera that acquired the first image data and the configuration of the second camera that acquired the second image data. For example, the first metadata may indicate an aperture size, an image sensor type, or other information that the processor 104 may use to determine an amount to adjust characteristics of the second image to match the same characteristics of the first image data. In the background blur compensation module 304A, the processor 104 may use the first aperture size of the first camera that acquired the first image data and the second aperture size of the second camera that acquired the second image data to determine an amount of background blur to add to the second image data to match the first image data.

[0068] A method according to some embodiments described herein is shown in FIG4. FIG4 is a flow chart illustrating a method of processing image data during conversion between a non-VA camera and a VA camera according to some embodiments of the present disclosure. Method 400 includes receiving first image data from a first image sensor at block 402. Processor 104 may receive the first image data from a first camera having a first image sensor 101 via image signal processor 112 and / or memory 106.

[0069] At block 404, second image data is received from a second image sensor, wherein at least one of the first image data or the second image data is received through a variable aperture. The variable aperture may be a portion of a camera including a corresponding one of the first image sensor or the second image sensor, and in some embodiments may be an additional camera component, such as a lens. The variable aperture may be any component capable of reducing the actual or effective aperture size of image data captured by the corresponding image sensor, such as a set of adjustable blades or a shutter. The variable aperture may be controlled between different aperture sizes according to an adjusted aperture setting applied by the camera, which may be controlled by a component external to the camera, such as an image signal processor 112. An image sensor coupled to a variable aperture for receiving light representing a scene through the variable aperture may be considered to be an image sensor configured with variable aperture (VA) operation. The processor 104 may receive the second image data from a second camera having a second image sensor 102 through the image signal processor 112 and / or the memory 106.

[0070] At block 406, a first output image frame is determined based on the first image data and the second image data by adjusting the second image data to match a characteristic of the first image data. The processing of block 406 may be performed, for example, by a processor 104 configured at least in part as shown in FIG. 3. For example, adjusting the second image data may include adjusting the second image data to match a depth of focus (DOF) of the first image data. For another example, adjusting the second image data may include deforming the second image data to match a field of view (FOV) of the first image data. For another example, adjusting the second image data may include adjusting the second image data to match an image brightness of the first image data.

[0071] In some embodiments, method 400 may be initiated to switch from a first camera to a second camera. For example, prior to box 406, method 400 may include determining to switch from the first image sensor to the second image sensor before determining the first output frame in box 406. After determining the first output image frame, the method may continue to process the image data to change the adjustment amount so that a smooth transition from the characteristics of the first camera to the characteristics of the second camera is obtained in the output image frame. For example, method 400 may also include receiving third image data from the first image sensor and receiving fourth image data from the second image sensor, which may be used to determine a first plurality of output frames based on the third image data and the fourth image data for switching between a VA image sensor and a non-VA image sensor. After determining that the transition is complete, method 400 may continue to process additional image data from the second camera to determine the output image frame, such as by receiving fifth image data from the second image sensor and determining a second plurality of output frames based on the fifth image data without reference to the image data from the first image sensor.

[0072] Additional details of an embodiment of processing when switching from a VA camera to a non-VA camera are described with reference to FIG. 5 . FIG. 5 is a flowchart illustrating an example method of processing image data during a transition from a VA camera to a non-VA camera according to some embodiments of the present disclosure. In the operation of FIG. 5 , the master camera is a VA camera and the slave camera is a non-VA camera. Method 500 may include determining a difference between a first aperture size of a first image sensor and a second aperture size of a second image sensor at box 502. For example, a processor may determine a difference in texture, noise, and / or depth of focus between the two aperture sizes. The determination may include estimating a relationship between texture / noise levels and blur by comparing the gradients between the two aperture sizes extracted. The gradient may be used to determine a noise level and / or a texture level, which levels may be used as a measure of clarity and sharpness of the image data. When switching from a first camera to a second camera, the gradient relationship between the two image data may be used to determine an amount of matching applied to the second image data.

[0073] At block 504, the second image data is adjusted to match the depth of focus (DOF) of the first image data, and possibly other characteristics of the first image data at block 506. The adjustments of blocks 504 and 506 may be applied to each image frame acquired from the first image sensor and the second image sensor when performing the transition. An output image frame may be determined based on the adjusted second image data at block 508, where the output image frame is based on both the first image data and the second image data because the adjustments of blocks 504 and 506 are based on the first image data. At block 510, the method 500 continues to determine additional output image frames to transition from VA operation to non-VA operation, such as by gradually decreasing the amount of adjustment at blocks 504 and 506 to reach a time when the second image data is not adjusted based on the first image data.

[0074] Additional details of an embodiment of processing when switching from a non-VA camera to a VA camera are described with reference to FIG. 6 . FIG. 6 is a flowchart illustrating an example method of processing image data during a transition from a non-VA camera to a VA camera according to some embodiments of the present disclosure. In the operation of FIG. 6 , the slave camera is a VA camera and the master camera is a non-VA camera. Method 600 includes determining a transition to a VA camera and a target aperture size for the VA camera at block 602 . Block 602 may include detecting a scene type, such as determining conditions such as a low-light scene, a multi-depth scene, or an infinite scene by a scene detection algorithm based on an AEC lux index and / or depth information and an environmental model. For example, when a low-light scene is detected, a larger aperture size of the VA camera than the current non-VA camera may be set as the target aperture size. For another example, when a multi-depth scene is detected, a smaller aperture size of the VA camera than the current non-VA camera may be set as the target aperture size. For another example, when an infinite scene is detected, when a first criterion (such as a preference for better corner sharpness) is met, a smaller aperture size of the VA camera than the current non-VA camera may be set as the target aperture size. Otherwise, a larger aperture size of the VA camera than the current non-VA camera may be set as the target aperture size when a second criterion is met, such as not favoring better corner sharpness.

[0075] The transition may then be initiated and an output image frame determined based on the first image data and the second image data. At block 604, the VA camera is adjusted to match the target aperture size determined at block 602. At block 606, the second image data from the VA camera is adjusted to match one or more characteristics of the first image data from the non-VA camera. At block 608, an output image frame is determined based on the adjusted second image data. In some embodiments, the VA camera configuration of block 604 may be performed during the generation of output image frames according to blocks 606 and 608. Additional output image frames may be determined to transition from non-VA operation to VA operation until output image frames are generated based solely on the second image data of the VA camera.

[0076] In one or more aspects, techniques for supporting image capture and / or image processing may include additional aspects, such as any single aspect or any combination of aspects described below or in conjunction with one or more other processes or devices described elsewhere herein. In a first aspect, supporting image capture may include an apparatus configured to capture a scene via multiple cameras, one or more of which may have a variable aperture. The apparatus is further configured to perform operations including: receiving first image data from a first image sensor of a first camera; receiving second image data from a second image sensor of a second camera, wherein at least one of the first image data or the second image data is received via a variable aperture (VA) having an adjusted aperture setting applied by the first camera or the second camera; and determining a first output frame based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data.

[0077] Additionally, the device may be performed or operated according to one or more aspects as described below. In some specific implementations, the device includes a wireless device, such as a UE. In some specific implementations, the device may include at least one processor and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other specific implementations, the device may include a non-transitory computer-readable medium having a program code recorded thereon, and the program code may be executable by a computer to enable the computer to perform the operations described herein with reference to the device. In some specific implementations, the device may include one or more components configured to perform the operations described herein. In some specific implementations, a wireless communication method for image data (including photos and / or videos) may include one or more operations described herein with reference to the device.

[0078] In a second aspect, in combination with the first aspect, adjusting the second image data includes adjusting the second image data to match a depth of focus (DOF) of the first image data.

[0079] In a third aspect, in combination with one or more of the first or second aspects, adjusting the second image data further comprises deforming the second image data to match a field of view (FOV) of the first image data.

[0080] In a fourth aspect, in combination with one or more of the first to third aspects, the apparatus may be further configured to determine to switch from the first image sensor to the second image sensor before determining the first output frame.

[0081] In the fifth aspect, in combination with one or more aspects of the first to fourth aspects, the device may also be configured to: receive third image data from the first image sensor after determining the first output frame, receive fourth image data from the second image sensor, determine a first plurality of output frames based on the third image data and the fourth image data for conversion between a VA camera and a non-VA camera, receive fifth image data from the second image sensor, and determine a second plurality of output frames based on the fifth image data.

[0082] In a sixth aspect, in combination with one or more aspects of the first to fifth aspects, the apparatus may be further configured to determine to switch from the first image sensor to the second image sensor before determining the first output frame, wherein the first camera includes a variable aperture (VA), wherein adjusting the second image data includes adjusting the second image data to match the background blur of the first image data.

[0083] In a seventh aspect, in combination with one or more of the first to sixth aspects, adjusting the second image data includes adjusting the second image data to match a depth of focus (DOF) of the first image data.

[0084] In an eighth aspect, in combination with one or more of the first to seventh aspects, adjusting the second image data further comprises adjusting the second image data to match the image brightness of the first image data.

[0085] In a ninth aspect, in combination with one or more aspects of the first to eighth aspects, the apparatus may also be configured to determine switching from the first image sensor to the second image sensor before determining the first output frame, wherein the second camera includes a variable aperture (VA); and adjust a second aperture size of the second camera to match a first aperture size of the first camera.

[0086] In a tenth aspect, in combination with one or more of the first to ninth aspects, adjusting the second image data includes adjusting the second image data to match a depth of focus (DOF) of the first image data.

[0087] In an eleventh aspect, in combination with one or more of the first to tenth aspects, adjusting the second image data further comprises adjusting the second image data to match the image brightness of the first image data.

[0088] It should be understood by those skilled in the art that any of a variety of different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0089] The components, functional blocks and modules described herein with respect to Figures 1 to 6 include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software codes, firmware codes, etc., or any combination thereof. Software should be broadly interpreted as meaning instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, and / or functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or other terms. In addition, the features discussed herein may be implemented via dedicated processor circuits, via executable instructions, or a combination thereof.

[0090] The technician will further understand that the various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the present disclosure can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. The technician will also easily recognize that the order or combination of the components, methods or interactions described herein are merely examples, and the components, methods or interactions of various aspects of the present disclosure can be combined or performed in a manner other than the manner illustrated and described herein.

[0091] The various illustrative logical components, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0092] The hardware and data processing apparatus for implementing the various illustrative logic components, logic boxes, modules and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller or state machine. In some specific implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In some specific implementations, specific processes and methods may be performed by circuits specific to a given function.

[0093] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents, or in any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage media for execution by data processing apparatus or for controlling the operation of data processing apparatus.

[0094] If implemented in software, the function may be stored as one or more instructions or codes on a computer-readable medium or sent through a computer-readable medium. The process of the method or algorithm disclosed herein may be implemented in a processor executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that can be implemented to transfer a computer program from one place to another. The storage medium may be any available medium that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media. Additionally, the operations of a method or algorithm may be located as a code and instruction set, or any combination of code and instruction sets, on a machine-readable medium or computer-readable medium, which may be incorporated into a computer program product.

[0095] Various modifications to the specific implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific implementations without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the specific implementations shown herein, but are to be consistent with the broadest scope consistent with this disclosure, the principles and novel features disclosed herein.

[0096] Additionally, one of ordinary skill in the art will readily appreciate that the terms "upper" and "lower" are sometimes used for ease of describing the drawings and indicate relative positions on a correctly oriented page corresponding to the orientation of the drawings, and may not reflect the correct orientation of any device as implemented.

[0097] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to a subcombination or variations of a subcombination.

[0098] Similarly, although the operations are depicted in a specific order in the figure, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all the illustrated operations to achieve the desired result. In addition, the accompanying drawings can schematically depict one or more example processes in the form of a flow chart. However, other operations that are not depicted can be incorporated into the schematically illustrated example process. For example, one or more additional operations can be performed before, after, simultaneously or between any illustrated operations. In some cases, multitasking and parallel processing are advantageous. In addition, the separation of various system components in the specific implementation described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific implementations also fall within the scope of the appended claims. In some cases, the actions narrated in the claims can be performed in different orders and still achieve the desired result.

[0099] As used herein, including in the claims, the term "or" used in a list of two or more items means that any of the listed items may be taken alone, or any combination of two or more of the listed items may be taken. For example, if a composition is described as comprising components A, B, or C, the composition may comprise A alone; B alone; C alone; A and B in combination; A and C in combination; B and C in combination; or A, B, and C in combination. In addition, as used herein, including in the claims, "or" as used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these. The term "substantially" is defined as to a large extent, but not necessarily entirely, what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any disclosed implementation, the term "substantially" (or "approximately") may be used in place of "within [percent] of" the specified content, where the percentage includes 0.1%, 1%, 5%, or 10%.

[0100] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Therefore, the disclosure is not intended to be limited to the examples and designs described herein, but to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method comprising: receiving first image data from a first image sensor of a first camera; receiving second image data from a second image sensor of a second camera, wherein at least one of the first image data or the second image data is received through a variable aperture having an adjusted aperture setting applied by the first camera or the second camera; A first output frame is determined based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data. 2 . The method of claim 1 , wherein adjusting the second image data comprises adjusting the second image data to match a depth of focus (DOF) of the first image data. 3 . The method of claim 2 , wherein adjusting the second image data further comprises deforming the second image data to match a field of view (FOV) of the first image data.

4. The method according to claim 2, further comprising: determining to switch from the first image sensor to the second image sensor before determining the first output frame; After determining the first output frame: receiving third image data from the first image sensor; receiving fourth image data from the second image sensor; determining a first plurality of output frames for switching between a variable aperture (VA) camera and a non-variable aperture (VA) camera based on the third image data and the fourth image data; receiving fifth image data from the second image sensor; as well as A second plurality of output frames is determined based on the fifth image data.

5. The method according to claim 1, further comprising: Before determining the first output frame, determining to switch from the first image sensor to the second image sensor, wherein the first camera includes the variable aperture, wherein adjusting the second image data includes adjusting the second image data to match a background blur of the first image data. The method of claim 5 , wherein adjusting the second image data comprises adjusting the second image data to match a depth of focus (DOF) of the first image data. 7 . The method of claim 6 , wherein adjusting the second image data further comprises adjusting the second image data to match an image brightness of the first image data.

8. The method according to claim 1, further comprising: Before determining the first output frame: determining to switch from the first image sensor to the second image sensor, wherein the second camera includes the variable aperture (VA); and The second aperture size of the second camera is adjusted to match the first aperture size of the first camera. 9 . The method of claim 8 , wherein adjusting the second image data comprises adjusting the second image data to match a depth of focus (DOF) of the first image data.

10. The method of claim 9, wherein adjusting the second image data further comprises adjusting the second image data to match an image brightness of the first image data.

11. An apparatus comprising: a first camera, the first camera comprising a first image sensor; a second camera, the second camera comprising a second image sensor; a variable aperture, wherein the variable aperture is included in one of the first camera or the second camera; a memory storing processor readable code; and at least one processor coupled to the memory, the first camera, and the second camera, wherein the at least one processor is configured to execute the processor readable code to cause the at least one processor to perform operations including: receiving first image data from the first image sensor; receiving second image data from the second image sensor, wherein at least one of the first image data or the second image data is received through the variable aperture according to an adjusted aperture setting applied by the first camera or the second camera; A first output frame is determined based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data. 12 . The apparatus of claim 11 , wherein adjusting the second image data comprises adjusting the second image data to match a depth of focus (DOF) of the first image data. 13 . The apparatus of claim 12 , wherein adjusting the second image data further comprises deforming the second image data to match a field of view (FOV) of the first image data.

14. The apparatus of claim 12, wherein the at least one processor is configured to perform further operations comprising: determining to switch from the first camera to the second camera before determining the first output frame; After determining the first output frame: receiving third image data from the first image sensor; receiving fourth image data from the second image sensor; determining a first plurality of output frames for switching between the first camera and the second camera based on the third image data and the fourth image data; receiving fifth image data from the second image sensor; as well as A second plurality of output frames is determined based on the fifth image data.

15. The apparatus of claim 14, wherein the at least one processor is configured to perform additional operations comprising: prior to determining the first output frame, determining to switch from the first camera to the second camera, wherein the first camera comprises the variable aperture (VA), wherein adjusting the second image data comprises adjusting the second image data to match a background blur of the first image data. 16 . The apparatus of claim 15 , wherein adjusting the second image data comprises adjusting the second image data to match a depth of focus (DOF) of the first image data.

17. The apparatus of claim 16, wherein adjusting the second image data further comprises adjusting the second image data to match image brightness of the first image data.

18. The apparatus of claim 11, wherein the at least one processor is configured to perform further operations comprising: Before determining the first output frame: determining to switch from the first camera to the second camera, wherein the second camera includes the variable aperture (VA); and The second aperture size of the second camera is adjusted to match the first aperture size of the first camera.

19. The apparatus of claim 18, wherein adjusting the second image data comprises adjusting the second image data to match a depth of focus (DOF) of the first image data.

20. The apparatus of claim 19, wherein adjusting the second image data further comprises adjusting the second image data to match image brightness of the first image data.

21. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving first image data from a first image sensor of a first camera; receiving second image data from a second image sensor of a second camera, wherein at least one of the first image data or the second image data is received through a variable aperture (VA) having an adjusted aperture setting applied by the first camera or the second camera; A first output frame is determined based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data.

22. The non-transitory computer readable medium of claim 21, wherein adjusting the second image data comprises adjusting the second image data to match a depth of focus (DOF) of the first image data.

23. The non-transitory computer readable medium of claim 22, wherein adjusting the second image data further comprises warping the second image data to match a field of view (FOV) of the first image data.

24. The non-transitory computer readable medium of claim 22, wherein the operations further comprise one or more of the following: determining to switch from the first image sensor to the second image sensor before determining the first output frame; After determining the first output frame: receiving third image data from the first image sensor; receiving fourth image data from the second image sensor; determining a first plurality of output frames for switching between a VA camera and a non-VA camera based on the third image data and the fourth image data; receiving fifth image data from the second image sensor; and A second plurality of output frames is determined based on the fifth image data.

25. The non-transitory computer readable medium of claim 21, wherein the operations further comprise one or more of the following: Before determining the first output frame: determining to switch from the first image sensor to the second image sensor, wherein the second camera includes the variable aperture (VA); and The second aperture size of the second camera is adjusted to match the first aperture size of the first camera.

26. An apparatus comprising: a memory storing processor readable code; and at least one processor coupled to the memory and configured to execute the processor readable code so that the at least one processor: receiving first image data from a first image sensor of a first camera; receiving second image data from a second image sensor of a second camera, wherein at least one of the first image data or the second image data is received through a variable aperture (VA) having an adjusted aperture setting applied by the first camera or the second camera; A first output frame is determined based on the first image data and the second image data by adjusting the second image data to match characteristics of the first image data.

27. The apparatus of claim 26, wherein adjusting the second image data comprises adjusting the second image data to match a depth of focus (DOF) of the first image data.

28. The apparatus of claim 27, wherein adjusting the second image data further comprises warping the second image data to match a field of view (FOV) of the first image data.

29. The apparatus of claim 27, wherein the at least one processor is configured to perform further operations comprising: determining to switch from the first image sensor to the second image sensor before determining the first output frame; After determining the first output frame: receiving third image data from the first image sensor; receiving fourth image data from the second image sensor; determining a first plurality of output frames for switching between a VA camera and a non-VA camera based on the third image data and the fourth image data; receiving fifth image data from the second image sensor; and A second plurality of output frames is determined based on the fifth image data.

30. The apparatus of claim 26, wherein the at least one processor is configured to perform further operations comprising: Before determining the first output frame: determining to switch from the first image sensor to the second image sensor, wherein the second camera includes the variable aperture (VA); and The second aperture size of the second camera is adjusted to match the first aperture size of the first camera.

Images