Image processing for aperture size transitions in variable aperture (VA) cameras

Abstract

This disclosure provides systems, methods, and devices for image signal processing that support variable aperture (VA) camera operation. In a first aspect, an image processing method includes receiving first image data acquired at a first aperture size from a variable aperture (VA) camera; determining a first set of output image frames based on the first image data; receiving, from the VA camera, second image data acquired during the aperture size transition; and determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting a characteristic of the second image data based on the first image data. Other aspects and features are also claimed and described.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 17 / 809,712, filed on June 29, 2022, entitled “IMAGE PROCESSING FOR APERTURE SIZE TRANSITION IN A VARIABLEAPERTURE (VA) CAMERA,” the disclosure of which is hereby incorporated by reference into this document. Technical Field

[0003] Aspects of the present disclosure relate generally to image processing, and more particularly to image processing for variable aperture cameras. Some features may enable and provide improved image processing, including improved brightness matching during aperture size transitions. Background Art

[0004] An image capture device is a device that can capture one or more digital images (whether still images for photographs or image sequences for videos). Capture devices can be incorporated into various devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computer devices (such as webcams, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0005] The image capture device may include multiple cameras with different fixed aperture sizes configured to provide different representations of a scene. For example, a larger aperture size may produce a shorter depth of focus, suitable for portrait photography with a blurred background. For another example, a smaller aperture size may produce a larger depth of focus, suitable for landscape photography with a clear foreground and background. The image capture device may switch between cameras with different fixed aperture sizes based on user input to select a photography type. Summary of the invention

[0006] The following summarizes some aspects of the present disclosure to provide a basic understanding of the technology discussed. This summary is not an exhaustive overview of all expected features of the present disclosure, and is neither intended to identify key elements or important elements of all aspects of the present disclosure, nor to delineate the scope of any or all aspects of the present disclosure. Its sole purpose is to present some concepts of one or more aspects of the present disclosure in a summarized form as a prelude to more specific embodiments presented later.

[0007] Adding a camera to an image capture device may increase cost and increase physical size. A variable aperture (VA) camera may be used to replace multiple fixed aperture size cameras, or to provide additional functionality to a fixed aperture size camera. In a VA camera, the aperture size may be variable, such as between discrete sizes or as a continuously variable size. The image capture device may control the VA camera to obtain a photo or video representing a scene with desired characteristics. In a VA camera, the sensitivity of the camera (which may include an aperture as well as a lens and an image sensor) may be different across different aperture settings. When a VA camera changes the aperture to obtain a representation of a scene with one desired characteristic (e.g., a clear or blurred background), changing the aperture size may result in undesirable changes in other characteristics of the scene representation. For example, reducing the aperture size to obtain a clear background may result in a lower brightness obtained by the VA camera at a larger aperture size. The techniques described herein may be used to process image data from a VA camera to reduce or eliminate undesirable changes in image characteristics caused by using a VA camera, particularly during aperture size transitions of the VA camera.

[0008] In one aspect of the present disclosure, a method for image processing includes: receiving first image data acquired at a first aperture size; determining a first set of output image frames based on the first image data; receiving second image data acquired during an aperture size transition; and determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting a characteristic (e.g., brightness) of the second image data based on the first image data. The first set of output image frames and the second set of output image frames may form an image sequence and may be displayed as an output preview in a camera application to view the output of an image capture device. The first set of output image frames and the second set of output image frames may reduce interruptions in the appearance of the preview during the aperture size transition.

[0009] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: receiving first image data acquired at a first aperture size; determining a first set of output image frames based on the first image data; receiving second image data acquired during an aperture size transition; and determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting a characteristic (e.g., brightness) of the second image data based on the first image data.

[0010] In additional aspects of the present disclosure, an apparatus includes: a component for receiving first image data acquired at a first aperture size; a component for determining a first set of output image frames based on the first image data; a component for receiving second image data acquired during an aperture size transition; and a component for determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting a characteristic (e.g., brightness) of the second image data based on the first image data.

[0011] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including: receiving first image data acquired at a first aperture size; determining a first set of output image frames based on the first image data; receiving second image data acquired during an aperture size transition; and determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting a characteristic (e.g., brightness) of the second image data based on the first image data.

[0012] Image capture devices (devices that can capture one or more digital images, whether still images, photographs or video sequences of images) can be incorporated into a variety of devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, camera-equipped wireless communication devices, cell phones (such as mobile phones, cellular or satellite radio phones), personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computer devices (such as webcams, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0013] In general, the present disclosure describes image processing techniques for digital cameras having image sensors and image signal processors (ISPs). The ISP may be configured to control the capture of image frames from one or more image sensors and process one or more image frames from the one or more image sensors to generate a view of a scene in a corrected image frame. The corrected image frame may be part of a sequence of image frames that forms a video sequence. A video sequence may include other image frames received from an image sensor or other image sensors and / or other corrected image frames based on inputs from an image sensor or another image sensor. In some embodiments, processing of one or more image frames may be performed within an image sensor, such as in a binning module. The image processing techniques described in the embodiments disclosed herein may be performed by circuits such as a binning module in an image sensor, in an image signal processor (ISP), in an application processor (AP), or in a combination of two or all of these components.

[0014] In one example, an image signal processor may receive instructions for capturing a sequence of image frames in response to loading of software (such as a camera application) to generate a preview display from an image capture device. The image signal processor may be configured to generate a single output frame stream based on image frames received from one or more image sensors. The single output frame stream may include raw image data from an image sensor, merged image data from an image sensor, or corrected image frames processed by one or more algorithms within the image signal processor (such as in a merging module). For example, image frames obtained from an image sensor (which may have performed some processing on the data before output to the image signal processor) may be processed in the image signal processor by processing the image frames through an image post-processing engine (IPE) and / or other image processing circuits for performing one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, etc.

[0015] After the image signal processor determines the output frames representing the scene using image correction (such as the merging described in various embodiments herein), the output frames may be displayed on a device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, sent over a network, and / or printed to an output medium. For example, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from different image sensors, and then generate corresponding output frames of image data (e.g., preview display frames, still image capture, frames for video, frames for object tracking, etc.). In other examples, the image signal processor may output frames of image data to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., auto focus (AF), automatic white balance (AWB), and automatic exposure control (AEC)), generate video files via output frames, configure frames for display, configure frames for storage, send frames over a network connection, etc. That is, an image signal processor may obtain incoming frames from one or more image sensors, each of which is coupled to one or more camera lenses, and may then generate and output an output frame stream to various output destinations.

[0016] In some aspects, a corrected image frame may be produced by combining aspects of the image correction of the present disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, a first image frame and a second image frame are captured using different exposure times, different apertures, different lenses, and / or other characteristics that may result in an improved dynamic range of the fused image when the two image frames are combined. In some aspects, the method may be performed for MFNR photography, wherein the first image frame and the second image frame are captured using the same or different exposure times, and the first image frame and the second image frame are fused to generate a corrected first image frame having reduced noise compared to the captured first image frame.

[0017] In some aspects, a device may include an image signal processor or processor (e.g., an application processor) that includes specific functionality for camera control and / or processing, such as enabling or disabling a merging module or otherwise controlling aspects of image correction. The methods and techniques described herein may be performed entirely by an image signal processor or processor, or various operations may be split between the image signal processor and the processor, and in some aspects split across additional processors.

[0018] The device may include one, two or more image sensors, such as including a first image sensor. When there are multiple image sensors, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a remote image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. The configuration may occur in a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located in offset positions on the front or back side of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image correction techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0019] In an additional aspect of the present disclosure, a device configured for image processing and / or image capture is disclosed. The device includes a component for capturing an image frame. The device also includes one or more components for capturing data representing a scene, such as an image sensor (including a charge coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor), a time-of-flight detector. The device may also include one or more components for gathering and / or focusing light into one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame input to the image processing techniques described herein.

[0020] Other aspects, features, and implementations will become apparent to those of ordinary skill in the art upon reviewing the following description of specific exemplary aspects in conjunction with the accompanying drawings. Although features may be discussed below with respect to certain aspects and drawings, various aspects may include one or more of the advantageous features discussed herein. In other words, while one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used according to various aspects. In a similar manner, although exemplary aspects may be discussed below as aspects of devices, systems, or methods, exemplary aspects may be implemented in various devices, systems, and methods.

[0021] The method may be embedded in a computer readable medium as a computer program code including instructions for causing a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device comprising: a first network adapter configured to send data, such as an image or video as recorded data or as streaming data, over a first network connection of a plurality of network connections; and a processor coupled to the first network adapter and the memory. The processor may cause the corrected image frame described herein to be sent over a wireless communication network, such as a 5G NR communication network.

[0022] The features and technical advantages of examples according to the present disclosure have been outlined quite extensively above so that the following specific embodiments may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be easily used as a basis for modifying or designing other structures for achieving the same purpose of the present disclosure. Such equivalent constructions do not depart from the scope of protection of the appended claims. The characteristics of the concepts disclosed herein, both in terms of their organization and method of operation, and the associated advantages will be better understood by considering the following description in conjunction with the accompanying drawings. Each of the accompanying drawings is provided for the purpose of illustration and description, and not as a definition of the limitations of the claims.

[0023] Although various aspects and specific implementations are described in this application by the illustration of some examples, it will be understood by those skilled in the art that additional specific implementations and use cases may be generated in many other arrangements and scenarios. The innovation described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects and / or uses can be generated via integrated chip specific implementations and other devices based on non-module components (e.g., end-user devices, vehicles, communication equipment, computing equipment, industrial equipment, retail / purchase equipment, medical equipment, devices that enable artificial intelligence (AI), etc.). Although some examples may or may not specifically point to use cases or applications, the applicability of a wide range of described innovations may occur. The scope of specific implementations may range from chip-level or module components to non-module, non-chip-level specific implementations, and further to aggregated, distributed or original equipment manufacturer (OEM) devices or systems in conjunction with one or more aspects of the described innovations. In some practical environments, the equipment in conjunction with the various aspects and features described may also necessarily include additional components and features for implementing and practicing the claimed and described aspects. For example, the transmission and reception of wireless signals necessarily include multiple components for analog and digital purposes (e.g., hardware components including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.). The innovations described herein are intended to be practiced in a variety of devices, chip-level components, systems, distributed arrangements, end-user devices, etc., having different sizes, shapes, and configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] A further understanding of the nature and advantages of the present disclosure may be achieved by reference to the following drawings. In the drawings, similar components or features may have the same reference label. In addition, various components of the same type may be distinguished by adding a dash and a second label to distinguish between similar components after the reference label. If only the first reference label is used in the specification, the description applies to any of the similar components having the same first reference label, regardless of the second reference label.

[0025] Figure 1 A block diagram of an example device 100 for performing image capture from one or more image sensors is shown.

[0026] Figure 2 is a block diagram illustrating the operation of a variable aperture camera, such as in an image capture device.

[0027] Figure 3 is a timing diagram illustrating acquisition of image data through aperture size transitions and processing of image frames acquired during the transitions to obtain similar brightness according to some embodiments of the present invention.

[0028] Figure 4 is a flow chart illustrating an example method for adjusting the brightness of an image frame determined from a variable aperture (VA) camera according to some embodiments of the present disclosure.

[0029] Figure 5 is a block diagram illustrating brightness adjustment in a variable aperture (VA) camera according to some embodiments of the present disclosure.

[0030] Figure 6 is a block diagram illustrating image brightness adjustment during rolling shutter correction (RSC) according to some embodiments of the present disclosure.

[0031] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0032] The "Detailed Description" set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to limit the scope of the present disclosure. Rather, the "Detailed Description" includes specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0033] The present disclosure provides systems, apparatus, methods, and computer-readable media that support image processing to improve the performance of image capture devices, and in particular, in image capture devices having variable aperture (VA) cameras. During an aperture size transition of a VA camera, image data acquired at a potentially unknown and / or varying aperture may be image processed to match one or more characteristics of the image data captured during the aperture size transition with image data captured in a stable, steady-state state prior to the aperture size transition.

[0034] Specific implementations of the subject matter described in the present disclosure may be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, the present disclosure provides techniques for improving the appearance of a sequence of images captured during an aperture size transition by matching characteristics of the image data captured during the aperture size transition with image data prior to the aperture size transition. For example, the brightness of a frame captured during an aperture size transition may be matched to the brightness of a frame captured prior to the aperture size transition to reduce sudden changes that may occur in an image data stream received during an aperture size transition. This may be particularly useful when the aperture size transition is faster than other camera control algorithms, such as automatic exposure (AE), adjust to aperture size changes. The aperture size transition may occur in a continuous manner so that the AE algorithm does not have time to stabilize at a value that obtains the desired brightness value of the image data, but the image processing described in accordance with the embodiments herein may respond during the aperture size transition by basing the modification of the image data on the previous image data.

[0035] An example device for capturing image frames using one or more image sensors, such as a smart phone, may include a configuration of two, three, four, or more cameras on the back side (e.g., the side opposite the user display) or the front side (e.g., the same side as the user display) of the device. Devices with multiple image sensors include one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., AI engines), or other suitable circuits for processing images captured by the image sensors. The one or more image signal processors may provide processed image frames to a memory and / or processor (such as an application processor, an image front end (IFE), an image processing engine (IPE), or other suitable processing circuits) for further processing, such as for encoding, storage, transmission, or other manipulation.

[0036] As used herein, an image sensor may refer to the image sensor itself and any specific other components coupled to the image sensor for generating image frames for processing by an image signal processor or other logic circuitry or for storage in a memory (whether a short-term buffer or a long-term non-volatile memory). For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which is provided to a digital circuit coupled to the image sensor.

[0037] In the following description, a large number of specific details (such as examples of specific components, circuits and processes) are set forth in order to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth in order to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that these specific details may not be required to implement the teachings disclosed herein. In other examples, in order to avoid obscuring the teachings of the present disclosure, known circuits and devices are shown in block diagram form.

[0038] Some portions of the following detailed description are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, programs, logic blocks, processes, etc. are conceived as a self-consistent sequence of steps or instructions leading to a desired result. These steps are steps requiring physical manipulations of physical quantities. Typically, although not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0039] In the accompanying drawings, a single box may be described as performing one or more functions. The one or more functions performed by the box may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, the following generally describes various exemplary components, blocks, modules, circuits, and steps in terms of their functionality. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints proposed for the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. Moreover, the example device may include components other than those shown, including known components such as processors, memories, etc.

[0040] Aspects of the present disclosure are applicable to any electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). In addition, aspects of the present disclosure can be implemented in image sensors or devices coupled to image sensors having the same or different capabilities and characteristics (e.g., resolution, shutter speed, sensor type, etc.). In addition, aspects of the present disclosure can be implemented in a device for processing image frames, whether or not the device includes or is coupled to an image sensor, such as a processing device that can retrieve stored images for processing, including a processing device present in a cloud computing system.

[0041] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "set," "generate," etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission, or display devices.

[0042] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some portions of the present disclosure. Although the description and examples below use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0043] Figure 1A block diagram of an example device 100 for performing image capture from one or more image sensors is shown. The device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some implementations, the device 100 also includes or is coupled to a processor 104 and a memory 106 storing instructions 108. The device 100 may also include or be coupled to a display 114 and an input / output (I / O) component 116. The I / O component 116 may be used to interact with a user, such as a touch screen interface and / or physical buttons. The I / O component 116 may also include a network interface for communicating with other devices including a wide area network (WAN) adapter 152, a local area network (LAN) adapter 153, and / or a personal area network (PAN) adapter 154. An example WAN adapter is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 153 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 154 is a Bluetooth wireless network adapter. Each of adapters 152, 153, and / or 154 may be coupled to an antenna, including multiple antennas configured for primary and diversity reception and / or configured to receive a particular frequency band. Device 100 may also include or be coupled to a power source 118 for device 100, such as a battery or a component that couples device 100 to an energy source. Device 100 may also include or be coupled to Figure 1 Additional features or components not shown in the figure. In one example, a wireless interface that may include multiple transceivers and a baseband processor may be coupled to or included in the WAN adapter 152 for wireless communication devices. In another example, an analog front end (AFE) for converting analog image frame data into digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

[0044] The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data about the movement of the device 100, data about the environment surrounding the device 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which may also be used to determine velocity and distance traveled by appropriately integrating the measured acceleration, and one or more of the acceleration, velocity, and / or distance may be included in the generated motion data. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub or directly to the image signal processor 112. In another example, the non-camera sensor may be a global positioning system (GPS) receiver.

[0045] An image signal processor (ISP) 112 may receive image data, such as for forming an image frame. In one embodiment, a local bus connection couples the image signal processor 112 to the image sensors 101 and 102 of the first camera 103 and the second camera 105, respectively. In another embodiment, a wire interface couples the image signal processor 112 to an external image sensor. In yet another embodiment, a wireless interface couples the image signal processor 112 to the image sensors 101, 102.

[0046] The first camera may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may be controlled by an associated auto focus (AF) algorithm 133 executed in the ISP 112, which adjusts the lenses 131 and 132 to focus on a specific focal plane at a certain scene depth from the image sensors 101 and 102. The AF algorithm 133 may be assisted by a depth sensor 140.

[0047] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. The lenses 131 and 132 focus light at the image sensors 101 and 102, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, one or more color filter arrays (CFA) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, and / or other suitable components for imaging. The first lens 131 and the second lens 132 can have different fields of view to capture different representations of the scene. For example, the first lens 131 can be an ultra-wide (UW) lens, and the second lens 132 can be a wide (W) lens. Multiple image sensors can include a combination of ultra-wide (high field of view (FOV)) sensors, wide sensors, long-range sensors, and ultra-long-range (low FOV) sensors. That is, each image sensor can be configured by hardware configuration and / or software settings to obtain different but overlapping fields of view. In one configuration, the image sensor is configured with different lenses with different magnifications, which results in different fields of view. The sensors may be configured such that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV may capture a field of view in the range of 64 to 84 degrees, a sensor configured for an ultra-lateral FOV may capture a field of view in the range of 100 to 140 degrees, a sensor configured for a long-range FOV may capture a field of view in the range of 10 to 30 degrees, and a sensor configured for an ultra-long-range FOV may capture a field of view in the range of 1 to 8 degrees.

[0048] The image signal processor 112 processes the image frames captured by the image sensors 101 and 102. Figure 1 Device 100 is illustrated as including two image sensors 101 and 102 coupled to image signal processor 112, but any number (e.g., one, two, three, four, five, six, etc.) of image sensors may be coupled to image signal processor 112. In some aspects, a depth sensor such as depth sensor 140 may be coupled to image signal processor 112, and the output from the depth sensor may be processed in a manner similar to that of image sensors 101 and 102. Furthermore, any number of additional image sensors or image signal processors may be present for device 100.

[0049] In some embodiments, the image signal processor 112 may execute instructions from a memory, such as instructions 108 from the memory 106, instructions stored in a separate memory coupled to or included in the image signal processor 112, or instructions provided by the processor 104. In addition or in an alternative form, the image signal processor 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in the present disclosure. For example, the image signal processor 112 may include one or more image front ends (IFEs) 135, one or more image post-processing engines (IPEs) 136, and one or more automatic exposure compensation (AEC) 134 engines. The AF 133, AEC 134, IFE 135, IPE 136 may each include a dedicated circuit that is embodied as software code executed by the ISP 112 and / or a combination of hardware within the ISP 112 and software code executed on the ISP 112.

[0050] In some implementations, the memory 106 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more operations described in the present disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 for generating images or videos. The instructions 108 may also include other applications or programs executed by the device 100, such as an operating system and specific applications other than for image or video generation. A camera application such as that executed by the processor 104 may enable the device 100 to generate images using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames, or may be accessed by the processor 104 to obtain processed frames. In some embodiments, the device 100 does not include the memory 106. For example, the device 100 may be a circuit including the image signal processor 112, and the memory may be external to the device 100. The device 100 may be coupled to an external memory and configured to access the memory to write output frames for display or long-term storage. In some embodiments, device 100 is a system on a chip (SoC) that combines image signal processor 112, processor 104, sensor hub 150, memory 106, and input / output components 116 into a single package.

[0051] In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform various operations described herein, including noise reduction operations. For example, execution of the instructions may instruct the image signal processor 112 to start or end capturing an image frame or a sequence of image frames, wherein the capture includes noise reduction as described in the embodiments herein. In some embodiments, the processor 104 may include one or more general-purpose processor cores 104A capable of executing scripts or instructions of one or more software programs (such as instructions 108 stored in the memory 106). For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 106.

[0052] When executing the camera application, the processor 104 can be configured to command the image signal processor 112 to perform one or more operations with reference to the image sensor 101 or 102. For example, the camera application can receive a command to start a video preview display, and upon receiving the command, capture and process a video including a sequence of image frames from one or more image sensors 101 or 102. Image correction, such as using a cascaded IPE, can be applied to one or more image frames in the sequence. Execution of instructions 108 by the processor 104 outside the camera application can also cause the device 100 to perform any number of functions or operations. In some embodiments, in addition to the ability to execute software to cause the device 100 to perform multiple functions or operations (such as the operations described herein), the processor 104 can also include an IC or other hardware (e.g., an artificial intelligence (AI) engine 124). In some other embodiments, the device 100 does not include the processor 104, such as when all the described functionality is configured in the image signal processor 112.

[0053] In some embodiments, display 114 may include one or more suitable displays or screens that allow user interaction and / or present items to the user (such as previews of image frames captured by image sensors 101 and 102). In some embodiments, display 114 is a touch-sensitive display. I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from a user and provide output to the user through display 114. For example, I / O component 116 may include (but is not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (e.g., a power button), a slider, a switch, etc.

[0054] Although shown coupled to each other via processor 104, components such as processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. Although image signal processor 112 is illustrated as being separate from processor 104, image signal processor 112 may be a core of processor 104, which is an application processor unit (APU), included in a system on chip (SoC), or otherwise included in processor 104. Although reference is made to device 100 in the examples herein to perform aspects of the present disclosure, some device components may not be present. Figure 1 100. In some embodiments, the present invention is not limited to the configuration of a particular device or component, including device 100.

[0055] In some embodiments of the image capture device, the first camera and / or the second camera may support variable aperture (VA) operation, allowing the aperture to be changed during capture of image data. Figure 2 1 is a block diagram illustrating the operation of a variable aperture camera, such as in an image capture device. A first camera 103 may include a first lens 131, a first image sensor 101, and a variable aperture (VA) 107. The VA 107 may be, for example, a shutter or a plurality of blades that adjust to vary the radius of an opening through the centers of the plurality of blades. The VA 107 may be configured with a first size 212, with a larger opening indicated by a smaller "f-stop" value. The VA 107 may be configured with a second size 214, with a smaller opening indicated by a larger "f-stop" value. The VA 107 may have multiple configurations corresponding to different f-stop values, or the VA 107 may have an infinitely adjustable aperture size to allow the aperture size to be selected between fixed aperture sizes.

[0056] Since different light reaches the image sensor through the aperture, the brightness in the image data can change with the changing aperture. During the transition of the VA camera, the brightness change may be more significant because the automatic exposure (AE) system and other image processing in the camera pipeline may not have time to stabilize with the change in aperture. Brightness correction can be applied to image frames (such as image frames captured during the VA transition) to obtain a series of image frames with substantially similar brightness that is not obvious to the observer. Brightness adjustment can be applied to image frames based on image data acquired during the aperture size transition and based on image data acquired at the original aperture size by using the original aperture image data as a reference point. In some embodiments, the image processing described herein can be performed separately from the capture of image data, such as when various aspects of the image processing described herein are performed after the image data is acquired. That is, it is possible to perform brightness adjustment outside the camera pipeline, where the camera pipeline (including IFE 135 and IPE 136) stores data in memory 106, retrieves the data from memory 106 later for brightness adjustment, and the adjusted image data is stored back in memory 106.

[0057] Different aperture sizes 212 and 214 may produce different amounts of light reaching image sensor 101, and therefore produce different brightness between image data captured at sizes 212 and 214. An automatic exposure (AE) algorithm may adjust the gain on image sensor 101 to obtain similar brightness between image data of sizes 212 and 214. However, the AE algorithm requires time to stabilize and reach the desired brightness. During this time, the brightness may vary significantly. When the aperture size changes, the AE may not have enough time to stabilize because the aperture size is constantly changing.

[0058] Figure 32 is a timing diagram illustrating acquisition of image data through aperture size transitions and processing of image frames acquired during the transition to obtain similar brightness according to some embodiments of the present invention. Image data may be collected from a VA camera over a period of time to determine image frames 302A-H. Image frames 302A-B may be determined based on image data acquired at aperture size 212. Image frames 302F-H may be determined based on image data acquired at aperture size 214. Image frames 302C-E may be determined based on image data acquired during an aperture size transition 320 from size 212 to size 214. The gain of an image sensor that detects light passing through the aperture may be adjusted by an AE algorithm from a first gain value 312 obtained at aperture size 212 to a second gain value 314 for the aperture size transition and operation at aperture size 214. As a result of the aperture size and the gain, the brightness of image frames 302A-H varies from one frame to the next during the aperture size transition 320. For example, image frames 302A-B have a brightness 316 that changes during aperture size transition 320 and that stabilizes back to the brightness 316 of image frames 302F-H.

[0059] The change in brightness may be reduced or eliminated by adjusting the image frames in the aperture size transition 320 to have a brightness 316. The resulting image frame stream may not have a significant brightness change during the aperture size transition, which increases image quality and improves the user experience of using a VA camera. For example, the image frame stream may be determined based on the image data by determining three groups of image frames 310A, 310B, and 310C. Image frame 310A may be determined based on first image data acquired at aperture size 212. Image frame 310B may be determined based on second image data acquired during aperture size transition 320 and by adjusting 330 the second image data to have a brightness 316 similar to image frame 310A based on the first image data. Image frame 310C may be determined based on third image data acquired at aperture size 214.

[0060] Image processing methods used with VA cameras are Figure 4 The flow chart is shown in FIG. Figure 4 is a flow chart illustrating an example method for adjusting the brightness of an image frame determined from a variable aperture (VA) camera according to some embodiments of the present disclosure.

[0061] Method 400 includes receiving first image data acquired at a first aperture size at block 402. At block 404, a first set of output image frames is determined based on the first image data received at block 402. Figure 3 In the example of FIG. 4 , image processing of blocks 402 and 404 may determine image frame 310A.

[0062] The method 400 continues by receiving second image data acquired during the aperture size transition at block 406. At block 408, a second set of output image frames is determined based on the second image data to match the brightness of the first aperture size by adjusting the brightness of the second image data. Figure 3 In the example of FIG. 4 , image processing of blocks 406 and 408 may determine image frame 310B.

[0063] In some embodiments, the VA camera may have limited feedback during the aperture size transition, such as in an open loop aperture motor. With an open loop aperture configuration, the AE algorithm is unaware of the current aperture of the VA camera during the aperture size transition, and the image signal processor (ISP) is also unaware of when the aperture size transition is complete. The brightness adjustment amount applied at block 408 during the aperture size transition may be used to determine when the aperture size transition is complete. For example, a determination that the aperture size transition is complete may be based on a brightness adjustment to an output image frame in the second set of output image frames being below a threshold adjustment amount. That is, when the adjustment applied to match the characteristics between the current frame and a previous frame captured prior to the aperture size transition is sufficiently small, the aperture size transition may be considered complete because the aperture size will stabilize at the new aperture size and the camera control algorithm (e.g., AE) has time to stabilize based on the current aperture size.

[0064] In some embodiments, global flicker correction may be applied as part of the adjustment at block 408. For example, the post-gain adjustment of block 408 may be used to compensate for brightness instabilities caused by flickering light sources in the acquired image data.

[0065] In some embodiments, the adjustment of block 408 may include controlling sensor operation in a consistent manner through the aperture size transition and using post-processing gain to adjust the characteristic. For example, when the characteristic is brightness, the image capture device operation may include measuring the brightness before the aperture size transition. During the aperture size transition, the image sensor is controlled with a fixed integration time and gain based on image sensor settings corresponding to the maximum aperture of the initial aperture size and the final aperture size. For each frame of image data captured during the aperture size transition, the brightness of the frame is measured using IFE statistics, and a gain adjustment applied to the image data is determined to match the brightness before the transition.

[0066] Figure 5 An example of modification of the second image data at block 408 is shown. Figure 5is a block diagram illustrating brightness adjustment in a variable aperture (VA) camera according to some embodiments of the present disclosure. First image data 502 may be formed to determine frames 302A-B in image frame 310A. Second image data 504 may be processed together with first image data 502 in processing blocks 512, 514, and 516 to determine image frames 302C, 302D, and 302E in image frame 310B, respectively. In some embodiments, a single processing block performs calculations for each of image frames 302C, 302D, and 302E. Processing blocks 512-516 may adjust second image data 504 based on first image data 502. In some embodiments, the adjustment may include averaging pixel values ​​of first image data 502 with pixel values ​​of second image data 504. In some embodiments, the adjustment may include determining a gain difference between first image data 502 and second image data 504 and modifying pixel values ​​of second image data 504 corresponding to the gain difference. In some embodiments, the adjustment may include determining first image statistics of the first image data and second image statistics of the second image data, and modifying the second image data 504 corresponding to a difference or ratio between the first image statistics and / or the second image statistics. In some embodiments, the adjustment may include multiplying IPE color correction matrix (CCM) coefficients with the second image data to apply post-pipeline gains. This operation may be performed in the IPE or in other blocks, such as other blocks within the ISP, a digital gain block of the ISP, a gamma correction block of the ISP, or in the CPU.

[0067] In another aspect, a brightness matching algorithm may calculate the aperture change and use this information to fine-tune and correct the final aperture. For example, a camera configured with a variable aperture capability may include a voice coil motor (VCM) for controlling the aperture size. VCM operation may result in inaccurate aperture size control, such that an aperture size transition may not end at the desired endpoint aperture size. Brightness measurements of image data captured by the camera may be used to determine the current aperture size at the end of an aperture size transition, and a second aperture size transition may be initiated to compensate for not reaching the target aperture size during the first aperture size transition, wherein the second aperture size transition is based on a difference between the brightness at the end of the first aperture size transition and the expected brightness of the endpoint aperture size.

[0068] The processing performed by processing blocks 512-516 may be performed outside of an image pipeline in an image capture device. Figure 1In an example image capture device of the present invention, an image signal processor 112 executing an IFE 135 and an IPE 136 may form an image pipeline. The IFE 135 may receive image data from one or more of the first camera 103 and / or the second camera 105 and perform processing on the image data, such as color correction, lens correction, rolling shutter correction, brightness correction, other corrections, and / or application of color filters. The output of the IFE 135 may be input to the IPE 136 for further processing, such as encoding into a specific file format such as JPEG or HEIF. The output of the IPE 136 may be input to the memory 106 for storage, display, transmission, and / or later retrieval. The IFE 135 and the IPE 136 form an image pipeline. The image data received from the VA camera may be processed in the image pipeline, output to the memory 106, and later retrieved for further processing to determine the corrected image data, which is then stored in the memory 106.

[0069] In some embodiments, the adjustment of the second image data may be limited by a maximum aperture error. For example, the maximum adjustment amount may be calculated as the subtraction of the squares of the starting and ending aperture sizes. Using an aperture size transition of 2.8 to 2.2, the maximum adjustment amount may be set to 2.8^2 / 2^2, which is approximately 2. A method for adjusting the second image data using the maximum adjustment amount as a limit on the adjustment may include: determining an adjustment limit based on a first aperture size before the aperture size transition; determining a second aperture size after the aperture size transition; and adjusting the brightness of the second image data by the following steps: determining an adjustment to the brightness of the second image data based on the second image data and the first image data; limiting the adjustment based on the adjustment limit; and after limiting the adjustment, modifying the second image data based on the adjustment to adjust the brightness of the second image data.

[0070] In some embodiments, the brightness comparison for adjusting the second image data based on the first image data can use portions of the first image data that meet certain criteria. For example, when determining the amount of adjustment for the second image data, the adjustment can be calculated by comparing only portions of the second image data to corresponding portions of the first image data that are within a certain brightness range (e.g., not saturated because they are not maximum values ​​and not too dark because they are above a threshold brightness level). In some embodiments, the portions can be selected by determining the differences or ratios of different portions and removing portions with outliers. The differences or ratios between the first image data and the second image data in these portions can be used to determine the amount of adjustment for matching the brightness of the second image data to the brightness of the first image data.

[0071] In such an embodiment, the second image data can be adjusted by: determining multiple statistical values ​​for corresponding multiple portions of the first image data; and determining that at least one outlier statistic among the multiple statistical values ​​exceeds a threshold difference from characteristics of the multiple statistical values, wherein adjusting the brightness of the second image data includes: modifying the second image data based on the multiple statistical values ​​in the absence of the at least one outlier statistic.

[0072] In some embodiments, the adjustment of the brightness of the second image data may include adjusting different amounts within the image frame. Figure 6 The image processing of exemplifies image brightness adjustments performed during image frames of a variable aperture. Figure 6 is a block diagram illustrating image brightness adjustment during rolling shutter correction (RSC) according to some embodiments of the present disclosure. Aperture size transitions may occur during reception of image data such that different aperture sizes are used to capture image data in different portions of an image frame. For example, line 600A of image data 600 may be received via aperture 602 having a first aperture size, and line 600F of image data 600 may be received via aperture 604 having a second aperture size. Adjusting the brightness of the second image data may include applying rolling shutter correction (RSC) to a first output image frame of a plurality of output image frames determined by an image capture device, and applying the RSC by: modifying a first portion of the second image data based on a brightness difference between a first portion of the second image data for the first output image frame and the first image data; and modifying a second portion of the second image data based on a brightness difference between a second portion of the second image data for the first output image frame and the first image data.

[0073] In one or more aspects, techniques for supporting image capture and / or image processing may include additional aspects, such as any single aspect or any combination of aspects described below or in conjunction with one or more other processes or devices described elsewhere herein. In a first aspect, supporting image processing may include a device configured to capture an image, such as a mobile device having a camera module configured to have variable aperture capabilities. The device is further configured to receive first image data acquired at a first aperture size from a variable aperture (VA) camera; determine a first set of output image frames based on the first image data; receive second image data acquired during an aperture size transition from the VA camera; and determine a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting characteristics of the second image data based on the first image data. The device may be configured to perform these operations, such as by including hardware and / or software to operate a statistical phase and a capture phase, wherein statistical data of the capture phase is compared with a reference frame analyzed by the statistical phase, and a current frame of the capture phase is adjusted to obtain desired characteristics of the current frame.

[0074] Additionally, the device may be performed or operated according to one or more aspects as described below. In some specific implementations, the device includes a wireless device, such as a UE. In some specific implementations, the device may include at least one processor, and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other specific implementations, the device may include a non-transitory computer-readable medium having a program code recorded thereon, and the program code may be executable by a computer for causing the computer to perform the operations described herein with reference to the device. In some specific implementations, the device may include one or more components configured to perform the operations described herein. In some specific implementations, the method of wireless communication may include one or more operations described herein with reference to the device.

[0075] In a second aspect, in combination with the first aspect, adjusting a characteristic of the second image data includes adjusting the brightness of the second image data based on the brightness of the first image data. The brightness may be adjusted by different amounts across an image frame of the image data, such that the brightness adjustment may adjust the uniformity of the image frame (e.g., by changing the brightness in the center of the image frame compared to the brightness in the corners of the image frame). The brightness may also be adjusted to modify vignetting effects, such as reducing vignetting, and / or adjusting the gradient of lens shading.

[0076] In a third aspect, in combination with one or more of the first aspect or the second aspect, the device may also be configured to process the second image data in an image signal processor (ISP) before adjusting the characteristics of the second image data; and before adjusting the characteristics of the second image data, store the second image data from the ISP in a memory, wherein adjusting the characteristics of the second image data includes: retrieving the second image data from the memory; modifying the second image data to adjust the characteristics of the second image data; and storing the second image data in the memory.

[0077] In a fourth aspect, in combination with one or more of the first to third aspects, the device may also be configured to command an aperture size change before receiving the second image data; receive third image data acquired after the aperture size change; determine a third group of output image frames based on the third image data; and output the first group of output image frames, the second group of output image frames, and the third group of output image frames as an output image stream.

[0078] In a fifth aspect, in combination with one or more of the first to fourth aspects, the apparatus may be further configured to determine that the aperture size transition is complete based on an adjustment to a characteristic of an output image frame in the second set of output image frames being lower than a threshold adjustment.

[0079] In a sixth aspect, in combination with one or more of the first to fifth aspects, the device may also be configured to determine first image statistics of the first image data; and determine second image statistics of the second image data, wherein characteristics of the second image data are adjusted based on the first image statistics and the second image statistics.

[0080] In a seventh aspect, in combination with one or more of the first to sixth aspects, determining first image statistical data of first image data includes: determining multiple statistical values ​​corresponding to multiple parts of the first image data; and determining that at least one outlier statistical value among the multiple statistical values ​​exceeds a threshold difference from characteristics of the multiple statistical values, wherein adjusting characteristics of the second image data includes: modifying the second image data based on the multiple statistical values ​​in the absence of the at least one outlier statistical value.

[0081] In an eighth aspect, in combination with one or more of the first to seventh aspects, the apparatus may be further configured to determine the adjustment limit based on the first aperture size and the second aperture size after the aperture size transition,

[0082] The adjusting characteristic comprises: determining an adjustment to the characteristic of the second image data based on the second image data and the first image data; limiting the adjustment based on the adjustment limit; and after limiting the adjustment, modifying the second image data based on the adjustment to adjust the characteristic of the second image data.

[0083] In a ninth aspect, in combination with one or more of the first to eighth aspects, adjusting the characteristics of the second image data includes applying rolling shutter correction (RSC) to a first output image frame among a plurality of output image frames, and applying the RSC includes: modifying a first portion of the second image data based on a difference in characteristics between a first portion of the second image data for the first output image frame and the first image data; and modifying a second portion of the second image data based on a difference in characteristics between a second portion of the second image data for the first output image frame and the first image data.

[0084] In a tenth aspect, in combination with one or more of the first to ninth aspects, the apparatus may be further configured to output the first set of output image frames and the second set of output image frames in a frame sequence as a preview of the output of the VA camera.

[0085] It will be appreciated by those skilled in the art that any of a variety of different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0086] This article is about Figures 1 to 6 The components, functional blocks and modules include processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software codes, firmware codes, etc., or any combination thereof. Software should be broadly interpreted as meaning instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, and / or functions, etc., whether they are referred to as software, firmware, middleware, microcode, hardware description language, or other terms. In addition, the features discussed herein may be implemented via dedicated processor circuits, via executable instructions, or a combination thereof.

[0087] The technician will further understand that the various illustrative logic boxes, modules, circuits and algorithm steps described in conjunction with the disclosure of this article can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, boxes, modules, circuits and steps have been roughly described in terms of their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. The technician will also easily recognize that the order or combination of components, methods or interactions described herein are only examples, and the components, methods or interactions of various aspects of the present disclosure can be combined or performed in a manner other than the manner illustrated and described herein.

[0088] The various illustrative logical components, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and is exemplified in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0089] The hardware and data processing apparatus for implementing the various exemplary logic components, logic blocks, modules and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller or state machine. In some specific implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In some specific implementations, specific processes and methods may be performed by circuits specific to a given function.

[0090] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents, or any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage media for execution by data processing apparatus or for controlling the operation of data processing apparatus.

[0091] If implemented in software, the function may be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium. The process of the method or algorithm disclosed herein may be implemented in a processor executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, and the communication media include any media that can be implemented to transfer a computer program from one place to another. The storage medium may be any available medium that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. Disks and optical disks as used herein include compact discs (CDs), laser discs, optical disks, digital versatile discs (DVDs), floppy disks, and blue discs, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as a code and instruction set, or any combination of code and instruction sets, on a machine-readable medium or computer-readable medium, which may be incorporated into a computer program product.

[0092] Various modifications to the specific implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific implementations without departing from the spirit or scope of the disclosure. Therefore, the claims are not intended to be limited to the specific implementations shown herein, but are to be consistent with the broadest scope consistent with the disclosure, the principles and novel features disclosed herein.

[0093] Additionally, one of ordinary skill in the art will readily recognize that the terms "upper" and "lower" are sometimes used for ease of describing the drawings and indicate relative positions corresponding to the orientation of the drawings on a correctly oriented page, and may not reflect the correct orientation of any device as implemented.

[0094] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0095] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all illustrated operations to achieve the desired result. In addition, the accompanying drawings can schematically depict one or more example processes in the form of a flow chart. However, other operations that are not depicted can be combined in the example processes schematically illustrated. For example, one or more additional operations can be performed before, after, at the same time, or between any illustrated operations. In some environments, multitasking and parallel processing are advantageous. In addition, the separation of various system components in the specific implementation described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific implementations also fall within the scope of the appended claims. In some cases, the actions narrated in the claims can be performed in different orders and still achieve the desired result.

[0096] As used herein, including in the claims, the term "or" used in a list of two or more items means that any of the listed items may be employed alone, or any combination of two or more of the listed items may be employed. For example, if a composition is described as containing components A, B, or C, the composition may contain A alone; B alone; C alone; A and B combined; A and C combined; B and C combined; or A, B, and C combined. In addition, as used herein, including in the claims, "or" as used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these. The term "substantially" is defined as to a large extent, but not necessarily entirely, what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any disclosed implementation, the term "substantially" may be replaced with "within [percent] of" a specified content, where the percentage includes 0.1%, 1%, 5%, or 10%.

[0097] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Therefore, the disclosure is not intended to be limited to the examples and designs described herein, but to be consistent with the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method comprising: receiving first image data acquired at a first aperture size from a variable aperture (VA) camera; determining a first set of output image frames based on the first image data; receiving second image data acquired during an aperture size transition from the VA camera; as well as A second set of output image frames is determined based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting a characteristic of the second image data based on the first image data. 2 . The method of claim 1 , wherein adjusting the characteristic of the second image data comprises adjusting brightness of the second image data based on brightness of the first image data.

3. The method according to claim 1, further comprising: processing the second image data in an image signal processor (ISP) before adjusting the characteristic of the second image data; as well as storing the second image data in a memory from the ISP before adjusting the characteristic of the second image data, Wherein adjusting the characteristic of the second image data comprises: retrieving the second image data from the memory; modifying the second image data to adjust the characteristic of the second image data; and The second image data is stored in the memory.

4. The method according to claim 1, further comprising: commanding the aperture size transition before receiving the second image data; receiving third image data acquired after the aperture size is changed; determining a third set of output image frames based on the third image data; as well as The first group of output image frames, the second group of output image frames and the third group of output image frames are output as an output image stream.

5. The method according to claim 4, further comprising: The aperture size transition is determined to be complete based on the adjustment to the characteristic of the output image frames in the second set of output image frames being below a threshold adjustment.

6. The method according to claim 1, further comprising: determining first image statistics of the first image data; as well as determining second image statistics of the second image data, Wherein adjusting the characteristic of the second image data is based on the first image statistics and the second image statistics.

7. The method of claim 6, wherein determining the first image statistics of the first image data comprises: determining a plurality of statistical values ​​corresponding to a plurality of portions of the first image data; as well as determining that at least one outlier statistic in the plurality of statistics exceeds a threshold difference from a characteristic of the plurality of statistics, Wherein adjusting the characteristic of the second image data comprises: The second image data is modified based on the plurality of statistical values ​​without the at least one outlier statistic.

8. The method according to claim 1, further comprising: determining an adjustment limit based on the first aperture size and a second aperture size after the aperture size transition, Wherein adjusting the characteristics comprises: determining an adjustment to the characteristic of the second image data based on the second image data and the first image data; limiting the adjusting based on the adjustment limit; and After limiting the adjustment, the second image data is modified based on the adjustment to adjust the characteristic of the second image data.

9. The method of claim 1 , wherein adjusting the characteristic of the second image data comprises applying a rolling shutter correction (RSC) to a first output image frame of the second image data, applying the RSC comprising: modifying the first portion of the second image data based on a difference in characteristics between a first portion of the second image data for the first output image frame and the first image data; as well as The second portion of the second image data for the first output image frame is modified based on a difference in characteristics between the second portion of the second image data and the first image data.

10. The method of claim 1, further comprising outputting the first set of output image frames and the second set of output image frames in a frame sequence as a preview of an output of the VA camera.

11. An apparatus comprising: a memory storing processor readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: receiving first image data acquired at a first aperture size from a variable aperture (VA) camera; determining a first set of output image frames based on the first image data; receiving second image data acquired during an aperture size transition from the VA camera; as well as A second set of output image frames is determined based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting a characteristic of the second image data based on the first image data. 12 . The apparatus of claim 11 , wherein adjusting the characteristic of the second image data comprises adjusting brightness of the second image data based on brightness of the first image data.

13. The apparatus of claim 11, wherein the at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations comprising: processing the second image data in an image signal processor (ISP) before adjusting the characteristic of the second image data; and storing the second image data in a memory from the ISP before adjusting the characteristic of the second image data, Wherein adjusting the characteristic of the second image data comprises: retrieving the second image data from the memory; modifying the second image data to adjust the characteristic of the second image data; as well as The second image data is stored in the memory.

14. The apparatus of claim 11, wherein the at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations comprising: commanding the aperture size transition before receiving the second image data; receiving third image data acquired after the aperture size is changed; determining a third set of output image frames based on the third image data; as well as The first group of output image frames, the second group of output image frames and the third group of output image frames are output as an output image stream.

15. The apparatus of claim 14, wherein the at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations comprising: The aperture size transition is determined to be complete based on the adjustment to the characteristic of the output image frames in the second set of output image frames being below a threshold adjustment.

16. The apparatus of claim 11, wherein the at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations comprising: determining first image statistics for the first image data; and determining second image statistics of the second image data, Wherein adjusting the characteristic of the second image data is based on the first image statistics and the second image statistics.

17. The apparatus of claim 16, wherein determining the first image statistics of the first image data comprises: determining a plurality of statistical values ​​corresponding to a plurality of portions of the first image data; as well as determining that at least one outlier statistic in the plurality of statistics exceeds a threshold difference from a characteristic of the plurality of statistics, Wherein adjusting the characteristic of the second image data comprises: The second image data is modified based on the plurality of statistical values ​​without the at least one outlier statistic.

18. The apparatus of claim 11, wherein the at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations comprising: determining an adjustment limit based on the first aperture size and a second aperture size after the aperture size transition, Wherein adjusting the characteristics comprises: determining an adjustment to the characteristic of the second image data based on the second image data and the first image data; limiting the adjustment based on the adjustment limit; as well as After limiting the adjustment, the second image data is modified based on the adjustment to adjust the characteristic of the second image data.

19. The apparatus of claim 11, wherein adjusting the characteristic of the second image data comprises applying a rolling shutter correction (RSC) to a first output image frame of the second image data, applying the RSC comprising: modifying the first portion of the second image data based on a difference in characteristics between a first portion of the second image data for the first output image frame and the first image data; as well as The second portion of the second image data for the first output image frame is modified based on a difference in characteristics between the second portion of the second image data and the first image data.

20. The apparatus of claim 11, wherein the at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations comprising: Also included is outputting the first set of output image frames and the second set of output image frames in a frame sequence as a preview of the output of the VA camera.

21. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving first image data acquired at a first aperture size from a variable aperture (VA) camera; determining a first set of output image frames based on the first image data; receiving second image data acquired during an aperture size transition from the VA camera; as well as A second set of output image frames is determined based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting a characteristic of the second image data based on the first image data.

22. The non-transitory computer readable medium of claim 21, wherein adjusting the characteristic of the second image data comprises adjusting a brightness of the second image data based on a brightness of the first image data.

23. The non-transitory computer readable medium of claim 21, wherein the operations further comprise one or more of the following: processing the second image data in an image signal processor (ISP) before adjusting the characteristic of the second image data; and storing the second image data in a memory from the ISP before adjusting the characteristic of the second image data, Wherein adjusting the characteristic of the second image data comprises: retrieving the second image data from the memory; modifying the second image data to adjust the characteristic of the second image data; as well as The second image data is stored in the memory.

24. The non-transitory computer readable medium of claim 21, wherein the operations further comprise one or more of the following: commanding the aperture size transition before receiving the second image data; receiving third image data acquired after the aperture size is changed; determining a third set of output image frames based on the third image data; as well as The first group of output image frames, the second group of output image frames and the third group of output image frames are output as an output image stream.

25. The non-transitory computer readable medium of claim 21, wherein the operations further comprise one or more of the following: determining first image statistics for the first image data; and determining second image statistics of the second image data, Wherein adjusting the characteristic of the second image data is based on the first image statistics and the second image statistics.

26. An apparatus comprising: a camera including a variable aperture; a memory storing processor readable code; and at least one processor coupled to the memory and to the camera, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: receiving first image data acquired at a first aperture size from the camera; determining a first set of output image frames based on the first image data; receiving second image data acquired during the aperture size transition from the camera; as well as A second set of output image frames is determined based on the first image data and the second image data, wherein determining the second set of output image frames includes adjusting a characteristic of the second image data based on the first image data.

27. The apparatus of claim 26, wherein adjusting the characteristic of the second image data comprises adjusting brightness of the second image data based on brightness of the first image data.

28. The apparatus of claim 26, wherein the at least one processor comprises an image signal processor, and wherein the at least one processor is configured to perform operations comprising: processing the second image data in the image signal processor (ISP) before adjusting the characteristic of the second image data; and storing the second image data from the ISP in the memory before adjusting the characteristic of the second image data, Wherein adjusting the characteristic of the second image data comprises: retrieving the second image data from the memory; modifying the second image data to adjust the characteristic of the second image data; as well as The second image data is stored in the memory.

29. The apparatus of claim 28, wherein adjusting the characteristic is performed by a second processor of the at least one processor that is different from the image signal processor (ISP).

30. The apparatus of claim 26, wherein the at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations comprising: commanding the aperture size transition before receiving the second image data; receiving third image data acquired after the aperture size is changed; determining a third set of output image frames based on the third image data; as well as The first group of output image frames, the second group of output image frames and the third group of output image frames are output as an output image stream.

Images