Individual exposure control for pixel sensors of image sensors

Abstract

The present disclosure provides a system, method, and apparatus for image signal processing that supports image processing. In a first aspect, a method of image capture includes determining a first exposure setting for a first array of light sensors of an image sensor; determining a second exposure setting for a second array of light sensors of the image sensor of a different type than the first array; capturing first image data under the first exposure setting by the first array of light sensors; and capturing second image data under the second exposure setting by the second array of light sensors. Other aspects and features are also claimed and described.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to image processing, and more particularly to determining settings for capturing images. Some features may enable and provide improved image processing, including improved exposure control of an image sensor used to capture an image. Background Art

[0002] An image capture device is a device that can capture one or more digital images (whether still images for photographs or image sequences for videos). Capture devices can be incorporated into various devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computing devices (such as web cameras, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0003] An image capture device captures a representation of a scene with an image sensor that converts light into electrical signals. The electrical signals may be converted into numerical values ​​representing the appearance of the scene, such as in a table of values ​​representing the scene, where each table entry includes red, green, and blue values ​​that indicate a color in the representation of the scene. Capturing light for conversion to electrical signals is performed by exposing the image sensor to light for a specified period of time, known as an exposure setting. Conventional image sensors operate with a single exposure setting so that all parts of the image sensor are exposed to light for the same duration of time. Summary of the invention

[0004] The following summarizes some aspects of the present disclosure to provide a basic understanding of the technology discussed. This summary is not an exhaustive overview of all the expected features of the present disclosure, and is neither intended to identify the key or important elements of all aspects of the present disclosure, nor to delineate the scope of any or all aspects of the present disclosure. The sole purpose of this summary is to present some concepts of one or more aspects of the present disclosure in a summarized form as a prelude to more specific embodiments presented later.

[0005] In some aspects, the image sensor may include two sensor arrays, the two sensor arrays including a first array and a second array, the first array may be a first type of sensor, and the second array may be a second type of sensor. For example, the image sensor may have a sparse phase detection (PD) configuration, in which one or more phase detection pixels are co-located with photo pixels. The photo pixels (or the first image sensor array) are configured to capture a representation of a scene, and the phase detection pixels (or the second image sensor array) are configured to capture aspects of the scene that can be used, for example, for focus determination, exposure determination, or determination of metadata about the scene. In some embodiments, the phase detection sensors are interleaved within the photo pixel sensors in the image sensor, where the number of phase detection sensors is less than the photo sensors. In some embodiments, the output of the phase detection sensors (or the second image sensor array) is provided to a phase detection autofocus (PDAF) algorithm for controlling the focus of the camera.

[0006] Autofocus is an algorithm involved in capturing photos. Automatic exposure control (AEC) is also involved in capturing photos by determining exposure settings for image sensors. AEC can determine exposure settings based on statistics extracted from captured image data by filtering phase detection pixel data from image data and determining exposure based on photo pixel data. In an embodiment of the present disclosure, AEC can determine exposure settings for image sensors based on photo pixel data and / or phase detection (PD) pixel data. Exposure settings for photo pixels and PD pixels can be determined separately. The camera can be configured to capture data using photo pixels and PD pixels based on different exposure settings. Separate control of exposure settings for different types of sensors (e.g., photo pixel sensors and PD pixel sensors) can provide better exposure settings for PD pixel sensors, which improves autofocus quality when using PDAF. Separate control of exposure settings for different types of sensors can also provide faster brightness convergence of photo pixels, which can allow the camera to better adjust to changing scene conditions.

[0007] The exposure setting for the phase detection (PD) pixels may be determined by the AEC algorithm to properly expose the PD pixels, thereby improving the PDAF quality. The PD pixel data may be excluded when determining the output image frame from the camera, so that the exposure setting for the PD pixels may not have a negative impact on the final image brightness and quality. In some embodiments, separate exposure controls for different image sensor types may be operated continuously. In some embodiments, separate exposure controls for different image sensor types may be triggered based on certain criteria, such as when a scene is determined to have a high dynamic range above a threshold level.

[0008] In some embodiments, the exposure setting may be determined by the AEC to intentionally underexpose the PD pixels. Underexposure may refer to a value that is at least a threshold amount below the saturation level. An overexposed capture may include a sensor exposed to a specific number of maximum intensity, and may include a first exposure range that results in this saturation condition for any scene. A properly exposed capture may be within a second exposure range that is lower than the first range. An underexposed capture may be within a third exposure range that is lower than the second range, where the second range and the third range may be threshold amounts different from the saturation level.

[0009] Underexposing PD pixels may allow the AEC to converge quickly to a new exposure setting for photo pixels after a scene change causes a large change in brightness. An underexposure setting, which may be undesirable for photo pixels because the resulting image has low brightness, may improve the ability to perform exposure determinations when scene conditions change rapidly, resulting in rapid or discontinuous increases in brightness, which would otherwise saturate the sensor. The AEC may be configured between different priorities for exposure settings. For example, one configuration may optimize exposure settings for PD pixels using settings for optimally exposing PD pixels for the PDAF algorithm. Another configuration may optimize exposure settings so that PD pixels are underexposed for use in saturated scenes. These two configurations, as well as potential additional configurations, may switch between a PD priority setting (which may be a default or higher priority configuration) and a PD underexposure setting.

[0010] In one aspect of the present disclosure, a method for image processing includes: determining a first exposure setting for a first array of light sensors of an image sensor, the first array of light sensors including a first type of light sensors; determining a second exposure setting for a second array of light sensors of the image sensor, the second array of light sensors including a second type of light sensors different from the first type; capturing first image data with the first array of light sensors at the first exposure setting; and capturing second image data with the second array of light sensors at the second exposure setting.

[0011] The method may allow a second array of light sensors to be operated in an underexposure setting (the second array of light sensors may be phase detection pixels) to improve operation of the image capture device, thereby allowing a saturation condition of the first array of light sensors (the first array of light sensors may be photo pixels) to be quickly resolved. For example, in some aspects of the present disclosure, the method may also include: determining an underexposure setting for the second array of light sensors; capturing third image data from the second array of light sensors at the second exposure setting as the underexposure setting; and determining a third exposure setting for the first array of light sensors based on the third image data. Determining the exposure for the first array based on the image data captured by the phase detection pixels at the underexposure setting may allow automatic exposure control (AEC) for the first array to resolve from saturation more quickly. In some aspects, this exposure control may continue until the exposure setting for the photo pixels is no longer in a certain condition, such as a saturated state. Thus, according to some aspects, the method may include: determining a new exposure setting for the first array of light sensors that is underexposed, and when predetermined criteria are met during the determination of the new exposure setting: capturing fourth image data from the second array of light sensors; and determining a focus setting for the image sensor based on the fourth image data.

[0012] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: determining a first exposure setting for a first array of light sensors of an image sensor, the first array of light sensors including a first type of light sensors; determining a second exposure setting for a second array of light sensors of the image sensor, the second array of light sensors including a second type of light sensors different from the first type; capturing first image data with the first array of light sensors at the first exposure setting; and capturing second image data with the second array of light sensors at the second exposure setting.

[0013] In an additional aspect of the present disclosure, an apparatus includes: components for determining a first exposure setting for a first array of light sensors of an image sensor, the first array of light sensors including a first type of light sensors; components for determining a second exposure setting for a second array of light sensors of the image sensor, the second array of light sensors including a second type of light sensors different from the first type; components for capturing first image data with the first array of light sensors at the first exposure setting; and components for capturing second image data with the second array of light sensors at the second exposure setting.

[0014] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including: determining a first exposure setting for a first array of light sensors of an image sensor, the first array of light sensors including a first type of light sensors; determining a second exposure setting for a second array of light sensors of the image sensor, the second array of light sensors including a second type of light sensors different from the first type; capturing first image data with the first array of light sensors at the first exposure setting; and capturing second image data with the second array of light sensors at the second exposure setting.

[0015] The image processing methods described herein may be performed by an image capture device and / or on image data captured by one or more image capture devices. Image capture devices (devices that can capture one or more digital images, whether still image photographs or image sequences of video) may be incorporated into various devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), panels or tablet devices, gaming devices, computing devices (such as web cameras, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0016] The image processing techniques described herein may involve a digital camera having an image sensor and a processing circuit (e.g., an application specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), or a central processing unit (CPU)). An image signal processor (ISP) may include one or more of these processing circuits and be configured to perform operations to obtain image data for processing according to the image processing techniques described herein and / or involved in the image processing techniques described herein. The ISP may be configured to control the capture of image frames from one or more image sensors and determine one or more image frames from the one or more image sensors to generate a view of a scene in an output image frame. The output image frame may be part of a sequence of image frames that forms a video sequence. The video sequence may include other image frames received from the image sensor or other image sensors.

[0017] In an example application, an image signal processor (ISP) may receive instructions for capturing a sequence of image frames in response to loading of software (such as a camera application) to generate a preview display from an image capture device. The image signal processor may be configured to generate a single output image frame stream based on image frames received from one or more image sensors. The single output image frame stream may include raw image data from an image sensor, merged image data from an image sensor, or corrected image data processed by one or more algorithms within the image signal processor. For example, the image frame may be processed by an image post-processing engine (IPE) and / or other image processing circuits to process the image frame obtained from the image sensor in the image signal processor (the image frame may have some processing performed on the data before being output to the image signal processor), thereby performing one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, etc. The output image frame from the ISP may be stored in a memory and retrieved by an application processor executing a camera application, which may perform further processing on the output image frame to adjust the appearance of the output image frame and reproduce the output image frame on a display for viewing by a user.

[0018] After the output frame representing the scene is determined by the image signal processor and / or the application processor (such as by the image processing techniques described in various embodiments of the present invention), the output image frame can be displayed on the device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, sent over a network, and / or printed to an output medium. For example, the image signal processor (ISP) can be configured to obtain an input frame of image data (e.g., pixel values) from one or more image sensors, and then generate a corresponding output image frame (e.g., a preview display frame, a still image capture, a frame for video, a frame for object tracking, etc.). In other examples, the image signal processor can output the image frame to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., auto focus (AF), automatic white balance (AWB), and automatic exposure control (AEC)), generate a video file via the output frame, configure the frame for display, configure the frame for storage, send the frame through a network connection, etc. In general, the image signal processor (ISP) can obtain an incoming frame from one or more image sensors, generate an output frame stream, and output the output frame stream to various output destinations.

[0019] In some aspects, the output image frame may be produced by combining aspects of the image correction of the present disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, the first image frame and the second image frame are captured using different exposure times, different apertures, different lenses, and / or other characteristics that may result in improved dynamic range of the fused image when the two image frames are combined. In some aspects, the method may be performed for MFNR photography, wherein the first image frame and the second image frame are captured using the same or different exposure times, and the first image frame and the second image frame are fused to generate a corrected first image frame having reduced noise compared to the captured first image frame.

[0020] In some aspects, a device may include an image signal processor or processor (e.g., an application processor) that includes specific functionality for camera control and / or processing, such as enabling or disabling a merging module or otherwise controlling aspects of image correction. The methods and techniques described herein may be performed entirely by an image signal processor or processor, or various operations may be split between the image signal processor and the processor, and in some aspects split across additional processors.

[0021] The device may include one, two or more image sensors, such as a first image sensor. When there are multiple image sensors, the configuration of these image sensors may be different. For example, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a long-range image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. Any of these or other configurations may be part of a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located in offset positions on the front or back side of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image processing techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0022] In an additional aspect of the present disclosure, a device configured for image processing and / or image capture is disclosed. The device includes components for capturing image frames. The device also includes one or more components for capturing data representing a scene, such as an image sensor (including a charge coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor), and a time-of-flight detector. The device may also include one or more components for gathering and / or focusing light into one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame input to the image processing techniques described herein.

[0023] For those of ordinary skill in the art, other aspects, features, and specific implementations will become apparent when reviewing the following description of specific exemplary aspects in conjunction with the accompanying drawings. Although each feature may be discussed below with respect to certain aspects and accompanying drawings, each aspect may include one or more of the advantageous features discussed herein. In other words, although one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used according to each aspect. In a similar manner, although the exemplary aspects may be discussed below as aspects of a device, system, or method, the exemplary aspects may be implemented in various devices, systems, and methods.

[0024] The method may be embedded in a computer readable medium as a computer program code including instructions for causing a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device including: a first network adapter configured to send data, such as an image or video as recorded data or as streaming data, over a first network connection of a plurality of network connections; and a processor coupled to the first network adapter and the memory. The processor may cause the output image frame described herein to be sent over a wireless communication network, such as a 5G NR communication network.

[0025] The features and technical advantages of examples according to the present disclosure have been outlined quite broadly above so that the following specific embodiments may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be easily used as a basis for modifying or designing other structures for achieving the same purpose of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. When considered in conjunction with the accompanying drawings, the characteristics of the concepts disclosed herein (both their organization and method of operation) and the associated advantages will be better understood from the following description. Each of the figures in the accompanying drawings is provided for the purpose of illustration and description and not as a definition of limitations of the claims.

[0026] Although various aspects and specific implementations are described in this application by way of illustration of some examples, it will be understood by those skilled in the art that additional specific implementations and use cases may be generated in many different arrangements and scenarios. The innovations described herein may be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects and / or uses may be generated via integrated chip specific implementations and other devices based on non-module components (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail / purchase devices, medical devices, devices that enable artificial intelligence (AI), etc.). Although some examples may or may not specifically point to use cases or applications, the applicability of a wide range of described innovations may occur. The scope of specific implementations may range from chip-level or modular components to non-modular, non-chip-level specific implementations, and further to the scope of aggregated, distributed, or original equipment manufacturer (OEM) devices or systems in conjunction with one or more aspects of the described innovations. In some practical environments, the devices in conjunction with the various aspects and features described may also necessarily include additional components and features for implementing and practicing the various aspects claimed and described. For example, the transmission and reception of wireless signals necessarily include multiple components for analog and digital purposes (e.g., hardware components including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.). The innovations described herein are intended to be practiced in a variety of devices, chip-level components, systems, distributed arrangements, end-user devices, etc., having different sizes, shapes, and configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] A further understanding of the nature and advantages of the present disclosure may be achieved by reference to the following drawings. In the drawings, similar components or features may have the same reference label. In addition, various components of the same type may be distinguished by adding a dash and a second label to distinguish between similar components after the reference label. If only the first reference label is used in the specification, the description applies to any of the similar components having the same first reference label, regardless of the second reference label.

[0028] Figure 1 A block diagram of an example device for performing image capture from one or more image sensors is shown.

[0029] Figure 2A is a block diagram illustrating an example data flow path for image data processing in an image capture device according to one or more embodiments of the present disclosure.

[0030] Figure 2Bis a block diagram illustrating an example image sensor having two pixel sensor arrays according to one or more embodiments of the present disclosure.

[0031] Figure 2C is a block diagram illustrating data flow between a processor and an image sensor having two pixel sensor arrays according to one or more embodiments of the present disclosure.

[0032] Figure 3 A flow chart is shown of an example method for processing image data to determine separate exposure settings for different pixel sensor arrays in an image sensor according to one or more embodiments of the present disclosure.

[0033] Figure 4 is a block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure.

[0034] Figure 5 is a block diagram illustrating an example method for changing exposure calculation parameters for different pixel sensor arrays in an image sensor according to one or more embodiments of the present disclosure.

[0035] The same reference numbers and designations in different drawings represent the same elements. DETAILED DESCRIPTION

[0036] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to limit the scope of the present disclosure. Instead, the specific embodiments include specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0037] The present disclosure provides systems, devices, methods, and computer-readable media that support image processing (including techniques for image capture and / or image processing), wherein exposure settings are determined separately for photo pixels and PD pixels. The image capture operation may include: determining a first exposure setting for a first array of light sensors of an image sensor; determining a second exposure setting for a second array of light sensors of the image sensor; capturing first image data with the first array of light sensors at the first exposure setting; and capturing second image data with the second array of light sensors at the second exposure setting.

[0038] Specific implementations of the subject matter described in this disclosure may be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, the present disclosure provides techniques for improving image quality and image capture operations by separately determining exposure settings for photo pixels and PD pixels. Separate control of exposure settings for different types of sensors (e.g., photo pixel sensors and phase detection sensors) may provide better exposure for PD pixels, which improves autofocus quality when using a PDAF sensor. Separate control of exposure settings for different types of sensors may also provide faster brightness convergence for photo pixels, which may allow the camera to better adjust to changing scene conditions.

[0039] Example devices for capturing image frames using one or more image sensors, such as smart phones, may include a configuration of one, two, three, four, or more cameras on the back side (e.g., the side opposite the primary user display) and / or the front side (e.g., the same side as the primary user display) of the device. These devices may include one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., AI engines), or other suitable circuits for processing images captured by the image sensors. The one or more image signal processors (ISPs) may store the output image frames in a memory and / or provide the output image frames to a processing circuit in other ways (such as via a bus). The processing circuit may perform further processing, such as encoding, storing, transmitting, or other manipulations of the output image frames.

[0040] As used herein, an image sensor may refer to the image sensor itself and any specific other components coupled to the image sensor for generating image frames for processing by an image signal processor or other logic circuitry or for storage in a memory (whether a short-term buffer or a long-term non-volatile memory). For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which is provided to a digital circuit coupled to the image sensor.

[0041] In the description of the embodiments herein, a large number of specific details (such as examples of specific components, circuits and processes) are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupling" means directly connected or connected by one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of the present disclosure. However, it will be understood by those skilled in the art that these specific details may not be needed to implement the teachings disclosed herein. In other examples, in order to avoid blurring the teachings of the present disclosure, known circuits and devices are shown in block diagram form.

[0042] Some portions of the following detailed description are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, procedures, logic blocks, processes, etc. are conceived as a self-consistent sequence of steps or instructions leading to a desired result. These steps are steps requiring physical manipulations of physical quantities. Typically, although not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0043] In the accompanying drawings, a single box may be described as performing one or more functions. The one or more functions performed by the box may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, the following generally describes various exemplary components, blocks, modules, circuits, and steps in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Moreover, the example device may include components other than those components shown, including known components such as processors, memories, etc.

[0044] Aspects of the present disclosure are applicable to any electronic device that includes, is coupled to, or otherwise processes data from one, two, or more image sensors capable of capturing image frames (or "frames"). The terms "output image frame" and "corrected image frame" may refer to image frames that have been processed by any of the techniques discussed. In addition, aspects of the present disclosure may be implemented in image sensors or devices coupled to image sensors having the same or different capabilities and characteristics (such as resolution, shutter speed, sensor type, etc.). In addition, aspects of the present disclosure may be implemented in a device for processing image frames, whether or not the device includes or is coupled to an image sensor, such as a processing device that can retrieve stored images for processing, including a processing device present in a cloud computing system.

[0045] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "set," "generate," etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission, or display devices.

[0046] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more parts that can implement at least some portions of the present disclosure. Although the description and examples herein use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0047] Certain components of a device or apparatus described as “means for accessing,” “means for receiving,” “means for sending,” “means for using,” “means for selecting,” “means for determining,” “means for normalizing,” “means for multiplying,” or other similarly named terms referring to one or more operations on data (such as image data) may refer to processing circuits (e.g., application specific integrated circuits (ASICs), digital signal processors (DSPs), graphics processing units (GPUs), central processing units (CPUs)) configured to perform the functions through hardware, software, or a combination of hardware configured by software.

[0048] Figure 1 A block diagram of an example device 100 for performing image capture from one or more image sensors is shown. The device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some implementations, the device 100 also includes or is coupled to a processor 104 and a memory 106 storing instructions 108. The device 100 may also include or be coupled to a display 114 and an input / output (I / O) component 116. The I / O component 116 may be used to interact with a user, such as a touch screen interface and / or physical buttons.

[0049] I / O components 116 may also include a network interface for communicating with other devices including a wide area network (WAN) adapter 152, a local area network (LAN) adapter 153, and / or a personal area network (PAN) adapter 154. An example WAN adapter is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 153 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 154 is a Bluetooth wireless network adapter. Each of adapters 152, 153, and / or 154 may be coupled to an antenna including multiple antennas configured for primary and diversity reception and / or configured for receiving a particular frequency band.

[0050] The device 100 may also include or be coupled to a power source 118 for the device 100, such as a battery or a component that couples the device 100 to an energy source. The device 100 may also include or be coupled to Figure 1 Additional features or components not shown in the figure. In one example, a wireless interface that may include multiple transceivers and a baseband processor may be coupled to or included in the WAN adapter 152 for wireless communication devices. In another example, an analog front end (AFE) for converting analog image frame data to digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

[0051] The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data about the movement of the device 100, data about the environment surrounding the device 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which may also be used to determine velocity and distance traveled by appropriately integrating the measured acceleration, and one or more of the acceleration, velocity, and / or distance may be included in the generated motion data. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub or directly to the image signal processor 112. In another example, the non-camera sensor may be a global positioning system (GPS) receiver.

[0052] The image signal processor 112 may receive image data such as for forming an image frame. In one embodiment, a local bus connection couples the image signal processor 112 to the image sensors 101 and 102 of the first camera 103 and the second camera 105, respectively. In another embodiment, a wire interface couples the image signal processor 112 to an external image sensor. In yet another embodiment, a wireless interface couples the image signal processor 112 to the image sensors 101, 102.

[0053] The first camera 103 may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may be controlled by an associated auto focus (AF) algorithm 133 executed in the ISP 112, which adjusts the lenses 131 and 132 to focus on a specific focal plane at a certain scene depth from the image sensors 101 and 102. The AF algorithm 133 may be assisted by a depth sensor 140.

[0054] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. The lenses 131 and 132 focus light at the image sensors 101 and 102, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, one or more color filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, and / or other suitable components for imaging. The first lens 131 and the second lens 132 may have different fields of view to capture different representations of the scene. For example, the first lens 131 may be an ultra-wide (UW) lens, and the second lens 132 may be a wide (W) lens. The plurality of image sensors may include a combination of ultra-wide (high field of view (FOV)) sensors, wide sensors, long-range sensors, and ultra-long-range (low FOV) sensors.

[0055] That is, each image sensor can be configured by hardware configuration and / or software settings to obtain a different but overlapping field of view. In one configuration, the image sensor is configured with different lenses with different magnifications, which results in different fields of view. The sensors can be configured so that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV can capture a field of view in the range of 64 degrees to 84 degrees, a sensor configured for an ultra-lateral FOV can capture a field of view in the range of 100 degrees to 140 degrees, a sensor configured for a long-range FOV can capture a field of view in the range of 10 degrees to 30 degrees, and a sensor configured for an ultra-long-range FOV can capture a field of view in the range of 1 degree to 8 degrees.

[0056] The camera 103 may have a variable aperture (VA) camera, where the aperture can be controlled to a specific size. Example aperture sizes are f / 2.0, f / 2.8, f / 3.2, f / 8.0, etc. Larger aperture values ​​correspond to smaller aperture sizes, and smaller aperture values ​​correspond to larger aperture sizes. The camera 103 may have different characteristics based on the current aperture size, such as different depths of focus (DOF) at different aperture sizes.

[0057] The image signal processor 112 processes the image frames captured by the image sensors 101 and 102. Figure 1 The device 100 is illustrated as including two image sensors 101 and 102 coupled to an image signal processor 112, but any number (e.g., one, two, three, four, five, six, etc.) of image sensors may be coupled to the image signal processor 112. In some aspects, a depth sensor such as a depth sensor 140 may be coupled to the image signal processor 112, and the output from the depth sensor may be processed in a manner similar to that of the image sensors 101 and 102. Example depth sensors include active sensors including one or more of indirect time of flight (iToF), direct time of flight (dToF), light detection and ranging (lidar), mmWave, radio detection and ranging (radar), and / or hybrid depth sensors (such as structured light). In embodiments without the depth sensor 140, similar information about the depth or depth map of an object can be generated in a passive manner from the parallax between the two image sensors (e.g., using parallax depth measurement or stereo depth measurement), a phase detection autofocus (PDAF) sensor, etc. In addition, there may be any number of additional image sensors or image signal processors for the device 100.

[0058] In some embodiments, the image signal processor 112 may execute instructions from a memory, such as instructions 108 from the memory 106, instructions stored in a separate memory coupled to or included in the image signal processor 112, or instructions provided by the processor 104. In addition or alternatively, the image signal processor 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in the present disclosure. For example, the image signal processor 112 may include one or more image front ends (IFEs) 135, one or more image post-processing engines 136 (IPEs), one or more automatic exposure compensation (AEC) 134 engines, and / or one or more video analysis engines (EVAs). The AF 133, AEC 134, IFE 135, IPE 136, and EVA 137 may each include dedicated circuits, may be embodied as software code executed by the ISP 112, and / or a combination of hardware and software code executed on the ISP 112.

[0059] In some implementations, the memory 106 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more operations described in the present disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 for generating images or videos. The instructions 108 may also include other applications or programs executed by the device 100, such as an operating system and specific applications other than for image or video generation. A camera application such as that executed by the processor 104 may enable the device 100 to generate images using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames or may be accessed by the processor 104 to obtain processed frames. In some embodiments, the device 100 does not include the memory 106. For example, the device 100 may be a circuit including the image signal processor 112, and the memory may be external to the device 100. The device 100 may be coupled to an external memory and configured to access the memory to write output frames for display or long-term storage. In some embodiments, device 100 is a system on a chip (SoC) that combines image signal processor 112 , processor 104 , sensor hub 150 , memory 106 , and input / output components 116 into a single package.

[0060] In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform various operations described herein, including exposure determination operations for different portions of the image sensor. For example, execution of the instructions may instruct the image signal processor 112 to begin or end capturing an image frame or sequence of image frames, wherein the capture includes autofocusing using focus parameters based on image data captured at an exposure level specific to the phase detection pixel sensor, as described in the embodiments herein. In some embodiments, the processor 104 may include one or more general-purpose processor cores 104A capable of executing scripts or instructions of one or more software programs (such as instructions 108 stored in the memory 106). For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 106.

[0061] When executing the camera application, the processor 104 may be configured to instruct the image signal processor 112 to perform one or more operations with reference to the image sensor 101 or 102. For example, the camera application executed on the processor 104 may receive a user command to start a video preview display, and upon receiving the user command, a video including a sequence of image frames is captured and processed from one or more image sensors 101 or 102 by the image signal processor 112. Image processing such as that used to produce an "output" or "corrected" image frame according to the techniques described herein may be applied to one or more image frames in the sequence. Execution of instructions 108 by the processor 104 outside the camera application may also cause the device 100 to perform any number of functions or operations. In some embodiments, the processor 104 may include an IC or other hardware (e.g., an artificial intelligence (AI) engine 124 or other coprocessor) to offload certain tasks from the core 104A. The AI ​​engine 124 may be used to offload tasks related to, for example, facial detection and / or object recognition. In some other embodiments, the device 100 does not include the processor 104, such as when all of the described functionality is configured in the image signal processor 112.

[0062] In some embodiments, display 114 may include one or more suitable displays or screens that allow user interaction and / or present items to the user (such as previews of image frames captured by image sensors 101 and 102). In some embodiments, display 114 is a touch-sensitive display. I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from a user and provide output to the user through display 114. For example, I / O component 116 may include (but is not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (e.g., a power button), a slider, a switch, etc.

[0063] Although shown as being coupled to each other via processor 104, components such as processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled to each other in various other arrangements, such as via one or more local buses, which arrangements are not shown for simplicity. Although image signal processor 112 is illustrated as being separate from processor 104, image signal processor 112 may be a core of processor 104, an application processor unit (APU), included in a system on a chip (SoC), or otherwise included in processor 104. Although reference is made to device 100 in the examples herein to perform aspects of the present disclosure, some device components may not be present. Figure 1 100. In some embodiments, the present invention is not limited to the configuration of a particular device or component, including device 100.

[0064] Figure 1 An exemplary image capture device may be operated to obtain a better exposure setting and / or focus position for a photograph by separately controlling a first exposure setting for a first sensor array of a first type in the image sensor and a second exposure setting for a second sensor array of a second type in the image sensor, thereby obtaining an improved image. Figure 2A One or more example methods of operating one or more cameras, such as camera 103 , are shown in FIG. 1 and described below.

[0065] Figure 2A 2 is a block diagram illustrating an example data flow path for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 of the system 200 may communicate with an image signal processor (ISP) 112 via a bidirectional bus and / or separate control and data lines. The processor 104 may control the camera 103 via a camera control 210, such as for configuring the camera 103 through a driver executed on the processor 104. The camera control 210 may be managed by a camera application 204 executed on the processor 104. The camera application 204 may provide a user-accessible settings menu so that the user may specify individual camera settings or select a profile with corresponding camera settings. The camera application 204 may additionally or alternatively automatically determine the camera settings without user input. The camera control 210 communicates with the camera 103 to configure the camera 103 according to commands received from the camera application 204. The camera application 204 may be, for example, a photography application, a document scanning application, a messaging application, or other applications that process image data acquired from the camera 103.

[0066] The camera configuration may include parameters specifying, for example, a frame rate, an image resolution, a readout duration, an exposure level, an aspect ratio, and / or an aperture size. The camera 103 may obtain image data while being configured with the camera configuration. For example, the processor 104 may execute the camera application 204 to instruct the camera 103, through the camera control 210, to set a first image sensor configuration of the camera 103, obtain first image data from the camera 103 operating with the first camera configuration, instruct the camera 103 to set a second camera configuration of the camera 103, and obtain second image data from the camera 103 operating with the second camera configuration.

[0067] In some embodiments where the camera 103 is a variable aperture (VA) camera system, the processor 104 may execute the camera application 204 to instruct the camera 103 to be configured to a first aperture size, obtain first image data from the camera 103, instruct the camera 103 to be configured to a second aperture size, and obtain second image data from the camera 103. The reconfiguration of the aperture and the obtaining of the first image data and the second image data may occur with little or no change in the scene captured at the first aperture size and at the second aperture size. Example aperture sizes are f / 2.0, f / 2.8, f / 3.2, f / 8.0, etc. Larger aperture values ​​correspond to smaller aperture sizes, and smaller aperture values ​​correspond to larger aperture sizes. That is, f / 2.0 is a larger aperture size than f / 8.0.

[0068] Image data received from camera 103 may be processed in one or more blocks of ISP 112 to form image frames 230 that are stored in memory 106 and / or provided to processor 104. Processor 104 may further process the image data to apply effects to image frames 230. Effects may include Bokeh, lighting, color cast, and / or high dynamic range (HDR) merging. In some embodiments, functionality may be embedded in different components, such as ISP 112, DSP, ASIC, or other custom logic circuitry for performing additional image processing.

[0069] Figure 2B An example image sensor 101 for image capture is shown in more detail in FIG. Figure 2B 1 is a block diagram illustrating an example image sensor having two pixel sensor arrays according to one or more embodiments of the present disclosure. The image sensor 101 may include a first sensor array including photo pixels 201A and a second sensor array including phase detection (PD) pixels 201B. In the illustrated embodiment, the pixels may be organized with a Bayer pattern of color filters, however other color organizations may also be implemented, including monochrome configurations. In addition, Figure 2BPixels of 201B show PD pixels 201B interspersed between photo pixels 201A. The arrangement of the two pixel arrays may have other configurations. For example, PD pixels may be grouped together at one location in the image sensor 101, such as a corner edge or the center.

[0070] Figure 2C An image sensor 101 having two pixel sensor arrays (such as Figure 2B 1 . An example data flow between an image sensor 101 ). Figure 2C 1 is a block diagram illustrating a data flow between a processor and an image sensor having two pixel sensor arrays according to one or more embodiments of the present disclosure. The ISP 112 may perform communications related to the photo pixels 201A separately from the PD pixels 201B. Although the communications are illustrated with different arrows, the communications may be performed on the same channel or bus, such as by sending one or more commands related to the photo pixels 201A on the bus, and then sending one or more commands related to the PD pixels 201B on the same bus in a time-division multiplexed manner. In another example, a single command may be sent to the image sensor 101 to configure the photo pixels 201A and the PD pixels 201B. The photo pixel settings may be configured with an optimal exposure setting to expose the frame in order to improve the image quality for the user. The PD pixel settings may be configured with an optimal exposure setting to expose only the PD pixels. In some embodiments, the use of different exposure settings may be triggered by certain conditions, such as determining that the scene has a high dynamic range (e.g., a dynamic range above a threshold) and / or determining that the image sensor supports only a single exposure mode of operation. When different exposure settings are not triggered, the exposure settings for the two groups of pixels may be the same exposure settings.

[0071] One example configuration for controlling different exposure settings may be based on criteria related to the brightness of image data captured from photo pixels. For example, ISP 112 may determine whether a photo pixel is exposed above a threshold level based on previous photo pixel data, where the threshold level corresponds to a saturation level or is within a threshold amount of a saturation level. When the first array of light sensors is not exposed above the threshold level, ISP 112 may determine a first exposure setting based on the photo pixel data and also determine a second exposure setting based solely on the photo pixel data. The same or different parameters may be applied to the photo pixel data to determine the first exposure setting and the second exposure setting. When the photo pixel is not exposed above the threshold level such that the photo pixel is not close to saturation, ISP 112 may determine a first exposure setting for the photo pixel based on the photo pixel data and determine a second exposure setting for the PD pixel based on the PD pixel data.

[0072] Figure 2AThe camera configuration may include information for configuring exposure settings for photo pixels 201A and PD pixels 201B. This camera configuration may be sent by processor 104, ISP 112, ISP 112 under the control of processor 104, or other logic circuits. In some embodiments, automatic exposure control (AEC) 134 may receive photo pixel data and PD pixel data or information about the corresponding pixel data. AEC 134 may determine separate exposure settings for photo pixels 201A and PD pixels 201B and send the photo pixel settings and PD pixel settings to image sensor 101.

[0073] FIG. 2A to FIG. 2C The system 200 of various embodiments may be configured to perform reference Figure 3 The operations described are to determine the output image frame 230 . Figure 3 A flow chart is shown of an example method for processing image data to determine separate exposure settings for different pixel sensor arrays in an image sensor according to one or more embodiments of the present disclosure. Figure 3 Capturing in the image can result in an improved digital representation of the scene, resulting in a photo or video with higher image quality (IQ).

[0074] At block 302, first image data is received from an image sensor, such as when the image sensor is configured with a camera configuration. The first image data may be received at the ISP 112, processed by an image front end (IFE), a video analytics engine (EVA), and / or an image post-processing engine (IPE) of the ISP 112, and / or other processors 104, and stored in a memory. In some embodiments, capture of image data may be initiated by a camera application executing on the processor 104, which causes the camera control 210 to activate capture of image data by the camera 103 and cause the image data to be provided to a processor, such as the processor 104 or the ISP 112. The first image data may include image data from a first array of light sensors (e.g., photo pixels) and / or a second array of light sensors (e.g., phase detection (PD) pixels).

[0075] Exposure settings may be determined separately for different image sensor arrays. At box 304, a first exposure setting is determined for a first array of light sensors based on the first image data. At box 306, a second exposure setting is determined for a second array of light sensors based on the first image data. The determined exposure settings may be configured on the image sensor. The first exposure setting and the second exposure setting may be determined based in part on an average brightness of a portion of the first image data. For example, the first exposure setting may be based on a first portion of the first image data, which may be data received from a first photo pixel array. Similarly, the second exposure setting may be based on a second portion of the first image data, which may be data received from a second photo pixel array. In some embodiments, the first exposure setting and the second exposure setting may be based on image statistics of the first portion and the second portion of the first image data, respectively, where the statistics may be determined by the EVA of ISP 112.

[0076] The image data may be captured by the image sensor using two or more pixel sensor arrays at two or more different exposure settings. At block 308, second image data may be captured using the first pixel sensor array at a first exposure setting. At block 310, third image data may be captured using the second pixel sensor array at a second exposure setting.

[0077] At block 312, a focus position of the image sensor may be determined. For example, the focus position may be configured by a command that sets a lens position corresponding to a lens of the image sensor. The focus position may be determined by an autofocus (AF) algorithm using information from a phase detection (PD) pixel sensor as part of a phase detection autofocus (PDAF) system. In some embodiments, the phase detection information may be enhanced by other information such as object detection or other ranging techniques.

[0078] By using data captured from a second pixel sensor array (e.g., a PD pixel sensor), the focus position determined at block 312 may be more accurate and / or achieved in a shorter amount of time because the PD pixel sensor may have a different exposure setting than the photo pixel sensor from which the focus information may alternatively be derived. With focus and exposure determined for the scene, a representation of the scene may be captured from the photo pixel sensor for one or more output image frames. At block 314, fourth image data may be captured at the focus position, the fourth image data including data from the photo pixels. At block 316, an output image frame is determined based on the fourth image data including data from the photo pixels but not including the PD pixel data. Image frame 230 may be determined by processor 104 and / or ISP 112 and stored in memory 106. The stored image frame may be read by processor 104 and used to form a preview display on a display of device 100 and / or processed to form a photograph for storage in memory 106 and / or transmission to another device.

[0079] Figure 4 is a block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 and / or other processing circuitry may be configured to operate on image data to perform Figure 3 The image data may be processed to determine one or more output image frames 410. The processor 104 may be a single die including a processor for Figure 4 Logic and memory circuits for each of the modules shown.

[0080] Block 404A is a photo pixel statistics calculator. Calculator 404A may process first image data received from a first pixel array of a first type of image sensor. Statistics may include, for example, average brightness levels of pixels in the first image data.

[0081] Block 404B is a phase detection (PD) pixel statistics calculator. Calculator 404B may process second image data received from a second pixel array of a second type of image sensor. Statistics may include, for example, average brightness levels of pixels in the second image data.

[0082] Although blocks 404A-404B are illustrated as two different calculators, the processor 104 may be configured with a single calculator that calculates statistics for the image data, and the single calculator is separately provided with the first image data and the second image data to determine the first statistics and the second statistics. For example, a video analytics engine (EVA) of the processor 104 may be used to determine the first statistics based on the first image data and determine the second statistics based on the second image data.

[0083] Block 404C is a photo exposure calculator that determines an exposure setting for the first pixel array based on the first image data from the phase detection pixels and / or the second image data from the photo pixels. In some embodiments, the exposure setting may also be determined based on other information, such as detection of a scene change, motion sensor information (such as accelerometer, gyroscope, or magnetometer data indicating rapid movement of the image capture device indicating a scene change), and / or criteria applied to the first image data or the second image data. Example criteria may include a first threshold applied to a characteristic of the first image data and / or a second threshold applied to a characteristic of the second image data.

[0084] Block 404D is a phase detection (PD) exposure calculator that determines an exposure setting for the second pixel array based on the first image data from the phase detection pixels and / or the second image data from the photo pixels. In some embodiments, the exposure setting may also be determined based on other information, such as detection of a scene change, motion sensor information (such as accelerometer, gyroscope, or magnetometer data indicating rapid movement of the image capture device indicating a scene change), and / or criteria applied to the first image data or the second image data. Example criteria may include a first threshold applied to a characteristic of the first image data and / or a second threshold applied to a characteristic of the second image data.

[0085] Block 404E is a camera control module. Camera control 404E may receive photo pixel exposure settings from calculator 404C and phase detection pixel exposure settings from calculator 404D. Camera control 404E may use the exposure settings and / or other data to determine a camera configuration to send to the camera, e.g., to configure separate exposure settings for two different types of pixel sensor arrays in the camera's image sensor.

[0086] Block 404F is a frame processor module. Frame processor 404F may receive image data, such as image data of photo pixels from an image sensor, and determine an output image frame 410 containing a representation of a scene in the camera view. Frame processor 404F may perform operations including, for example, cropping image data by discarding boundary pixels; applying electronic image stabilization (EIS) or digital image stabilization (DIS); applying motion compensation; combining pixels of a second image data as part of a high dynamic range (HDR) merge; and / or applying a lighting effect such as a portrait effect.

[0087] Although blocks 404A to 404F are shown as part of one processor, these blocks may be combined into two or more different processors located on a single or multiple dies. For example, statistics calculators 404A to 404B may be part of IPE 136 in ISP 112, and exposure calculators 404C to 404D may be part of AEC 134 in ISP 112.

[0088] The priority of exposure determination for photo pixels and / or phase detection pixels may be adjusted within photo pixel exposure calculator 404C and / or PD exposure calculator 404D. In some embodiments, the priority may be adjusted based on detected scene changes. Figure 5 404D. is a block diagram illustrating an example method for changing exposure calculation parameters for different pixel sensor arrays in an image sensor according to one or more embodiments of the present disclosure. A scene change occurs at block 502. A scene change may be detected based on determining that a brightness change from one frame to another exceeds a threshold amount, a brightness change between previous pixel data and current data exceeds a threshold amount, accelerometer data indicates motion exceeds a threshold amount, global positioning data indicates motion exceeds a threshold amount, motion vector information from one frame to another exceeds a threshold amount, and / or other predetermined criteria that may be configured on the image capture device. A scene change may cause either or both of the PD pixels or photo pixels of the image sensor to be saturated. Therefore, when a scene change is detected, one or more parameters of the PD exposure setting calculator 404D may be adjusted to prioritize underexposure of the PD pixels, which may speed up exposure calculations and / or focus position determinations.

[0089] After the scene change at box 502, AEC convergence continues at box 504. During AEC convergence, the exposure calculator determines an exposure setting that advances the exposure setting toward an optimal exposure setting. The optimal exposure setting in the first configuration can be determined by a PD underexposure setting 512, where the optimal exposure setting is an underexposure value that provides sufficient scene information while also providing sufficient headroom at the image sensor to reduce the possibility of overexposure as brightness increases rapidly. The AEC convergence at box 504 can determine a criterion of achievement based on parameters associated with the PD underexposure setting 512, which indicates that the exposure setting has converged to the optimal setting (e.g., the cost function result decreases to less than a threshold). When the AEC is stable, an autofocus (AF) trigger at box 506 can be activated. At box 506, an autofocus (AF) algorithm can determine a focus position for capturing photo pixels that form a representation of the scene. During operation of the AF algorithm, the exposure calculator 404D is configured with parameters associated with the PD priority setting 514. PD priority setting 514 configures the AEC algorithm to optimize exposure settings, thereby improving the effectiveness of the AF algorithm. Optimized AEC operation based on setting 514 may result in improved autofocus, but less headroom may be used to prevent oversaturation of PD pixels when the scene changes to a scene with increased brightness.

[0090] In one or more aspects, techniques for supporting image processing may include additional aspects, such as any single aspect or any combination of aspects described below or in conjunction with one or more other processes or devices described elsewhere herein. In a first aspect, supporting image processing may include an apparatus configured to perform operations including: determining a first exposure setting for a first array of light sensors of an image sensor, the first array of light sensors including a first type of light sensors; determining a second exposure setting for a second array of light sensors of the image sensor, the second array of light sensors including a second type of light sensors different from the first type; capturing first image data with the first array of light sensors at the first exposure setting; and capturing second image data with the second array of light sensors at the second exposure setting.

[0091] Additionally, the device may be performed or operated according to one or more aspects as described below. In some specific implementations, the device includes a wireless device, such as a UE. In some specific implementations, the device includes a remote server (such as a cloud-based computing solution) that receives image data for processing to determine an output image frame. In some specific implementations, the device may include at least one processor, and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other specific implementations, the device may include a non-transitory computer-readable medium having a program code recorded thereon, and the program code may be executable by a computer to enable the computer to perform the operations described herein with reference to the device. In some specific implementations, the device may include one or more components configured to perform the operations described herein. In some specific implementations, the method of wireless communication may include one or more operations described herein with reference to the device.

[0092] In a second aspect, in combination with the first aspect, the device is further configured to receive previous image data, wherein: determining the first exposure setting includes determining first image statistics for a first portion of the previous image data corresponding to the first array of light sensors, wherein the first exposure setting is based on the first image statistics; and determining the second exposure setting includes determining second image statistics for a second portion of the previous image data corresponding to the second array of light sensors, wherein the second exposure setting is based on the second image statistics.

[0093] In a third aspect, in combination with one or more of the first aspect or the second aspect, determining the first exposure setting includes determining an exposure setting for photo pixels of the image sensor, and determining the second exposure setting includes determining an exposure setting for phase detection pixels of the image sensor.

[0094] In a fourth aspect, in combination with one or more of the first to third aspects, determining the first exposure setting includes determining an exposure setting for capturing a representation of a scene, and determining the second exposure setting includes determining an exposure setting for capturing autofocus information.

[0095] In a fifth aspect, in combination with one or more of the first to fourth aspects, capturing the second image data with the second array of light sensors at the second exposure setting includes capturing underexposed image data at an underexposed exposure setting, wherein the processor is further configured to perform operations including determining a third exposure setting for the first array of light sensors of the image sensor based on the second image data.

[0096] In a sixth aspect, in combination with one or more of the first to fifth aspects, the processor is further configured to perform operations including: determining a focus position based on the second image data; capturing third image data at the focus position using the first array of light sensors; and determining an output image frame based on the third image data.

[0097] In a seventh aspect, in combination with one or more of the first to sixth aspects, the processor is further configured to perform operations including the following: receiving previous image data, the previous image data including previous photo pixel data from the first array of light sensors of the image sensor and previous phase detection pixel data from the second array of light sensors of the image sensor, wherein: determining whether the first array of light sensors is exposed above a threshold level based on the previous photo pixel data, wherein the threshold level corresponds to a saturation value for the first array of light sensors; when the first array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; and determining the second exposure setting based on the first image data; and when the second array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; and determining the second exposure setting based on the second image data.

[0098] In an eighth aspect, in combination with one or more of the first to seventh aspects, the processor is further configured to perform operations including: determining a scene change for the image sensor before determining the second exposure setting, wherein determining the second exposure setting for the second array of light sensors includes determining the exposure setting based on determining the scene change.

[0099] In a ninth aspect, in combination with one or more of the first to eighth aspects, determining the first exposure setting includes determining an exposure setting for a photo pixel of the image sensor, and determining the second exposure setting includes determining an exposure setting for a phase detection pixel of the image sensor.

[0100] In the tenth aspect, in combination with one or more of the first to ninth aspects, the processor is further configured to perform operations including: determining a focus position based on the second image data; capturing third image data at the focus position with the first array of light sensors; and determining an output image frame based on the third image data.

[0101] In an eleventh aspect, in combination with one or more of the first to tenth aspects, determining the second exposure setting for the second array of light sensors includes determining an underexposure setting for the second array of light sensors, and the processor is further configured to: capture third image data from the second array of light sensors at the second exposure setting; and determine a third exposure setting for the first array of light sensors based on the third image data.

[0102] In a twelfth aspect, in combination with one or more of the first to eleventh aspects, the processor is further configured to: determine a new exposure setting for the first array of light sensors that is underexposed; and when a predetermined criterion is met during the determination of the new exposure setting: capture fourth image data from the second array of light sensors; and determine a focus setting for the image sensor based on the fourth image data.

[0103] It should be understood by those skilled in the art that any of a variety of different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0104] In this article Figures 1 to 5 The components, functional blocks and modules described include processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software codes, firmware codes, etc. or any combination thereof. Software should be broadly interpreted to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, and / or functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language or other terms. In addition, the features discussed herein may be implemented via dedicated processor circuits, via executable instructions, or a combination thereof.

[0105] It should be understood by those skilled in the art that: Figure 4 and Figure 5 One or more blocks (or operations) described herein may be combined with one or more blocks (or operations) described in another figure in the reference figure. Figure 4 One or more boxes (or operations) can be combined with Figures 1 to 3 Combine one or more boxes (or operations) of

[0106] It should also be appreciated by those of ordinary skill in the art that the various illustrative logic boxes, modules, circuits, and algorithmic steps described in conjunction with the disclosure herein can all be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. The technician will also easily recognize that the order or combination of components, methods, or interactions described herein are merely examples, and the components, methods, or interactions of various aspects of the present disclosure can be combined or performed in a manner other than the manner illustrated and described herein.

[0107] The various illustrative logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0108] The hardware and data processing devices for implementing the various exemplary logics, logic blocks, modules and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some specific implementations, the processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In some specific implementations, specific processes and methods may be performed by circuits specific to a given function.

[0109] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents, or any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage media for execution by data processing apparatus or for controlling the operation of data processing apparatus.

[0110] If implemented in software, the function may be stored as one or more instructions or codes on a computer-readable medium or sent through a computer-readable medium. The process of the method or algorithm disclosed herein may be implemented in a processor-executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, which include any media that can be implemented to transfer a computer program from one place to another. The storage medium may be any available medium that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as a code and instruction set, or any combination of code and instruction sets, on a machine-readable medium or computer-readable medium, which may be incorporated into a computer program product.

[0111] Various modifications to the specific implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific implementations without departing from the spirit or scope of the disclosure. Therefore, the claims are not intended to be limited to the specific implementations shown herein, but are to be consistent with the broadest scope consistent with the disclosure, the principles and novel features disclosed herein.

[0112] In addition, one of ordinary skill in the art will readily recognize that, for ease of describing the drawings, opposing terms such as "upper" and "lower" or "front" and "back" or "top" and "bottom" are sometimes used and indicate relative positions corresponding to the orientation of the drawings on a correctly oriented page, and may not reflect the correct orientation of any device as implemented.

[0113] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0114] Similarly, although the operations are depicted in a specific order in the figure, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all the illustrated operations to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations that are not depicted may be combined in the schematically illustrated example process. For example, one or more additional operations may be performed before, after, simultaneously or between any illustrated operations. In some environments, multitasking and parallel processing are advantageous. In addition, the separation of various system components in the specific implementation described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the program components and systems described can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific implementations also fall within the scope of the appended claims. In some cases, the actions described in the claims can be performed in different orders and still achieve the desired result.

[0115] As used herein, including in the claims, the term "or" used in a list of two or more items means that any one of the listed items may be taken alone, or any combination of two or more of the listed items may be taken. For example, if a composition is described as containing components A, B, or C, the composition may contain A alone; B alone; C alone; A and B in combination; A and C in combination; B and C in combination; or A, B, and C in combination. Furthermore, as used herein, including in the claims, "or" as used in a list of items beginning with "at least one" indicates a disjunctive list, so that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these.

[0116] The term "substantially" is defined as being largely, but not necessarily entirely, what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any disclosed implementation, the term "substantially" may be replaced with "within [percentage] of" what is specified, where percentages include 0.1%, 1%, 5%, or 10%.

[0117] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Therefore, the disclosure is not intended to be limited to the examples and designs described herein, but to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method, comprising: determining a first exposure setting for a first array of light sensors of the image sensor, the first array of light sensors comprising photo pixels; determining a second exposure setting for a second array of light sensors of the image sensor, the second array of light sensors comprising phase detection pixels; capturing first image data with the first array of light sensors at the first exposure setting; as well as Second image data is captured with the second array of light sensors at the second exposure setting.

2. The method according to claim 1, further comprising: receiving previous image data, in: determining the first exposure setting comprises determining first image statistics corresponding to a first portion of the previous image data of the first array of light sensors, wherein the first exposure setting is based on the first image statistics; and Determining the second exposure setting includes determining second image statistics corresponding to a second portion of the previous image data of the second array of light sensors, wherein the second exposure setting is based on the second image statistics.

3. The method according to claim 1, wherein: determining the first exposure setting comprises determining an exposure setting for capturing a representation of a scene, and Determining the second exposure setting includes determining an exposure setting for capturing auto-focus information.

4. The method according to claim 3, further comprising: determining a focus position based on the second image data; capturing third image data with the first array of light sensors at the focus position; as well as An output image frame is determined based on the third image data.

5. The method according to claim 1, wherein: determining the second exposure setting for the second array of light sensors comprises determining an underexposure setting for the second array of light sensors, The method further comprises: capturing third image data from the second array of light sensors at the second exposure setting; and A third exposure setting for the first array of light sensors is determined based on the third image data.

6. The method according to claim 5, further comprising: determining a new underexposed exposure setting for said first array of light sensors; When a predetermined criterion is met during said determining of said new exposure setting: capturing fourth image data from said second array of light sensors; as well as A focus setting for the image sensor is determined based on the fourth image data.

7. The method according to claim 1, further comprising: receiving previous image data, the previous image data comprising previous photo pixel data from the first array of light sensors of the image sensor and previous phase detection pixel data from the second array of light sensors of the image sensor, in: determining whether the first array of light sensors is exposed above a threshold level based on the previous photograph pixel data, wherein the threshold level corresponds to a saturation value for the first array of light sensors; When the first array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; and determining the second exposure setting based on the first image data; and When the second array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; and The second exposure setting is determined based on the second image data.

8. The method according to claim 1, further comprising: determining a scene change for the image sensor before determining the second exposure setting, Wherein determining the second exposure setting for the second array of light sensors comprises determining an exposure setting based on determining the scene change.

9. The method of claim 1, wherein the second array of light sensors is fewer in number than the first array of light sensors.

10. The method according to claim 1, further comprising: determining a focus position based on the second image data; capturing third image data with the first array of light sensors at the focus position; as well as An output image frame is determined based on the third image data.

11. A device, comprising: a memory storing processor readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: determining a first exposure setting for a first array of light sensors of the image sensor, the first array of light sensors comprising photo pixels; determining a second exposure setting for a second array of light sensors of the image sensor, the second array of light sensors comprising phase detection pixels; capturing first image data with the first array of light sensors at the first exposure setting; as well as Second image data is captured with the second array of light sensors at the second exposure setting.

12. The apparatus of claim 11, wherein the processor is further configured to perform operations comprising: receiving previous image data, in: determining the first exposure setting includes determining first image statistics corresponding to a first portion of the previous image data of the first array of light sensors, wherein the first exposure setting is based on the first image statistics; and Determining the second exposure setting includes determining second image statistics corresponding to a second portion of the previous image data of the second array of light sensors, wherein the second exposure setting is based on the second image statistics.

13. The device according to claim 11, wherein: determining the first exposure setting comprises determining an exposure setting for capturing a representation of a scene, and Determining the second exposure setting includes determining an exposure setting for capturing auto-focus information.

14. The apparatus of claim 13, wherein capturing the second image data with the second array of light sensors at the second exposure setting comprises capturing underexposed image data at an underexposed exposure setting, and The processor is further configured to perform operations including: A third exposure setting for the first array of light sensors of the image sensor is determined based on the second image data.

15. The device according to claim 11, wherein: determining the second exposure setting for the second array of light sensors includes determining an underexposure setting for the second array of light sensors, and The processor is further configured to perform operations including: capturing third image data from the second array of light sensors at the second exposure setting; and A third exposure setting for the first array of light sensors is determined based on the third image data.

16. The apparatus of claim 15, wherein the processor is further configured to perform operations comprising: determining a new underexposed exposure setting for said first array of light sensors; When a predetermined criterion is met during said determining of said new exposure setting: capturing fourth image data from said second array of light sensors; and A focus setting for the image sensor is determined based on the fourth image data.

17. The apparatus of claim 11, wherein the processor is further configured to perform operations comprising: receiving previous image data, the previous image data comprising previous photo pixel data from the first array of light sensors of the image sensor and previous phase detection pixel data from the second array of light sensors of the image sensor, in: determining whether the first array of light sensors is exposed above a threshold level based on the previous photograph pixel data, wherein the threshold level corresponds to a saturation value for the first array of light sensors; When the first array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; and determining the second exposure setting based on the first image data; and When the second array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; and The second exposure setting is determined based on the second image data.

18. The apparatus of claim 11, wherein the processor is further configured to perform operations comprising: determining a scene change for the image sensor before determining the second exposure setting, Wherein determining the second exposure setting for the second array of light sensors comprises determining an exposure setting based on determining the scene change.

19. The device of claim 11, wherein the second array of light sensors is fewer in number than the first array of light sensors.

20. The apparatus of claim 11, wherein the processor is further configured to perform operations comprising: determining a focus position based on the second image data; capturing third image data with the first array of light sensors at the focus position; and An output image frame is determined based on the third image data.

21. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: determining a first exposure setting for a first array of light sensors of the image sensor, the first array of light sensors comprising photo pixels; determining a second exposure setting for a second array of light sensors of the image sensor, the second array of light sensors comprising phase detection pixels; capturing first image data with the first array of light sensors at the first exposure setting; as well as Second image data is captured with the second array of light sensors at the second exposure setting.

22. The non-transitory computer readable medium of claim 21, wherein the operations further comprise one or more of the following: receiving previous image data, in: determining the first exposure setting includes determining first image statistics corresponding to a first portion of the previous image data of the first array of light sensors, wherein the first exposure setting is based on the first image statistics; and Determining the second exposure setting includes determining second image statistics corresponding to a second portion of the previous image data of the second array of light sensors, wherein the second exposure setting is based on the second image statistics.

23. The non-transitory computer readable medium of claim 22, wherein: determining the first exposure setting comprises determining an exposure setting for capturing a representation of a scene, and Determining the second exposure setting includes determining an exposure setting for capturing auto-focus information.

24. The non-transitory computer readable medium of claim 21, wherein the operations further comprise one or more of the following: receiving previous image data, the previous image data comprising previous photo pixel data from the first array of light sensors of the image sensor and previous phase detection pixel data from the second array of light sensors of the image sensor, in: determining whether the first array of light sensors is exposed above a threshold level based on the previous photograph pixel data; When the first array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; as well as determining the second exposure setting based on the first image data; and When the second array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; and The second exposure setting is determined based on the second image data.

25. The non-transitory computer readable medium of claim 21, wherein the operations further comprise one or more of the following: determining a scene change for the image sensor before determining the second exposure setting, Wherein determining the second exposure setting for the second array of light sensors comprises determining an exposure setting based on determining the scene change.

26. An image capturing device, the image capturing device comprising: an image sensor comprising a first array of light sensors and a second array of light sensors, wherein the first array of light sensors comprises photo pixels and the second array of light sensors comprises phase detection (PD) pixels; a memory storing processor readable code; and at least one processor coupled to the memory and to the image sensor, the at least one processor configured to execute the processor readable code so that the at least one processor: determining a first exposure setting for a first array of light sensors of the image sensor, the first array of light sensors; determining a second exposure setting for a second array of light sensors of the image sensor; capturing first image data with the first array of light sensors at the first exposure setting; as well as Second image data is captured with the second array of light sensors at the second exposure setting.

27. The image capture device of claim 26, wherein the processor is further configured to perform operations comprising: receiving previous image data, in: determining the first exposure setting includes determining first image statistics corresponding to a first portion of the previous image data of the first array of light sensors, wherein the first exposure setting is based on the first image statistics; and Determining the second exposure setting includes determining second image statistics corresponding to a second portion of the previous image data of the second array of light sensors, wherein the second exposure setting is based on the second image statistics.

28. An image capture device according to claim 27, wherein: determining the first exposure setting comprises determining an exposure setting for capturing a representation of a scene, and Determining the second exposure setting includes determining an exposure setting for capturing auto-focus information.

29. The image capture device of claim 26, wherein the processor is further configured to perform operations comprising: receiving previous image data, the previous image data comprising previous photo pixel data from the first array of light sensors of the image sensor and previous phase detection pixel data from the second array of light sensors of the image sensor, in: determining whether the first array of light sensors is exposed above a threshold level based on the previous photograph pixel data; When the first array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; as well as determining the second exposure setting based on the first image data; and When the second array of light sensors is not exposed above the threshold level: determining the first exposure setting based on the first image data; and The second exposure setting is determined based on the second image data.

30. The image capture device of claim 26, wherein the processor is further configured to perform operations comprising: determining a scene change for the image sensor before determining the second exposure setting, Wherein determining the second exposure setting for the second array of light sensors comprises determining an exposure setting based on determining the scene change.

Images