Sensitivity-biased pixel

Abstract

Systems, methods, and non-transitory media for automatic exposure control are provided. An example method may include: receiving pixel values captured by a photoelectric sensor pixel array (PPA) of an image sensor, where one or more of the pixel values correspond to one or more sensitivity-biased photoelectric sensors (SBPs) pixels in the PPA, and one or more saturated pixel values in the pixel values correspond to one or more other photoelectric sensor pixels in the PPA; determining, based on the one or more pixel values, an actual pixel value of the one or more pixel values and / or the one or more saturated pixel values; determining an adjustment factor for at least one of the pixel values based on the actual pixel value and a target exposure value; and correcting, based on the adjustment factor, an exposure setting associated with the image sensor and / or at least one of the pixel values.

Description

Technical Field

[0001] The present disclosure generally relates to automatic exposure convergence and high dynamic range imaging. Background Art

[0002] Camera devices with image sensors are typically integrated into a variety of electronic devices, such as mobile phones, autonomous systems (e.g., autonomous drones, cars, robots, etc.), computers, smart wearable devices, cameras, and many other devices. The camera device allows a user to capture videos and images from various electronic devices. The videos and images can be captured for entertainment purposes, professional photography, surveillance and automation, and other applications. The quality of the video or image may depend on the capabilities of the camera device used to capture the video or image and various factors such as exposure. Exposure is related to the amount of light reaching the image sensor, which is determined by the shutter speed or exposure time, the lens aperture, and the scene brightness.

[0003] The lighting conditions in a scene (such as the brightness and darkness of the scene) may change rapidly. In many cases, the exposure settings of the camera device may not be suitable for different lighting conditions, which may result in overexposed images, underexposed images, degradation in the quality of the captured images, and / or the performance of subsequent image processing algorithms. To limit such problems, the exposure settings of the camera device can be adjusted to account for changes in the lighting conditions. However, the exposure correction mechanism may have a large processing cost, limited adaptability, increased convergence time, and generally requires wasteful and inefficient use of multiple exposures to calculate a new exposure setting for the scene. Summary of the Invention

[0004] Systems, methods, and computer-readable media for automatic exposure control using sensitivity-biased photoelectric sensor (SBP) pixels are disclosed. According to at least one example, a method for automatic exposure control using SBP pixels is provided. The method may include: receiving a plurality of pixel values of an image captured by a photoelectric sensor pixel array of an image sensor, wherein one or more of the plurality of pixel values correspond to one or more sensitivity-biased photoelectric sensor (SBP) pixels in the photoelectric sensor pixel array, and wherein one or more saturated pixel values of the plurality of pixel values correspond to one or more other photoelectric sensor pixels in the photoelectric sensor pixel array; determining an estimated actual pixel value of one or more of the plurality of pixel values and / or one or more saturated pixel values based on one or more of the pixel values corresponding to the one or more SBP pixels; determining an adjustment factor for at least one of the plurality of pixel values based on the estimated actual pixel value and a target exposure value; and correcting an exposure setting associated with the image sensor and / or at least one of the plurality of pixel values based on the adjustment factor.

[0005] According to at least one example, an apparatus for automatic exposure control using SBP pixels is provided. The apparatus may include a memory and one or more processors coupled to the memory, the one or more processors being configured to: receive a plurality of pixel values of an image captured by a photosensor pixel array of an image sensor, wherein one or more of the plurality of pixel values correspond to one or more sensitivity-biased photosensors (SBP) pixels in the photosensor pixel array, and wherein one or more saturated pixel values of the plurality of pixel values correspond to one or more other photosensors pixels in the photosensor pixel array; determine an estimated actual pixel value of one or more of the plurality of pixel values and / or one or more saturated pixel values based on one or more of the pixel values corresponding to the one or more SBP pixels; determine an adjustment factor for at least one of the plurality of pixel values based on the estimated actual pixel value and a target exposure value; and correct an exposure setting associated with the image sensor and / or at least one of the plurality of pixel values based on the adjustment factor.

[0006] According to at least one example, another apparatus for automatic exposure control using SBP pixels is provided. The apparatus may include means for performing the following operations: receiving a plurality of pixel values of an image captured by a photosensor pixel array of an image sensor, wherein one or more of the plurality of pixel values correspond to one or more sensitivity-biased photosensors (SBP) pixels in the photosensor pixel array, and wherein one or more saturated pixel values of the plurality of pixel values correspond to one or more other photosensors pixels in the photosensor pixel array; determining an estimated actual pixel value of one or more of the plurality of pixel values and / or one or more saturated pixel values based on one or more of the pixel values corresponding to the one or more SBP pixels; determining an adjustment factor for at least one of the plurality of pixel values based on the estimated actual pixel value and a target exposure value; and correcting an exposure setting associated with the image sensor and / or at least one of the plurality of pixel values based on the adjustment factor.

[0007] According to at least one example, a non-transitory computer-readable medium for automatic exposure control using SBP pixels is provided. The non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause the one or more processors to: receive a plurality of pixel values of an image captured by a photoelectric sensor pixel array of an image sensor, wherein one or more of the plurality of pixel values correspond to one or more sensitivity-biased photoelectric sensors (SBP) pixels in the photoelectric sensor pixel array, and wherein one or more saturated pixel values among the plurality of pixel values correspond to one or more other photoelectric sensor pixels in the photoelectric sensor pixel array; determine an estimated actual pixel value of one or more of the plurality of pixel values and / or one or more saturated pixel values based on one or more of the pixel values corresponding to the one or more SBP pixels; determine an adjustment factor for at least one of the plurality of pixel values based on the estimated actual pixel value and a target exposure value; and correct an exposure setting associated with the image sensor and / or at least one of the plurality of pixel values based on the adjustment factor.

[0008] In some aspects, the methods, apparatuses, and non-transitory computer-readable media described above may generate an image based on the estimated actual pixel value, one or more saturated pixel values, and / or at least a portion of the plurality of pixel values modified based on the adjustment factor.

[0009] In some aspects, the methods, apparatuses, and non-transitory computer-readable media described above may identify one or more of the pixel values corresponding to one or more SBP pixels. In some examples, one or more of the pixel values may be identified based on a difference between one or more of the pixel values and one or more saturated pixel values, a position of one or more SBP pixels, and / or a position of one or more of the pixel values within an image array including the plurality of pixel values.

[0010] In some examples, determining the estimated actual pixel value may include: multiplying one or more of the pixel values corresponding to one or more SBP pixels by an SBP light sensitivity factor, which is calculated based on a percentage of light that one or more SBP pixels are configured to filter. In some cases, determining the adjustment factor may include dividing the target exposure value by the estimated actual pixel value.

[0011] In some cases, each of the one or more SBP pixels may include a photoelectric sensor pixel having a mask configured to filter a portion of light before the portion of light reaches the photoelectric sensor pixel, a photoelectric sensor pixel having an aperture different from one or more other photoelectric sensor pixels in the photoelectric sensor pixel array, and / or a photoelectric sensor pixel that converts photons into charge at a modified ratio.

[0012] In some cases, one or more SBP pixels are located at one or more boundaries of the photoelectric sensor pixel array, and the one or more boundaries include the bottom row, the top row, the left column, and / or the right column. In some cases, one or more SBP pixels are located in one or more non-boundary regions of the photoelectric sensor pixel array.

[0013] In some examples, one or more saturated pixel values may include over-saturated pixel values, and one or more SBP pixels may have reduced light sensitivity. In some examples, one or more SBP pixels may include SBP photoelectric sensor pixels without light filters, SBP pixels having a larger aperture than other photoelectric sensor pixels in the photoelectric sensor pixel array, and / or SBP photoelectric sensor pixels configured to convert photons into charges at an increased rate; and one or more saturated pixel values may include one or more under-saturated pixel values. In some cases, one or more SBP pixels may include a first group of pixels and a second group of pixels, where the first group of pixels has reduced light sensitivity and the second group of pixels has increased light sensitivity.

[0014] In some aspects, any of the devices described above may include one or more image sensors. In some examples, any of the devices described above may be a camera (e.g., an IP camera), a mobile device (e.g., a mobile phone or a so-called "smartphone", or other mobile devices), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other devices, or may be a part thereof. In some aspects, any of the devices described above may include one or more cameras for capturing one or more images. In some aspects, any of the devices described above may further include a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, any of the devices described above may include one or more sensors.

[0015] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0016] The foregoing and other features and embodiments will become more apparent with reference to the following specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To describe the manner in which the above and other advantages and features of the present disclosure can be obtained, a more specific description of the principles described above will be presented by referring to specific embodiments thereof shown in the accompanying drawings. It should be understood that these drawings depict only example embodiments of the present disclosure and should not be considered as limiting its scope. By using the drawings, the principles herein are described and explained with additional specific content and details, in the drawings:

[0018] Figure 1 is a block diagram showing the architecture of an example image capture and processing system according to some examples of the present disclosure.

[0019] Figure 2A shows a side view of an example photosensor pixel of a photosensor pixel array in an image sensor according to some examples of the present disclosure.

[0020] Figure 2B and Figure 2C shows a side view of an example sensitivity-biased photosensor pixel according to some examples of the present disclosure.

[0021] Figures 3A to 3E is a block diagram showing an example configuration of a photosensor pixel array including a photosensor pixel and a sensitivity-biased photosensor pixel according to some examples of the present disclosure.

[0022] Figure 4 shows various example configurations of a photosensor pixel according to some examples of the present disclosure.

[0023] Figure 5 is a diagram showing a system flow for exposure control using a sensitivity-biased photosensor pixel according to some examples of the present disclosure.

[0024] Figure 6 is a flowchart showing an example method for automatic exposure control using a sensitivity-biased photosensor pixel according to some examples of the present disclosure.

[0025] Figure 7 shows an example computing device architecture according to some examples of the present disclosure. Detailed Description

[0026] Certain aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which will be obvious to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it is obvious that various embodiments can be implemented without these specific details. The drawings and the description are not intended to be restrictive.

[0027] The following description provides only example embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. On the contrary, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes can be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0028] In video and imaging, exposure refers to the amount of light per unit area that reaches an image sensor, which is determined by, for example, shutter speed, lens aperture, gain, and luminance. Within a given exposure, the image sensor provides a certain amount of dynamic range, such as pixel values from 0 to 255, as further described below. Generally, an image sensor has a limited exposure range and a limited dynamic range, which limits the range of pixel values that can be represented. For example, if an image sensor can represent a maximum of 255 (8-bit) pixel values, any scene element with a luminance that exceeds the combination of the exposure and the sensor's dynamic range will result in a pixel value capped at 255 and being recorded as white (overexposed), rather than the accurate color and luminance details or values. Similarly, any scene element with a luminance less than the combination of the exposure and the sensor's dynamic range will result in a minimum pixel value of 0 and being recorded as black (underexposed), rather than the accurate details or values. Thus, in a very bright scene where pixel values may exceed the maximum range of the image sensor, the pixels of that part of the scene can be recorded as white, while in a very dark scene where pixel values may be 0, the pixels of that part of the scene can be recorded as black.

[0029] In addition, when an image has a loss of highlight details, the image captured by the image sensor can be considered oversaturated, in which case the bright part of the image (or the entire image) may be white (e.g., faded) or cropped, and when an image has a loss of shadow details, the image can be considered undersaturated, in which case the dark parts may not be distinguishable from black or be compressed. When one or more parts of an image exceed the maximum pixel value range supported by the image sensor (e.g., when one or more parts of the image are oversaturated / overexposed or undersaturated / underexposed), the image sensor may not be able to determine the actual pixel values of these parts of the image or properly render the scene captured by the image without cropping the pixel values, so the cropped values do not exceed the maximum pixel value range. In the case of less accurate pixel values, the accuracy of the exposure settings calculated for the image sensor and the image quality are reduced.

[0030] In addition, the lighting conditions in the scene captured by a camera device, such as the brightness level and darkness level of the scene, can change rapidly. In many cases, the exposure settings of the camera device may not be suitable for different lighting conditions. Inappropriate exposure settings of the camera device can result in overexposed / over-saturated images, underexposed / under-saturated images, degradation of the quality of the captured images, and / or the performance of subsequent image processing algorithms. As previously mentioned, in many cases, the brightness level of the scene may exceed the dynamic range of the camera device, and the camera device may need to perform multiple exposures to render the scene and for exposure convergence from a dark scene to a bright scene, and vice versa. In some cases, the camera device can perform exposure convergence with high dynamic range. However, the cropping operation can have a negative impact on the metrics used to calculate the exposure settings and thus on the exposure convergence.

[0031] For example, underexposure and overexposure can cause problems with video preview exposure convergence. Auto exposure (AE) involves converging exposure when the scene changes from bright to dark or from dark to bright. However, if the pixel values in the captured image are cropped (e.g., because the scene suddenly changes from dark to bright and the image becomes white or too bright) or compressed (e.g., because when the scene suddenly changes from bright to dark and the image becomes black or too dark), the statistics used to calculate the exposure settings are affected, and the AE may instead need to make a general or informed guess about the appropriate exposure settings by increasing or decreasing the exposure, without knowing the more exact or accurate target. Therefore, for calculating the exposure settings of an image sensor, the image sensor may need more accurate statistics that are not based on pixel values that have been cropped (e.g., too bright) or compressed (e.g., too dark).

[0032] In some examples, the techniques described herein can implement sensitivity-biased photoelectric sensor (SBP) pixels to address these and other challenges. SBP pixels can be implemented in the photoelectric sensor pixel array that the image sensor uses to capture the image data of the scene. SBP pixels can include a photodiode to capture and measure the light in the scene and can be configured to have a reduced or increased light sensitivity, which can allow the SBP pixels to capture higher brightness levels in a dark scene (and / or an underexposed image) and lower brightness levels in a bright scene (and / or an overexposed image). This property of the SBP pixels can prevent the pixel values measured by the SBP pixels with reduced light sensitivity from becoming over-saturated / overexposed in a bright scene and can also prevent the pixel values measured by the SBP pixels with increased light sensitivity from becoming under-saturated / underexposed in a dark scene. As used herein, the term "saturation" can include over-saturation / overexposure and / or under-saturation / underexposure.

[0033] The pixel values calculated from the SBP pixels can also be used to calculate a more accurate brightness level in a scene where other photoelectric sensor pixels cannot capture the actual brightness level for various reasons. For example, when the brightness level in the scene exceeds the maximum saturation / exposure level of the image sensor, when the light sensitivity of other photoelectric sensor pixels is too low to capture a higher brightness level in a dark scene, and / or in other cases, other photoelectric sensor pixels may not be able to capture the actual brightness level.

[0034] The more accurate pixel values calculated from the SBP pixels can be used to calculate a more accurate exposure setting for the image sensor because they can take into account brightness levels that other photoelectric sensor pixels may not consider. A more accurate exposure setting can not only improve image quality but also reduce the exposure convergence time. In addition, the SBP pixels can provide such an exposure setting and reduced exposure convergence time without the need for a cropping operation or multiple exposures, which would reduce efficiency and waste image data that could otherwise be used for other imaging tasks and benefits. Therefore, the pixel values from the SBP pixels can be used to improve the quality and exposure of the generated image, as well as the performance of subsequent image processing algorithms.

[0035] In some examples, to increase or decrease the light sensitivity of the SBP pixels and obtain a more accurate pixel value of the scene, the SBP pixels can implement a mask or filter configured to filter a certain amount of light, a larger or smaller aperture, and / or a modified ratio for converting photons into charge. For example, by applying a mask or filter that blocks a certain amount of light on the SBP pixels, the SBP pixels can be made less sensitive to light, and by not applying a mask or filter that blocks light on the SBP pixels, increasing its aperture, converting photons into charge at an increased ratio, and / or combining multiple photodiodes to create a super diode (as Figure 2C shown), it can be made more sensitive to light.

[0036] In an illustrative example involving a bright scene, one or more SBP pixels can implement a mask or filter to block a certain amount of light. Thus, while other photoelectric sensor pixels in the bright scene may become oversaturated / overexposed due to the light in the scene, the less sensitive SBP pixels can filter a certain amount of light and provide a lower pixel value that is not oversaturated / overexposed. In the illustrative example, to calculate the accurate pixel value of the less sensitive SBP pixels, the pixel value captured by the less sensitive SBP pixels can be multiplied by the amount of light that such SBP pixels are configured to filter. For illustration, if an SBP pixel is configured to capture half of the light and filter the other half of the received light, and the SBP pixel captures a pixel value of 500, then the pixel value 500 can be multiplied by 2 (e.g., based on the SBP pixel being configured to filter half of the light) to estimate the actual pixel value as 1000. The pixel value 1000 can be a more accurate pixel value and can be used to infer the actual pixel value of oversaturated / overexposed pixel values in the image.

[0037] In addition, to determine an adjustment factor (also referred to as an adjustment value) that can be used to determine a more accurate exposure, the target exposure value can be divided by the actual pixel value calculated for the pixel value associated with the SBP pixel (e.g., 1000 in the previous example). For illustration, based on the previous example, if the target value is 50, then the target value 50 can be divided by the actual pixel value 1000 to obtain an adjustment factor of 0.05. Then, the adjustment factor 0.05 can be used to adjust the exposure settings of the image sensor and / or correct the pixel values of oversaturated / overexposed pixels.

[0038] In an illustrative example involving a dark scene, one or more SBP pixels can be implemented without a filter, with a larger aperture, and / or with an increased rate of converting photons to charge in order to capture more light and brightness levels in the dark scene. Thus, while other photoelectric sensor pixels in the dark scene may become undersaturated / underexposed due to the lack or limitation of light in the dark scene, the more sensitive SBP pixels can capture more light and brightness levels and provide a higher pixel value that is not undersaturated / underexposed. The higher pixel value can be used to infer a more accurate pixel value of undersaturated / underexposed pixels. In some cases, as previously explained, the higher pixel value can be adjusted based on the light sensitivity of the SBP pixel (e.g., by reducing the pixel value proportionally to the increase in the light sensitivity of the SBP pixel). In other cases, the higher pixel value can be used without further adjustment. In addition, to determine an adjustment factor for the exposure settings, the target exposure value can be divided by or multiplied by the pixel value of the SBP pixel.

[0039] In some cases, SBP pixels in a photoelectric sensor pixel array can be implemented in one or more boundaries of the photoelectric sensor pixel array. For example, one or more SBPs can be implemented on the top row and / or bottom row of the photoelectric sensor pixel array and / or on the left column and / or right column of the photoelectric sensor pixel array. The portion of the photoelectric sensor pixel array outside the boundary can be referred to as a non-boundary region. Typically, a camera system reads pixel values row by row. Thus, by placing SBP pixels on the (one or more) boundaries of the photoelectric sensor pixel array, when reading pixel values row by row, the camera system can quickly identify the pixel values from the SBP pixels and can place the pixel values from the SBP pixels in a buffer for use as previously described, rather than performing more expensive calculations to detect or index such SBPs (if such SBPs were located in other regions of the photoelectric sensor pixel array).

[0040] The techniques described herein will be described in more detail in the following disclosure. As Figures 1 to 5 shown, the discussion begins with a description of example systems, techniques, and applications for implementing sensitivity-biased photoelectric sensor pixels and performing automatic exposure control using sensitivity-biased photoelectric sensors. Subsequently, an example method for automatic exposure control using sensitivity-biased photoelectric sensor pixels will be described as Figure 6 shown. The discussion ends with a description of an example computing device architecture that includes example hardware components suitable for automatic exposure control using sensitivity-biased photoelectric sensor pixels, as Figure 7 shown. This disclosure now turns to Figure 1 .

[0041] Figure 1 is a block diagram showing the architecture of an example image capture and processing system 100. The image capture and processing system 100 can include various components for capturing and processing images of a scene, such as one or more images of scene 110. The image capture and processing system 100 can capture individual images (or photos) and / or can capture video including multiple images (or video frames) in a particular sequence. The system 100 can include a lens 115 that faces scene 110 and receives light from scene 110. The lens 115 can bend the light towards the image sensor 130. The light received by the lens 115 can pass through an aperture controlled by one or more control mechanisms 120 and is then received by the image sensor 130.

[0042] One or more control mechanisms 120 may control one or more features, mechanisms, components, and / or settings, such as, for example, exposure, focus, zoom, etc. For example, the control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include other control mechanisms, such as control mechanisms for controlling analog gain, flash, high dynamic range (HDR), depth of field, and / or other image capture attributes. The one or more control mechanisms 120 may control features, mechanisms, components, and / or settings based on information from the image sensor 130, information from the image processor 150, and / or other information.

[0043] The focus control mechanism 125B may obtain a focus setting. In some examples, the focus control mechanism 125B may store the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B may adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may move the lens 115 closer to or farther from the image sensor 130 by actuating a motor or servo mechanism (or other lens mechanism) to adjust the focus. In some cases, additional lenses may be included in the system 100, such as one or more microlenses on each photodiode of the image sensor 130. Each microlens may bend the light received from the lens 115 toward the corresponding photodiode before the light reaches the photodiode. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0044] The exposure control mechanism 125A may obtain, determine, and / or adjust one or more exposure settings. In some cases, the exposure control mechanism 125A may store the one or more exposure settings in a memory register. Based on the one or more exposure settings, the exposure control mechanism 125A may control the size of the aperture (e.g., aperture size), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, and / or any other exposure setting. The one or more exposure settings may be referred to as an image capture setting and / or an image processing setting.

[0045] The zoom control mechanism 125C can obtain a zoom setting. In some examples, the zoom control mechanism 125C can store the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servo mechanisms (or other lens mechanisms) to move one or more lenses relative to each other. The zoom setting can be referred to as an image capture setting and / or an image processing setting.

[0046] The image sensor 130 can include one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image generated by the image sensor 130. In some cases, different photodiodes can be covered by different color filters and can thus measure light that matches the color of the filter covering the photodiode. For example, a Bayer color filter includes red filters, blue filters, and green filters, and each pixel of the image is generated based on red light data from at least one photodiode covered by a red filter, blue light data from at least one photodiode covered by a blue filter, and green light data from at least one photodiode covered by a green filter. Other types of color filters can use yellow, magenta, and / or cyan (also known as "emerald") filters to replace or supplement the red, blue, and / or green filters. Some image sensors may have no color filters at all and / or may use different photodiodes (vertically stacked in some cases) throughout the array. In some cases, different photodiodes in the array may have different spectral sensitivity curves and thus respond to different wavelengths of light.

[0047] In some cases, the image sensor 130 can alternatively or additionally include a mask that blocks or prevents a certain amount of light from reaching certain photodiodes or portions of certain photodiodes. In some examples, the mask can be implemented to increase or decrease the sensitivity of the associated photodiodes to light. For example, a more opaque mask can be used to reduce the sensitivity of the photodiodes to light, while a less opaque mask (or no mask) can be used to increase the sensitivity of the photodiodes to light. Additionally, in some examples, the mask can be used for phase detection autofocus (PDAF). Additionally, in some examples, the mask can include filters, layers, films, materials, and / or other elements or properties that can be applied to filter a certain amount of light and prevent the filtered light from reaching the associated photodiodes.

[0048] In some cases, the image sensor 130 may include a gain amplifier to amplify the analog signal output by the photodiode, and / or an analog-to-digital converter (ADC) to convert the analog signal output by the photodiode (and / or amplified by the analog gain amplifier) into a digital signal. In some cases, certain components or functions discussed with respect to one or more of the control mechanisms 120 may alternatively or additionally be included in the image sensor 130. In some examples, the image sensor 130 may include a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), complementary metal-oxide-semiconductor (CMOS), N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), and / or any other combination.

[0049] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) 154, one or more processors 152, and / or one or more of any other type of processor discussed with respect to Figure 7 the computing device 700 described. The main processor 152 may be a digital signal processor (DSP) and / or other type of processor. In some embodiments, the image processor 150 may include or be implemented by an integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes the processor 152 and the image signal processor 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a modem (e.g., 3G, 4G or LTE, 5G, etc.), a memory, connection components (e.g., Bluetooth TM , Global Positioning System (GPS), etc.), and / or other components and / or combinations thereof.

[0050] The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an inter-integrated circuit 2 (I2C) interface, an inter-integrated circuit 3 (I3C) interface, a serial peripheral interface (SPI) interface, a serial general-purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High-Performance Bus (AHB), any combination thereof, and / or other input / output ports. In an illustrative example, the processor 152 may communicate with the image sensor 130 using an I2C port, and the image signal processor 154 may communicate with the image sensor 130 using a MIPI port.

[0051] The image processor 150 may perform multiple tasks such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging of image frames to form an HDR image, image recognition, object recognition, feature recognition, image processing, image enhancement, computer vision, illumination, input reception, management of output, management of memory, HDR processing, and / or any combination thereof. The image processor 150 may store the image frames and / or the processed images in the memory 140 (e.g., random access memory (RAM), read-only memory (ROM), etc.), cache, another storage device, and / or any other memory or storage component.

[0052] One or more input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, keyboard, keypad, touch screen, trackpad, touch-sensitive surface, printer, and / or any other output device, any other input device, communication interface, peripheral device, and / or any combination thereof.

[0053] In some cases, the image capture and processing system 100 may be part of a single device or may be implemented by a single device. In other cases, the image capture and processing system 100 may be part of two or more separate devices or may be implemented by two or more separate devices. In Figure 1 In the example shown, the image capture and processing system 100 includes an image capture device 105A (e.g., a camera device) and an image processing device 105B. In some examples, the image capture device 105A and the image processing device 105B may be part of the same system or device or may be implemented by the same system or device. In other examples, the image capture device 105A and the image processing device 105B may be part of separate systems or devices or may be implemented by separate systems or devices. For example, in some embodiments, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device such as a mobile handheld, laptop computer, or other computing device.

[0054] In some embodiments, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0055] In Figure 1 In the illustrative example shown, the vertical dashed line separates Figure 1The image capture and processing system 100 is divided into two parts, representing the image capture device 105A and the image processing device 105B respectively. The image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an image signal processor 154 and a processor 152), a memory 140, and an I / O 160. In some cases, certain components shown in the image capture device 105A, such as the image signal processor 154 and / or the processor 152, may be included in the image capture device 105A, and vice versa.

[0056] The image capture and processing system 100 may include electronic devices such as mobile or fixed telephones (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, smart wearable devices (e.g., smartwatches, smart glasses, head-mounted displays (HMDs), etc.), and / or any other suitable electronic devices. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 WIFI communication, and / or any other wireless communication.

[0057] Although the image capture and processing system 100 is shown as including certain components, those of ordinary skill in the art will understand that the image capture and processing system 100 may include Figure 1 other components in addition to those shown. The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware and / or may be implemented using the same, the electronic circuits or other electronic hardware may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits); and / or may include computer software, firmware, or any combination thereof and / or may be implemented using the same to perform the various operations described herein. The software and / or firmware may include one or more instructions stored in a computer-readable storage medium and executable by one or more processors of the electronic device implementing the image capture and processing system 100.

[0058] Figure 2AA side view of an example photosensor pixel of a photosensor pixel array in an image sensor (e.g., 130) is shown. The photosensor pixel may include a photodiode / photodetector implemented by the image sensor to capture photons and associated pixel values. In this example, the photosensor pixel 200 includes a microlens 206 located above a color filter 204 (e.g., a Bayer filter or other types of color filters discussed below) and a photodiode 202. Light 210 may pass through the microlens 206 and the color filter 204 before reaching the photodiode 202.

[0059] The color filter 204 may represent a filter for a specific color, such as red, blue, or green, for reflecting a certain amount of that color. Red, green, and blue color filters are commonly used in image sensors and are often referred to as Bayer color filters. A Bayer filter array typically includes more green Bayer filters than red or blue Bayer filters, for example, in a ratio of 50% green, 25% red, and 25% blue, to simulate the physiological sensitivity of the human eye to green light. Some color filter arrays (CFAs) may use an alternating color scheme and may even include more or fewer colors. For example, some CFAs may use cyan, yellow, and magenta color filters instead of the red, green, and blue Bayer color filter scheme. In some cases, color filters for one or more colors in the color-matching scheme may be omitted, leaving only two colors or even one color.

[0060] In addition, in some cases, one or more photosensor pixels or the entire photosensor pixel array may lack a color filter, and thus the color sensitivity of the photosensor pixel or the entire photosensor pixel array is not reduced. For example, in some cases, the photosensor pixel 200 may include the microlens 206 and the photodiode 202 without the color filter 204.

[0061] Figure 2B A side view of an example sensitivity-biased photosensor pixel 215 is shown. In this example, the photosensor pixel 215 includes a microlens 206 and a mask 220 that covers the photodiode 202 to reduce the light sensitivity of the photodiode 202. Light 210 may pass through the microlens 206 and the mask 220 before reaching the photodiode 202. The mask 220 may be configured to filter a certain amount of light 210 to reduce the amount of light 210 reaching the photodiode 202, thereby reducing the light sensitivity of the photodiode 202.

[0062] For example, to prevent pixels from becoming oversaturated in bright scenes, mask 220 can block a certain amount of light in the scene from reaching photodiode 202 or filter a certain amount of light. Thus, mask 220 can reduce the sensitivity of photodiode 202 to light to allow photodiode 202 to better capture the luminance levels in bright scenes. As further described herein, the luminance levels captured by photodiode 202 based on the light passing through mask 220 can be used to calculate pixel values for oversaturated pixels measured by photodiodes in an image sensor (including other photodiodes in the image sensor that may not include a mask for filtering light in the scene), and / or to calculate a normalized value of the pixel values. The pixel values calculated for the image captured by the photodiode array can be used to adjust the exposure level, improve convergence, render scenes having luminance levels beyond the dynamic range of the photodiode array without cropping and / or requiring multiple exposures, improve the performance of subsequent (one or more) image processing algorithms, and the like.

[0063] In some cases, a photoelectric sensor pixel array can implement multiple photoelectric sensor pixels with masks, such as photoelectric sensor pixel 215. In some examples, the photoelectric sensor pixels with masks can be implemented in one or more regions of the photoelectric sensor pixel array. For example, the photoelectric sensor pixels with masks can be dispersed over the photoelectric sensor pixel array. In other examples, the photoelectric sensor pixels with masks can be implemented in one or more boundaries of the photoelectric sensor pixel array.

[0064] For illustration, in some cases, the photoelectric sensor pixels with masks can be implemented on the top and / or bottom boundaries of the photoelectric sensor pixel array and / or on the left and / or right boundaries of the photoelectric sensor pixel array. Camera systems typically read pixels row by row. Thus, in some examples, by placing the photoelectric sensor pixels with masks on the boundaries of the photoelectric sensor pixel array, the camera system can quickly identify the pixels from the photoelectric sensor pixels with masks when reading the pixels, and as further described herein, use them to calculate the pixel values of the image, rather than performing more expensive calculations to detect or index such pixels in the case where the photoelectric sensor pixels with masks are dispersed throughout the image or mixed with other pixels in the photoelectric sensor pixels without masks.

[0065] In different embodiments, the amount of light filtered by mask 220 can vary. For example, in some cases, mask 220 can be configured to block more light (e.g., via increased darkening or using a material with higher light filtering ability) to further reduce the light sensitivity of photodiode 202. This can allow photodiode 202 to better capture brightness levels in brighter scenes. In other cases, mask 220 can be configured to block less light to limit the amount by which the light sensitivity of photodiode 202 is reduced. In still other cases, the photoelectric sensor pixel may not include a mask, so the light sensitivity of photodiode 202 is not reduced.

[0066] In some embodiments, to increase the light sensitivity of photodiode 202 and better capture brightness levels in darker scenes, the aperture of the photoelectric sensor pixel can be increased. For example, referring to Figure 2C , the photoelectric sensor pixel 230 can include multiple photodiodes 202 (or a single larger photodiode) and a larger microlens 232 extending over the multiple photodiodes 202 (or the single larger photodiode). This configuration can allow the photoelectric sensor pixel 230 to capture more light 210 and thus increase the light sensitivity of the photoelectric sensor pixel 230. Due to the higher light sensitivity, the photoelectric sensor pixel 230 can better capture brightness levels in darker scenes.

[0067] The photoelectric sensor pixel 230 can optionally include a mask 220 to filter a certain amount of light. For example, in some cases, the photoelectric sensor pixel 230 can not include a mask to avoid reducing the light sensitivity of the photoelectric sensor pixel 230. In other cases, the photoelectric sensor pixel 230 can include a larger mask covering the multiple photodiodes 202 to block a certain amount of light and reduce the light sensitivity of the photoelectric sensor pixel 230 by a certain amount.

[0068] Although photoelectric sensor pixels 215 and 230 without color filters are shown in Figure 2B and Figure 2C , it should be understood that in some examples, the photoelectric sensor pixels 215 and / or 230 can include color filters as previously described with reference to Figure 2A . Additionally, it should be understood that the photoelectric sensor pixels in the photoelectric sensor pixel array can have other aperture and / or mask configurations different from those shown in Figures 2A to 2C . For example, Figures 3A to 4 shows various example configurations of a photoelectric sensor pixel array that can be implemented to increase and / or decrease the light sensitivity of one or more photoelectric sensor pixels in an image sensor.

[0069] Figure 3AFIG. 0 is a diagram showing an example configuration of a photoelectric sensor pixel array 300, which includes photoelectric sensor pixels 310 (shown in white) and sensitivity-biased photoelectric sensor (SBP) pixels 320 (shown in gray). The photoelectric sensor pixels 310 do not include a mask for filtering light, such as mask 220, but may or may not include a color filter, such as color filter 204. As previously described, the SBP pixels 320 may include a mask for filtering a certain amount of light and thus reducing the light sensitivity of the photodiodes in the SBP pixels 320.

[0070] In some examples, the photoelectric sensor pixel array 300 may include SBP pixels 320 located on the top boundary 302, the bottom boundary 304, the right boundary 306, and / or the left boundary 308 of the photoelectric sensor pixel array 300. The portions outside the top boundary 302, the bottom boundary 304, the right boundary 306, and the left boundary 308 of the photoelectric sensor pixel array may be referred to as the non-boundary regions of the photoelectric sensor pixel array 300. In Figure 3A the example shown, the photoelectric sensor pixel array 300 includes SBP pixels 320 on the top boundary 302 and the right boundary 306 of the photoelectric sensor pixel array 300.

[0071] In some examples, the SBP pixels 320 can be used to determine more accurate pixel values for under-saturated / under-exposed and / or over-saturated / over-exposed images captured by the photodiodes in the photoelectric sensor pixel array. For example, the SBP pixels 320 can measure the under-saturated pixel values in an over-saturated / over-exposed image and can be used to calculate more accurate pixel values that would otherwise exceed the dynamic range of the image sensor. Even if the SBP pixels 320 are also saturated, the SBP pixels 320 can still result in a more accurate prediction. For example, assume that the SBP pixels 320 in this illustrative example are 10 times less sensitive to light than the photoelectric sensor pixels 310, and a scene change causes all the photoelectric sensor pixels to saturate, including both the photoelectric sensor pixels 310 and the SBP pixels 320. The fact that the SBP pixels 320 are 10 times less sensitive to light than the photoelectric sensor pixels 310 and are still saturated means that reducing the exposure of the photoelectric sensor pixels 310 (which do not have a sensitivity bias) by 10 times will not provide a sufficient or large enough adjustment. In this example, the fact that the SBP pixels 320 are 10 times less sensitive to light but still saturated indicates that the exposure of the photoelectric sensor pixels 310 should be reduced by even more than 10 times. Thus, the SBP pixels 320 can be used to determine that the exposure of the photoelectric sensor pixels 310 should be reduced by more than 10 times, and thus will result in a more accurate prediction than would be possible if the photoelectric sensor pixel array 300 did not include the SBP pixels 320, because the photoelectric sensor pixels 310 without the SBP pixels 310 would not provide enough information to infer that the exposure should be reduced by more than 10 times.

[0072] In some cases, the more accurate pixel values can be used to calculate an adjustment factor. In some examples, the adjustment factor can be used to determine and / or implement an adjusted (e.g., more correct / accurate) exposure of the scene captured by the photoelectric sensor pixel array 300. In some examples, the adjustment factor can be used to correct under-saturated / over-saturated, and / or over-saturated / over-exposed pixel values. In some examples, the adjustment factor can be used to adjust the exposure of the scene and can also be used to correct under-saturated / over-saturated, and / or over-saturated / over-exposed pixel values.

[0073] In some examples, when the scene suddenly changes from bright to dark or from dark to bright, the more accurate pixel values and / or the calculated adjustment factor can be used for exposure convergence. For example, when the brightness level of the scene suddenly increases to a level such that the captured pixel values become over-saturated and / or appear white, the adjustment factor can be used to normalize or adjust the exposure and / or the over-saturated pixel values to a specific brightness level, which can allow the image capture and processing system (e.g., system 100) to render the scene and / or reduce the exposure convergence time.

[0074] As previously noted, in different examples, the sensitivity of the SBP pixels 320 can vary to better capture luminance levels in darker and / or brighter scenes. For example, the SBP pixels 320 can be made more sensitive to light by increasing their aperture, configuring the SBP pixels 320 without any masks, and / or increasing the ratio of converting photons to charges. Additionally, in some examples, the SBP pixels 320 can be made less sensitive to light by applying a mask that blocks a certain amount of light on the SBP pixels 320, decreasing their aperture, and / or reducing the ratio of converting photons to charges. In an illustrative example involving a bright scene, the SBP pixels 320 can implement a mask to block a certain amount of light. Thus, while the photoelectric sensor pixels 310 without a mask may become oversaturated / overexposed due to the light in a brighter scene, the less sensitive SBP pixels 320 can filter a certain amount of light and provide a lower pixel value that is not oversaturated / overexposed.

[0075] In some cases, to calculate the accurate pixel value of the less sensitive SBP pixels 320, the pixel value captured by the less sensitive SBP pixels 320 can be multiplied by the amount of filtering provided by such SBP pixels 320. For example, if the SBP pixels 320 are configured to capture half of the light they receive and filter the other half of the light, and the SBP pixels 320 capture a pixel value of 500, then the pixel value 500 can be multiplied by 2 (in this example, this corresponds to the light sensitivity of the SBP pixels being reduced by 1 / 2) to estimate the actual pixel value as 1000. In some examples, the pixel value 1000 can be used to infer the actual pixel values of the pixels captured by other photoelectric sensor pixels 310 in the photoelectric sensor pixel array 300 and / or adjust the captured pixel values. Additionally, the pixel value 1000 can be used to calculate an adjustment factor for adjusting the exposure settings of the image sensor.

[0076] For example, in some cases, to determine the adjustment factor for adjusting the exposure settings, a target exposure value can be selected and divided by the pixel value calculated using the SBP pixels 320 (e.g., 1000 in the previous example). For illustration, in the previous example, if the target exposure value is 50, then the target exposure value 50 can be divided by the actual pixel value 1000 calculated using the SBP pixels 320 to obtain an adjustment factor of 0.05. Then, the adjustment factor 0.05 can be used to adjust the exposure settings and / or correct the pixel values of saturated pixels.

[0077] In addition, by implementing SBP pixels 320 in the boundary regions (e.g., 302, 304, 306, 308) of the photoelectric sensor pixel array 300, the SBP pixels 320 can be detected more efficiently and / or have lower-cost computation. For example, camera systems typically read pixel values measured by the photoelectric sensor pixel array row by row. Thus, in some examples, by placing the SBP pixels 320 on the (one or more) boundary regions of the photoelectric sensor pixel array 300, the camera system can quickly identify the pixel values measured by the SBP pixels 320 in the boundary regions based on the difference in brightness levels between the pixel values from the SBP pixels 320 and other pixel values, the positions of the SBP pixels 320, the rows in the pixel value stream, and / or the differences between the pixel values in the rows corresponding to the boundary regions and the pixel values corresponding to other photoelectric sensor pixels. In this way, the camera system can detect the pixels corresponding to the SBP pixels 320 in the boundary regions without performing more expensive computations that might otherwise be required to detect the pixel values from SBP pixels 320 located in other regions of the photoelectric sensor pixel array 300. In some cases, the camera system can place the pixel values identified as corresponding to the SBP pixels 320 in a buffer or memory for use, as described herein.

[0078] In some examples, the photoelectric sensor pixel array 300 can include one or more SBP pixels 320 in other regions of the photoelectric sensor pixel array 300. For example, the photoelectric sensor pixel array 300 can include one or more SBP pixels 320 dispersed in the internal or non-boundary regions of the photoelectric sensor pixel array 300. In some examples, the SBP pixels 320 on other regions (e.g., internal or non-boundary regions) of the photoelectric sensor pixel array 300 can be used for PDAF. In other examples, such SBP pixels 320 can be used to determine more accurate pixel values, as described herein. In some cases, as previously described, the more accurate pixel values can be used to calculate an adjustment factor to determine and / or implement an adjusted exposure of the scene captured by the photoelectric sensor pixel array 300 and / or correct over-saturated pixel values.

[0079] Figure 3BIt is a block diagram showing another configuration of the photosensor pixel array 340 with SBP pixels 320. In this example, the SBP pixels 320 are configured to span the top boundary 302 of the photosensor pixel array 340, and the remaining area of the photosensor pixel array 340 (including the bottom boundary 304, the right boundary 306, and the left boundary 308) is configured with photosensor pixels 310 that do not have a sensitivity bias (e.g., do not include a mask). In some cases, since pixel values can be read row by row by the camera system, when the camera system reads the row and / or pixel values corresponding to the top boundary 302 (and / or the row associated with the top boundary 302), the pixel values from the SBP pixels 320 spanning the top boundary 302 can be detected. In some examples, the camera system can detect the pixel values of the SBP pixels 320 based on the difference between the pixel values associated with the top boundary 302 (and / or the row associated with the top boundary 302) and the pixel values corresponding to one or more other regions in the photosensor pixel array 340 (and / or other rows in the image).

[0080] Figure 3C It is a block diagram showing another configuration of the photosensor pixel array 350 with SBP pixels 320. In this example, the SBP pixels 320 are configured to span the top boundary 302, the bottom boundary 304, the right boundary 306, and the left boundary 308 of the photosensor pixel array 340. In other words, the SBP pixels 320 are placed on all the boundaries of the photosensor pixel array 350. The remaining area of the photosensor pixel array 350 is configured with photosensor pixels 310 that do not have a sensitivity bias (e.g., do not include a mask). In some cases, since pixel values can be read row by row by the camera system, when the camera system reads the row and / or pixel values corresponding to the boundaries (and / or the rows associated with the boundaries), and / or based on the difference between the pixel values associated with the boundaries (and / or the rows associated with the boundaries) and the pixel values corresponding to one or more other regions in the photosensor pixel array 340 (and / or other rows in the image), the pixel values of the SBP pixels 320 spanning the boundaries 302, 304, 306, and 308 can be detected.

[0081] Figure 3DFIG. is a block diagram showing another configuration of a photosensor pixel array 360 having SBP pixels 320. In this example, the SBP pixels 320 are configured to span the top boundary 302 and the bottom boundary 304 of the photosensor pixel array 340, and the remaining regions of the photosensor pixel array 340 (including the right boundary 306 and the left boundary 308) are configured with photosensor pixels 310 that do not have a sensitivity bias (e.g., do not include a mask). Since pixel values can be read row by row by the camera system, when the camera system reads the rows and / or pixel values corresponding to the top boundary 302 and the bottom boundary 304 (and / or the rows associated with the top boundary 302 and the bottom boundary 304), and / or based on the difference between the pixel values associated with the top boundary 302 and the bottom boundary 304 (and / or the rows associated with the top boundary 302 and the bottom boundary 304) and the pixel values corresponding to one or more other regions in the photosensor pixel array 340 (and / or other rows in the image), the pixels in the SBP pixels 320 spanning the top boundary 302 and the bottom boundary 304 can be easily detected.

[0082] Figure 3E FIG. is a block diagram showing another configuration of a photosensor pixel array 370 having SBP pixels 320. In this example, the SBP pixels 320 are configured to span the right boundary 306 and the left boundary 308 of the photosensor pixel array 370, and the remaining regions of the photosensor pixel array 370 (including the top boundary 302 and the bottom boundary 304) are configured with photosensor pixels 310 that do not have a sensitivity bias (e.g., do not include a mask). Since pixel values can be read row by row by the camera system, when the camera system reads the rows and / or pixel values corresponding to the right boundary 306 and the left boundary 308 (and / or the rows associated with the right boundary 306 and the left boundary 308), and / or based on the difference between the pixel values associated with the right boundary 306 and the left boundary 308 (and / or the rows associated with the right boundary 306 and the left boundary 308) and the pixel values corresponding to one or more other regions in the photosensor pixel array 370 (and / or other rows in the image), the pixel values of the SBP pixels 320 spanning the right boundary 306 and the left boundary 308 can be easily detected.

[0083] Although Figures 3A to 3EThe photoelectric sensor pixels 310 and the SBP pixels 320 in [it] are shown as having the same size and shape. However, it should be noted that in other examples, the sizes and / or shapes of the photoelectric sensor pixels 310 and / or the SBP pixels 320 in the same photoelectric sensor pixel array can be different. For example, in some cases, the SBP pixels 320 can have a different size and / or shape from the photoelectric sensor pixels 310 in the same photoelectric sensor pixel array. In addition, in some cases, the sizes and / or shapes of some of the SBP pixels 320 can be different from the sizes and / or shapes of other SBP pixels 320 in the same photoelectric sensor pixel array.

[0084] Figure 4 Various example configurations of the photoelectric sensor pixels are shown. Here, the photoelectric sensor pixel 310 represents a photoelectric sensor pixel without sensitivity bias. The photoelectric sensor pixel 310 does not have a mask for filtering a certain amount of light and does not have an adjusted aperture to allow the photoelectric sensor pixel 310 to capture more light and thus increase the light sensitivity of the photoelectric sensor pixel 310. On the other hand, the SBP pixels 320A to 320H have different mask and / or aperture configurations.

[0085] In this example, the SBP pixels 320A to 320F have the same aperture as the photoelectric sensor pixel 310 but include different mask patterns configured to block different amounts and / or patterns of light. For example, the SBP pixel 320A has a solid mask covering its associated photodiode, and the SBP pixel 320B has a patterned mask covering its associated photodiode. The SBP pixel 320C has a mask covering the right part of its associated photodiode, but the left part 402 of its associated photodiode is not masked or covered by the mask. Similarly, the SBP pixel 320D has a mask covering the top of its associated photodiode, but the bottom 404 of its associated photodiode is not masked or covered by the mask. On the other hand, the SBP pixel 320E has a mask covering a part of its associated photodiode, but leaves the internal region 406 (e.g., the central region) of its associated photodiode uncovered.

[0086] Like the SBP pixel 320A, the SBP pixel 320F has the same aperture as the photoelectric sensor pixel 310 and a solid mask covering its associated photodiode. However, the solid mask in the SBP pixel 320F is darker than the solid mask in the SBP pixel 320A and can filter more light. Therefore, the SBP pixel 320F is less sensitive to light than the SBP pixel 320A.

[0087] In addition, like SBP pixel 320A, SBP pixel 320G has a solid mask covering its associated photodiode, but has a different aperture in other respects. Therefore, SBP pixel 320G can capture more light than SBP pixel 320A and can be more sensitive to light than SBP pixel 320A. Thus, SBP pixel 320G can better capture the brightness level in a darker scene than SBP pixel 320A.

[0088] Like photoelectric sensor pixel 310, SBP pixel 320H does not have a mask covering its associated photodiode. However, SBP pixel 320H has a larger aperture than photoelectric sensor pixel 310. Therefore, SBP pixel 320H can capture more light than photoelectric sensor pixel 310 and can be more sensitive to light than photoelectric sensor pixel 310. Thus, SBP pixel 320H can better capture the brightness level in a darker scene than photoelectric sensor pixel 310.

[0089] As described above, the size of the SBP pixel and / or the mask implemented for the SBP pixel (and its characteristics) can be used to change, modulate, and / or affect the light sensitivity of the SBP pixel. However, other strategies for changing, modulating, and / or affecting the light sensitivity of the SBP pixel are also contemplated herein. For example, in some cases, the light sensitivity of the SBP pixel can be reduced by decreasing the rate or amount of photons received by the SBP pixel that are converted into charge, to calculate pixel values and / or saturation levels in a bright scene (e.g., when the pixel value is saturated). Similarly, the light sensitivity of the SBP pixel can be increased by increasing the rate or amount of photons received by the SBP pixel that are converted into charge, to calculate pixel values in a dark scene (e.g., when the pixel value is undersaturated).

[0090] Figure 4 The various configurations of the SBP pixels shown are merely illustrative examples provided for explanatory purposes. Those skilled in the art will recognize that other SBP pixel configurations, such as other shapes, sizes, masks, mask patterns, etc., can be used to change, modulate, and / or affect the light sensitivity of the SBP pixel.

[0091] Figure 5FIG. 0 is a diagram showing a system flow 500 for exposure control using SBP pixels. As shown, image sensor 130 includes a photosensor pixel array 502, such as any of the photosensor pixel arrays described previously. The photosensor pixel array 502 includes photosensor pixels (e.g., 200, 310) and SBP pixels (e.g., 215, 230, 320). The SBP pixels in the photosensor pixel array 502 may include photosensor pixels with reduced light sensitivity and / or photosensor pixels with increased light sensitivity. Additionally, the photosensor pixel array 502 may include one or more SBP pixels at one or more boundary positions and / or one or more internal positions.

[0092] The image sensor 130 may generate pixel values 504 corresponding to an image of a scene captured by the image sensor 130. The pixel values 504 may include pixel values calculated / measured by the photosensor pixel array 502 and / or a stream of image pixel values. The pixel values 504 may include one or more pixel values calculated / measured by photosensor pixels not biased by sensitivity, and one or more pixel values calculated / measured by SBP pixels.

[0093] In some cases, if the scene is bright, the pixel values from photosensor pixels not biased by sensitivity may be oversaturated / overexposed (e.g., white or overly white) and / or may exceed the maximum saturation value that the image sensor 130 can calculate / measure. However, the photosensor pixel array 502 may include one or more SBP pixels with reduced light sensitivity. Since one or more SBP pixels have reduced light sensitivity, one or more SBP pixels are able to calculate / measure undersaturated and / or pixel values that do not exceed the maximum saturation / exposure value associated with the image sensor 130. In other words, due to the lower light sensitivity of one or more SBP pixels, the pixel values from one or more SBP pixels may have a lower brightness level and may not be oversaturated / overexposed. As previously explained, even if the SBP pixels are saturated, the SBP pixels may provide additional information that allows for more accurate prediction. For example, if the light sensitivity of an SBP pixel is reduced by an amount X, and a change in the scene still causes the SBP pixel to saturate, the fact that the SBP pixel is saturated will indicate that the exposure of normal photosensor pixels (e.g., photosensor pixels not biased by sensitivity) should be reduced by more than the amount X (e.g., the amount corresponding to how much the light sensitivity of the SBP pixel has been reduced).

[0094] The pixel values of one or more SBP pixels can be used to estimate the actual pixel values of an image, including pixel values that would otherwise exceed the maximum saturation / exposure value associated with image sensor 130, in order to more accurately and efficiently calculate and adjust the exposure settings of image sensor 130 and improve exposure convergence, as further described herein. In some examples, the pixel values from one or more SBP pixels can be used to adjust and / or correct the pixel values from photoelectric sensor pixels that are not sensitivity-biased.

[0095] On the other hand, in some cases, if the scene is dark, the pixel values from photoelectric sensor pixels that are not sensitivity-biased may be undersaturated / underexposed (e.g., black or too dark). However, the photoelectric sensor pixel array 502 can include one or more SBP pixels with increased light sensitivity and can thus measure pixel values that are not undersaturated / underexposed, have increased brightness levels, and / or capture light better in a dark scene. Such pixel values from SBP pixels can be used to estimate the actual pixel values of an image, including pixel values that are not undersaturated / underexposed, in order to more accurately and efficiently calculate and adjust the exposure settings of image sensor 130 and improve exposure convergence, as further described herein. In some examples, the pixel values of one or more SBP pixels can be used to adjust and / or correct the pixel values from photoelectric sensor pixels that are not sensitivity-biased.

[0096] Image sensor 130 can provide pixel values 504 to pixel parser 506. In some examples, pixel parser 506 can be part of image sensor 130 or implemented by image sensor 130. In some examples, pixel parser 506 and image sensor 130 can be part of the same device or implemented by the same device, such as image capture device 105A or image capture and processing system 100. In other examples, pixel parser 506 can be part of a device or component different from image sensor 130 or implemented by a different device or component. For example, in some cases, pixel parser 506 can be implemented by image processor 150 or a device separate from image capture and processing system 100.

[0097] The pixel parser 506 can parse the pixel values 504 and identify the pixel values from the SBP pixels in the photoelectric sensor pixel array 502. For example, the pixel parser 506 can parse the pixel values 504 to identify pixel values that are undersaturated (e.g., undersaturated and / or oversaturated) and / or have a different brightness level from other saturated pixel values. In some examples, the pixel parser 506 can identify the pixel values from the SBP pixels based on the differences between the pixel values in the pixel values 504. For example, as previously explained, if the scene is bright, the pixel values from the photoelectric sensor pixels that are not sensitivity-biased may be oversaturated and / or may exceed the maximum saturation value associated with the image sensor 130, but the pixel values from the SBP pixels with reduced light sensitivity may not be saturated (e.g., oversaturated) and / or may not exceed the maximum saturation value associated with the image sensor 130. Thus, in some cases, the pixel parser 506 can distinguish the pixel values from the photoelectric sensor pixels that are not sensitivity-biased and the pixel values from the SBP pixels with reduced light sensitivity based on their saturation / brightness level and / or the differences in pixel values.

[0098] Conversely, if the scene is dark, the pixel values from the photoelectric sensor pixels that are not sensitivity-biased may be undersaturated, but the pixel values from the SBP pixels with increased light sensitivity may not be undersaturated, may have a higher brightness level, and / or may better capture the light intensity. Thus, in some cases, the pixel parser 506 can distinguish the pixel values from the photoelectric sensor pixels that are not sensitivity-biased and the pixel values from the SBP pixels with increased light sensitivity based on their saturation / brightness level and / or the differences in pixel values.

[0099] In some examples, the pixel parser 506 can read and / or receive the pixel values 504 row by row and detect the pixel values of the SBP pixels in the boundary of the photoelectric sensor pixel array 502 based on the position of the read row within the image pixel array (e.g., the readout in the top row of the image pixel array can correspond to the SBP pixels in the top boundary of the photoelectric sensor pixel array, etc.), the position of the SBP pixels, the position of the read row with the pixel value stream, and / or the differences between the pixel values in the read row and the pixel values in one or more other read rows. For example, in some cases, the pixel parser 506 can detect the pixel values of the SBP pixels in the boundary of the photoelectric sensor pixel array 502 based on the read row corresponding to the boundary position of the SBP pixels, the position of the SBP pixels relative to the positions of other photoelectric sensor pixels, the sequence / order and / or position associated with the read row, and / or the determination that the read row corresponds to the boundary in the image array associated with the pixel values 504.

[0100] In some examples, the pixel parser 506 can detect the difference in saturation / brightness levels between pixel values from SBP pixels and other pixel values. For example, if the pixel parser 506 identifies oversaturated / overexposed pixel values and non-oversaturated / non-overexposed pixel values, the pixel parser 506 can determine that the non-oversaturated / non-overexposed pixel values correspond to SBP pixels with reduced light sensitivity. Similarly, if the pixel parser 506 identifies undersaturated / underexposed pixel values and pixel values with a higher brightness level, the pixel parser 506 can determine that the pixel values with the higher brightness level correspond to SBP pixels with increased light sensitivity.

[0101] After parsing the pixel values 504, the pixel parser 506 can send the SBP pixel data 508 to the exposure control 510. In some examples, the exposure control 510 can be part of or implemented by the image sensor 130. However, in other examples, the exposure control 510 can be a control mechanism implemented by a different device, such as the image signal processor 154, the processor 152, the image processor 150, the image capture device 105A, the image processing device 105B, or a separate device.

[0102] In some cases, after parsing the pixel values 504, the pixel parser 506 can also send the pixel data 514 to the image signal processor 154. The pixel data 514 sent to the image signal processor 154 can include pixel values from photoelectric sensor pixels that are not sensitivity-biased. In some cases, the image signal processor 154 can use the pixel data 514 to generate an image data output 518. In some examples, the image data output 518 can be an image and / or an image pixel array generated by the image signal processor 154.

[0103] In some cases, the pixel data 514 sent to the image signal processor 154 can also include pixel values measured by SBP pixels. For example, the pixel parser 506 can provide pixel values from photoelectric sensor pixels that are not sensitivity-biased to the image signal processor 154 and supply these pixel values to the image signal processor 154. In some cases, the pixel parser 506 or the image signal processor 154 can store the pixel values measured by SBP pixels in the buffer 516. The buffer 516 can be part of or implemented by the image signal processor 154 or a separate device. In some cases, the image signal processor 154 can use the pixel values measured by SBP pixels to adjust the brightness level of the image data output 518. For example, if the image sensor 130 has a 10-bit limit and can only provide a maximum pixel value of 1023, the pixel values of the image sensor 130 will saturate at 1023 and will not be able to capture higher brightness levels.

[0104] However, if the light sensitivity of the SBP pixels is reduced by half (e.g., if the SBP pixels are configured to capture half of the light they receive and filter the other half of the light), and the SBP pixels capture a pixel value of 600 in a scene, the image signal processor 154 may multiply the pixel value 600 by 2 (which reflects the amount of light configured to be filtered from the light used to calculate the pixel value 600) to generate an estimated pixel value of 1200 for the scene. Thus, in this example, the estimated pixel value 1200 can take into account the sensitivity level of the SBP pixels (e.g., 1 / 2). Since the image signal processor 154 may not have the 10-bit limitation of the image sensor 130 and may be able to exceed a pixel value of 1023, the image signal processor 154 can use the 1200 estimated pixel value when generating the image data output 518. It should be noted that the pixel values, 10-bit limitation, and light sensitivity in this example are merely illustrative examples provided for explanatory purposes, and other examples may include other pixel values, bit limitations, and / or light sensitivities.

[0105] The SBP pixel data 508 sent by the pixel parser 506 to the exposure control 510 may include pixel values identified as corresponding to the SBP pixels. For example, the SBP pixel data 508 may include pixel values having a lower luminance level than other pixel values that are over-saturated / over-exposed, and / or pixel values having a higher luminance level than other pixel values that are under-saturated / under-exposed. In some examples, in addition to including pixel values identified as corresponding to the SBP pixels, the SBP pixel data 508 may include one or more over-saturated / over-exposed, and / or under-saturated / under-exposed pixel values from the photoelectric sensor pixels.

[0106] The exposure control 510 can use the SBP pixel data 508 to calculate the exposure control data 512 and provide the exposure control data 512 to the image sensor 130. In some examples, the exposure control data 512 may include automatic exposure control settings for the image sensor 130, such as an exposure adjustment factor and / or an exposure setting. In some examples, the exposure control 510 can use the SBP pixel data 508 to calculate an adjustment factor and / or an exposure setting for the image sensor 130. In some cases, the image sensor 130 can be used for automatic exposure control during exposure convergence.

[0107] In some examples, exposure control 510 may adjust the pixel values of SBP pixels based on the sensitivity level of the SBP pixels. For example, if the light sensitivity of the SBP pixels is reduced by half (e.g., if the SBP pixels are configured to capture half of the light they receive and filter the other half of the light), and the SBP pixels capture a pixel value of 500 in a scene, then exposure control 510 may multiply the pixel value 500 by 2 to generate an estimated or actual pixel value of 1000 for the scene. In this example, the estimated or actual pixel value 1000 may be the adjusted SBP pixel value, which may take into account the sensitivity level of the SBP pixels (e.g., 1 / 2). Since the adjusted SBP pixel value may take into account brightness levels that other photoelectric sensor pixels may not be able to capture, the adjusted SBP pixel value may provide a more accurate way to calculate the exposure settings for image sensor 130 than the pixel values of other photoelectric sensor pixels.

[0108] In some cases, then, exposure control 510 may divide the target exposure value calculated for image sensor 130 (and / or the image pixel values captured by image sensor 130) by the adjusted SBP pixel value (e.g., pixel value 1000 in the previous example) to determine an adjustment factor or exposure setting. For example, if the target value is 50, then exposure control 510 may divide the target value 50 by the adjusted SBP pixel value 1000 to obtain an adjustment factor of 0.05. Exposure control 510 may provide the adjustment factor 0.05 to image sensor 130 for image sensor 130 to adjust its exposure settings (e.g., exposure time, shutter speed, gain, etc.) and / or improve exposure convergence.

[0109] Having disclosed example systems, components, and concepts, the present disclosure now turns to an example method 600 of automatic exposure control using SBP pixels, as Figure 6 shown. The steps outlined herein are provided as non-limiting examples for illustrative purposes and may be implemented in any combination thereof, including combinations that exclude, add, or modify certain steps.

[0110] At block 602, method 600 may include receiving a plurality of pixel values (e.g., 504) of an image captured by a photoelectric sensor pixel array (e.g., 300, 340, 350, 360, 370, 502) of an image sensor (e.g., 130). The plurality of pixel values may include one or more pixel values corresponding to one or more SBP pixels (e.g., 215, 230, 300) in the photoelectric sensor pixel array, and one or more saturated (e.g., under-saturated, over-saturated) pixel values corresponding to one or more other photoelectric sensor pixels (e.g., 200, 310) in the photoelectric sensor pixel array. The one or more other photoelectric sensor pixels may include photoelectric sensor pixels that are not sensitivity-biased (e.g., do not have a mask or filter, do not have a modified aperture, do not have a modified photon-to-charge conversion rate, etc.).

[0111] In some cases, the SBP pixels among the one or more SBP pixels may include photoelectric sensor pixels having a mask configured to filter a portion of light before the portion of light reaches the photoelectric sensor pixel, photoelectric sensor pixels having an aperture different (e.g., larger or smaller aperture) from one or more other photoelectric sensor pixels in the photoelectric sensor pixel array (e.g., compared to photoelectric sensor pixels that are not sensitivity-biased), and / or photoelectric sensor pixels configured to convert photons into charges at a modified rate (e.g., at a reduced or increased rate). Additionally, in some examples, the one or more saturated pixel values may include one or more under-saturated pixel values.

[0112] In some examples, at least some of the one or more SBP pixels among the one or more SBP pixels may be located at one or more boundaries of the photoelectric sensor pixel array. The one or more boundaries may include a bottom row (e.g., bottom boundary), a top row (e.g., top boundary), a left column (e.g., left boundary), and / or a right column (e.g., right boundary). In some examples, at least some of the one or more SBP pixels among the one or more SBP pixels may be located in one or more non-boundary regions (e.g., internal regions) of the photoelectric sensor pixel array.

[0113] In some cases, the one or more saturated pixel values may include over-saturated pixel values, and the one or more SBP pixels may have a reduced light sensitivity. In some cases, the one or more saturated pixel values may include under-saturated pixel values, and the one or more SBP pixels may have an increased light sensitivity.

[0114] In some cases, the one or more SBP pixels may include a first group of pixels and a second group of pixels. The first group of pixels may have a reduced light sensitivity, and the second group of pixels may have an increased light sensitivity.

[0115] At block 604, method 600 may include determining an estimated actual pixel value of one or more of the plurality of pixel values and / or one or more saturated pixel values based on one or more pixel values corresponding to one or more SBP pixels. In some examples, determining the estimated actual pixel value may include multiplying one or more pixel values corresponding to one or more SBP pixels by an SBP light sensitivity factor, which is calculated based on the percentage of light that one or more SBP pixels are configured to filter. In some cases, the SBP light sensitivity factor may be a factor by which the light sensitivity of one or more SBP pixels is adjusted.

[0116] For example, if the light sensitivity of one or more SBP pixels is reduced by 1 / n, where n is a positive number greater than 1, the light sensitivity factor may be n, and the estimated actual pixel value may be calculated by multiplying one or more pixel values corresponding to one or more SBP pixels by n. If the light sensitivity of one or more SBP pixels is increased by n, where n is a positive number greater than 1, the light sensitivity factor may be 1 / n, and the estimated actual pixel value may be calculated by multiplying one or more pixel values corresponding to one or more SBP pixels by 1 / n.

[0117] At block 606, method 600 may include determining an adjustment factor (also referred to as an adjustment value) for at least one of the plurality of pixel values based on the estimated actual pixel value and a target exposure value. In some examples, determining the adjustment factor may include dividing the target exposure value by the estimated actual pixel value.

[0118] At block 608, method 600 may include correcting an exposure setting associated with the image sensor and / or at least one of the plurality of pixel values based on the adjustment factor. For example, in some cases, method 600 may use the adjustment factor to correct an exposure setting associated with the image sensor. In some examples, the exposure setting may include exposure time, shutter speed, gain, lens aperture, brightness, any combination thereof, and / or other exposure settings. In some cases, method 600 may use the adjustment factor to correct at least some of the plurality of pixel values. For example, as described above, the adjustment factor may be used to correct under-saturated / over-saturated and / or over-saturated / over-exposed pixel values. In some cases, method 600 may use the adjustment factor to correct an exposure setting associated with the image sensor and also to correct at least some of the plurality of pixel values.

[0119] In some aspects, method 600 may include generating an image based on the estimated actual pixel value, one or more saturated pixel values, and / or at least a portion of the plurality of pixel values modified based on the adjustment factor.

[0120] In some aspects, method 600 may include: receiving a second plurality of pixel values of an additional image captured by a photoelectric sensor pixel array of an image sensor, wherein one or more additional pixel values correspond to one or more additional SBP pixels in the photoelectric sensor pixel array, and each SBP pixel in the one or more additional SBP pixels includes an SBP photoelectric sensor pixel without an optical filter (e.g., a mask, a color filter), an SBP pixel having a larger aperture than other photoelectric sensor pixels in the photoelectric sensor pixel array, and / or an SBP photoelectric sensor pixel configured to convert photons into charges at an increased rate; determining, based on the one or more additional pixel values, a second estimated actual pixel value of one or more of the additional pixel values and / or one or more different pixel values among the second plurality of pixel values; and determining an additional adjustment factor for one or more under-saturated pixel values among the second plurality of pixel values based on the second estimated actual pixel value and a second target exposure value of one or more under-saturated pixel values.

[0121] In some cases, method 600 may further include correcting a second exposure setting associated with the image sensor and / or at least a portion of the pixel values among the second plurality of pixel values based on the additional adjustment factor.

[0122] In some aspects, method 600 may include identifying one or more pixel values corresponding to one or more SBP pixels. In some aspects, method 600 may include determining an estimated actual pixel value and an adjustment factor based on the identified one or more pixel values corresponding to one or more SBP pixels. In some examples, one or more pixel values are identified based on a difference between the one or more pixel values and one or more saturated pixel values, a location of the one or more SBP pixels, and / or a location of the one or more pixel values within an image array including a plurality of pixel values.

[0123] In some examples, method 600 may be performed by one or more computing devices or apparatuses. In an illustrative example, method 600 may be performed by Figure 1 the image capture and processing system 100 shown, and / or having Figure 7One or more computing devices of the architecture of computing device 700 shown are executed. In some cases, such a computing device or apparatus may include a processor, a microprocessor, a microcomputer, or other components of a device configured to perform the steps of method 600. In some examples, such a computing device or apparatus may include one or more sensors configured to capture image data. For example, the computing device may include a smart phone, a head-mounted display, a mobile device, or other suitable device. In some examples, such a computing device or apparatus may include a camera configured to capture one or more images or videos. In some cases, such a computing device may include a display for displaying images. In some examples, one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such a computing device may also include a network interface configured to communicate data.

[0124] The components of the computing device may be implemented in circuitry. For example, the components may include electronic circuitry or other electronic hardware and / or may be implemented using the same, the electronic circuitry or other electronic hardware may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits); and / or may include computer software, firmware, or any combination thereof and / or be implemented using the same to perform the various operations described herein. The computing device may also include a display (as an example of an output device or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or one or more other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0125] Method 600 is shown as a logic flow diagram, the operations of which represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored in one or more computer-readable storage media, which when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be implemented in any order and / or in parallel combinations to carry out these processes.

[0126] Additionally, method 600 can be performed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed jointly on one or more processors by hardware or a combination thereof. As noted above, the code can be stored in a computer-readable or machine-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0127] Figure 7 FIG. 700 illustrates an example computing device architecture of an example computing device that can implement the various techniques described herein. For example, computing device architecture 700 can implement Figure 1 at least some portions of the image capture and processing system 100 shown and perform sHDR operations as described herein. The components of computing device architecture 700 are shown as being in electrical communication with each other using a connection 705 such as a bus. Example computing device architecture 700 includes a processing unit (CPU or processor) 710 and a computing device connection 705 that couples various computing device components including a computing device memory 715 (such as read only memory (ROM) 720 and random access memory (RAM) 725) to the processor 710.

[0128] Computing device architecture 700 can include a cache of high-speed memory that is directly connected to, adjacent to, or integrated as part of the processor 710. Computing device architecture 700 can copy data from the memory 715 and / or the storage device 730 to the cache 712 for quick access by the processor 710. In this way, the cache can provide a performance boost that avoids latency while the processor 710 waits for data. These and other modules can control or be configured to control the processor 710 to perform various actions. Other computing device memories 715 can also be used. The memory 715 can include a variety of different types of memory with different performance characteristics. The processor 710 can include any general-purpose processor, and hardware or software services stored in the storage device 730 and configured to control the processor 710 and in which software instructions are incorporated into the processor design. The processor 710 can be a self-contained system that includes multiple cores or processors, buses, memory controllers, caches, etc. The multi-core processor can be symmetric or asymmetric.

[0129] To enable a user to interact with the computing device architecture 700, the input device 745 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The output device 735 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device. In some instances, a multimodal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 700. The communication interface 740 can generally govern and manage user input and computing device output. There is no limitation on the operation on any specific hardware arrangement, and thus the basic features herein can be easily replaced by improved hardware or firmware arrangements (when they are developed).

[0130] The storage device 730 is non-volatile memory and can be a hard disk or other types of computer-readable media capable of storing computer-accessible data, such as magnetic tape cartridges, flash memory cards, solid-state storage devices, digital versatile discs, cassette tapes, random access memory (RAM) 725, read-only memory (ROM) 720, and their hybrids. The storage device 730 can include software, code, firmware, etc. for controlling the processor 710. Other hardware modules or software modules are also contemplated. The storage device 730 can be connected to the computing device connection 705. In some aspects, a hardware module that performs a specific function can include a software component stored in a computer-readable medium, as well as the necessary hardware components, such as the processor 710, the connection 705, the output device 735, etc., to implement the function.

[0131] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying (one or more) instructions and / or data. The computer-readable medium can include non-transitory media, where data can be stored, and excludes carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media can include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or storage devices. Code and / or machine-executable instructions can be stored in the computer-readable medium, which can represent a process, a function, a subroutine, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. By passing and / or receiving information, data, arguments, parameters, or memory contents, a code segment can be coupled to another code segment or a hardware circuit. Information, arguments, parameters, data, etc. can be passed, forwarded, or sent via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0132] In some embodiments, a computer-readable storage device, medium, and memory may include a wired or wireless signal including a bitstream and the like. However, when mentioned, a non-transitory computer-readable storage medium explicitly does not include media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0133] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those of ordinary skill in the art will understand that these embodiments may be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks that include devices, device components, steps or routines in a method embodied in software or a combination of hardware and software. Additional components other than those shown and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as not to obscure the embodiments.

[0134] The various embodiments may be described above as a process or method, which is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart may describe operations as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. When the operations of a process are completed, the process is terminated, but there may be additional steps not included in the figure. A process may correspond to a method, function, program, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0135] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored in or otherwise obtained from a computer-readable medium. For example, such instructions may include instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or a group of functions. Portions of the computer resources used may be accessed via a network. For example, the computer-executable instructions may be binary, an intermediate format instruction such as assembly language, firmware, source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0136] Devices implementing the disclosed processes and methods can include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., a computer program product) for performing the necessary tasks can be stored in a computer-readable or machine-readable medium. One or more processors can perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein can also be embodied in peripheral devices or add-on cards. As a further example, such functions can also be implemented in different chips or different processes executed in a single device on a circuit board.

[0137] Instructions, the medium for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functions described in this disclosure.

[0138] In the foregoing description, aspects of the present application have been described with reference to specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the illustrative embodiments of the present application have been described in detail herein, it should be understood that, except as limited by the prior art, the inventive concept can be embodied and used in different ways, and the appended claims are intended to be construed to include such variations. The various features and aspects of the applications described above can be used alone or in combination. Further, embodiments can be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Thus, the specification and drawings are to be regarded as illustrative rather than restrictive. For purposes of illustration, the methods have been described in a particular order. It should be understood that in alternative embodiments, these methods can be performed in a different order than that described.

[0139] One of ordinary skill in the art will understand that, without departing from the scope of this description, the less than (“<”) and greater than (“>”) symbols or terms used herein can be replaced with less than or equal to (“≤”) and greater than or equal to (“≥”) symbols, respectively.

[0140] In cases where a component is described as “configured to” perform certain operations, such configuration can be achieved, for example, by designing an electronic circuit or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or any combination thereof.

[0141] The phrase "coupled to" refers to any component that is physically connected, directly or indirectly, to another component, and / or any component that communicates, directly or indirectly, with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0142] Claim language or other language reciting "at least one" of a set and / or "one or more" of a set means that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one" of a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.

[0143] The various illustrative logical blocks, modules, circuits, and algorithmic steps described in connection with the examples disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0144] The techniques described herein may also be implemented by electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or implemented separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, for example, a random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. Additionally or alternatively, these techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0145] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such processors may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term “processor” as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0146] Exemplary aspects of this disclosure include:

[0147] Aspect 1: A method for processing image data, the method comprising: receiving a plurality of pixel values of an image captured by a photoelectric sensor pixel array of an image sensor, wherein one or more of the plurality of pixel values correspond to one or more sensitivity-biased photoelectric sensor (SBP) pixels in the photoelectric sensor pixel array, and wherein one or more saturated pixel values of the plurality of pixel values correspond to one or more other photoelectric sensor pixels in the photoelectric sensor pixel array; determining an estimated actual pixel value of at least one of one or more of the plurality of pixel values and one or more saturated pixel values based on one or more of the pixel values corresponding to one or more SBP pixels; determining an adjustment factor for at least one of the plurality of pixel values based on the estimated actual pixel value and a target exposure value; and correcting at least one of an exposure setting associated with the image sensor and at least one of the plurality of pixel values based on the adjustment factor.

[0148] Aspect 2: The method according to aspect 1, wherein determining the estimated actual pixel value comprises: multiplying one or more of the pixel values corresponding to one or more SBP pixels by an SBP light sensitivity factor, the SBP light sensitivity factor being calculated based on a percentage of light that one or more SBP pixels are configured to filter.

[0149] Aspect 3: The method according to any one of aspects 1 or 2, wherein determining the adjustment factor comprises dividing the target exposure value by the estimated actual pixel value.

[0150] Aspect 4: The method according to any one of aspects 1 to 3, wherein each of the one or more SBP pixels comprises at least one of a photoelectric sensor pixel having a mask configured to filter a portion of light before the portion of light reaches the photoelectric sensor pixel, a photoelectric sensor pixel having an aperture different from one or more other photoelectric sensor pixels in the photoelectric sensor pixel array, and a photoelectric sensor pixel configured to convert photons into charges at a modified ratio.

[0151] Aspect 5: The method according to any one of aspects 1 to 4, wherein one or more of the SBP pixels are located at one or more boundaries of the photoelectric sensor pixel array, and the one or more boundaries include at least one of a bottom row, a top row, a left column, and a right column.

[0152] Aspect 6: The method according to any one of aspects 1 to 5, wherein one or more of the SBP pixels are located in one or more non-boundary regions of the photoelectric sensor pixel array.

[0153] Aspect 7: The method according to any one of aspects 1 to 6 further includes generating an image based on at least one of the estimated actual pixel values, one or more saturated pixel values, and at least a portion of the plurality of pixel values modified based on an adjustment factor.

[0154] Aspect 8: The method according to any one of aspects 1 to 7, wherein the one or more saturated pixel values include over-saturated pixel values, and wherein the one or more SBP pixels have reduced light sensitivity.

[0155] Aspect 9: The method according to any one of aspects 1 to 8, wherein the one or more SBP pixels include at least one of SBP photoelectric sensor pixels without a light filter, SBP pixels having a larger aperture than other photoelectric sensor pixels in a photoelectric sensor pixel array, and SBP photoelectric sensor pixels configured to convert photons into charges at an increased rate, and wherein the one or more saturated pixel values include one or more under-saturated pixel values.

[0156] Aspect 10: The method according to any one of aspects 1 to 9, wherein the one or more SBP pixels include a first group of pixels and a second group of pixels, wherein the first group of pixels has reduced light sensitivity, and wherein the second group of pixels has increased light sensitivity.

[0157] Aspect 11: The method according to any one of aspects 1 to 10 further includes: identifying one or more pixel values corresponding to the one or more SBP pixels, wherein the one or more pixel values are identified based on at least one of a difference between the one or more pixel values and the one or more saturated pixel values, a position of the one or more SBP pixels, and a position of the one or more pixel values within an image array including a plurality of pixel values.

[0158] Aspect 12: An apparatus for processing image data, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: receive a plurality of pixel values of an image captured by a photoelectric sensor pixel array of an image sensor, wherein one or more of the plurality of pixel values correspond to one or more sensitivity-biased photoelectric sensor (SBP) pixels in the photoelectric sensor pixel array, and wherein one or more saturated pixel values of the plurality of pixel values correspond to one or more other photoelectric sensor pixels in the photoelectric sensor pixel array; determine an estimated actual pixel value of at least one of one or more of the plurality of pixel values and one or more saturated pixel values based on one or more of the pixel values corresponding to the one or more SBP pixels; determine an adjustment factor for at least one of the plurality of pixel values based on the estimated actual pixel value and a target exposure value; and correct at least one of an exposure setting associated with the image sensor and at least one of the plurality of pixel values based on the adjustment factor.

[0159] Aspect 13: The apparatus according to aspect 12, wherein determining the estimated actual pixel value includes multiplying one or more of the pixel values corresponding to the one or more SBP pixels by an SBP light sensitivity factor, the SBP light sensitivity factor being calculated based on a percentage of light that the one or more SBP pixels are configured to filter.

[0160] Aspect 14: The apparatus according to any one of aspects 12 or 13, wherein determining the adjustment factor includes dividing the target exposure value by the estimated actual pixel value.

[0161] Aspect 15: The apparatus according to any one of aspects 12 to 14, wherein each of the one or more SBP pixels includes at least one of a photoelectric sensor pixel having a mask configured to filter a portion of light before the portion of light reaches the photoelectric sensor pixel, a photoelectric sensor pixel having a different aperture from one or more other photoelectric sensor pixels in the photoelectric sensor pixel array, and a photoelectric sensor pixel configured to convert photons into charges at a modified ratio.

[0162] Aspect 16: The apparatus according to any one of aspects 12 to 15, wherein the one or more SBP pixels are located at one or more boundaries of the photoelectric sensor pixel array, the one or more boundaries including at least one of a bottom row, a top row, a left column, and a right column.

[0163] Aspect 17: The apparatus according to any one of aspects 12 to 16, wherein the one or more SBP pixels are located in one or more non-boundary regions of the photoelectric sensor pixel array.

[0164] Aspect 18: The apparatus according to any one of aspects 12 to 17, wherein one or more processors are configured to generate an image based on at least one of the estimated actual pixel values, one or more saturated pixel values, and at least some of the plurality of pixel values modified based on an adjustment factor.

[0165] Aspect 19: The apparatus according to any one of aspects 12 to 18, wherein one or more saturated pixel values include over-saturated pixel values, and wherein one or more SBP pixels have reduced light sensitivity.

[0166] Aspect 20: The apparatus according to any one of aspects 12 to 19, wherein one or more SBP pixels include at least one of SBP photoelectric sensor pixels without a light filter, SBP pixels having a larger aperture than other photoelectric sensor pixels in the photoelectric sensor pixel array, and SBP photoelectric sensor pixels configured to convert photons into charges at an increased rate, and wherein one or more saturated pixel values include one or more under-saturated pixel values.

[0167] Aspect 21: The apparatus according to any one of aspects 12 to 20, wherein one or more SBP pixels include a first group of pixels and a second group of pixels, wherein the first group of pixels has reduced light sensitivity, and wherein the second group of pixels has increased light sensitivity.

[0168] Aspect 22: The apparatus according to any one of aspects 12 to 21, wherein the one or more processors are configured to identify one or more pixel values corresponding to one or more SBP pixels, wherein the one or more pixel values are identified based on at least one of the difference between the one or more pixel values and the one or more saturated pixel values, the position of the one or more SBP pixels, and the position of the one or more pixel values within an image array including a plurality of pixel values; and determine the estimated actual pixel value and the adjustment factor based at least in part on the one or more pixel values corresponding to the one or more SBP pixels.

[0169] Aspect 23: The apparatus according to any one of aspects 12 to 22, wherein the apparatus is a mobile device.

[0170] Aspect 24: The apparatus according to any one of aspects 12 to 23, wherein the apparatus is a camera device.

[0171] Aspect 25: The apparatus according to any one of aspects 12 to 24, further comprising at least one of a display and an image sensor.

[0172] Aspect 26: A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of Aspects 1 to 24.

[0173] Aspect 27: An apparatus comprising one or more components for performing the operations according to any one of Aspects 1 to 24.

Claims

1. A method for processing image data, the method comprising: Receiving a plurality of pixel values of an image captured by a photoelectric sensor pixel array of an image sensor, wherein one or more of the plurality of pixel values correspond to one or more sensitivity-biased photoelectric sensors (SBP) pixels in the photoelectric sensor pixel array, and wherein one or more saturated pixel values of the plurality of pixel values correspond to one or more other photoelectric sensor pixels in the photoelectric sensor pixel array; Determining an estimated actual pixel value of at least one of the one or more pixel values or the one or more saturated pixel values of the plurality of pixel values based on the one or more pixel values corresponding to the one or more SBP pixels and based on the amount of light that the one or more SBP pixels are configured to filter; Determining an adjustment factor for at least one pixel value of the plurality of pixel values based on the estimated actual pixel value and a target exposure value; and Correcting at least one of an exposure setting associated with the image sensor or the at least one pixel value of the plurality of pixel values based on the adjustment factor.

2. The method according to claim 1, wherein, Determining the estimated actual pixel value includes: multiplying the one or more pixel values corresponding to the one or more SBP pixels by an SBP light sensitivity factor, the SBP light sensitivity factor being calculated based on the amount of light that the one or more SBP pixels are configured to filter.

3. The method according to claim 1, wherein Determining the adjustment factor includes: dividing the target exposure value by the estimated actual pixel value.

4. The method according to claim 1, wherein Each of the one or more SBP pixels includes at least one of the photoelectric sensor pixel having a mask configured to filter a portion of light before the portion of light reaches the photoelectric sensor pixel, the photoelectric sensor pixel having an aperture different from one or more other photoelectric sensor pixels in the photoelectric sensor pixel array, or the photoelectric sensor pixel configured to convert photons into charges at a modified ratio.

5. The method according to claim 1, wherein, The one or more SBP pixels are located at one or more boundaries of the photoelectric sensor pixel array, the one or more boundaries including at least one of a bottom row, a top row, a left column, or a right column.

6. The method according to claim 1, wherein The one or more SBP pixels are located in one or more non-boundary regions of the photoelectric sensor pixel array.

7. The method according to claim 1, further comprising generating the image based on at least one of the estimated actual pixel value, the one or more saturated pixel values, or at least a portion of the plurality of pixel values modified based on the adjustment factor.

8. The method according to claim 1, wherein The one or more saturated pixel values include over-saturated pixel values, and wherein the one or more SBP pixels have a reduced light sensitivity.

9. The method according to claim 1, wherein, The one or more SBP pixels include at least one of an SBP photosensor pixel without an optical filter, an SBP pixel having a larger aperture than other photosensor pixels in the photosensor pixel array, or an SBP photosensor pixel configured to convert photons into charge at an increased rate, and wherein the one or more saturated pixel values include one or more undersaturated pixel values.

10. The method according to claim 1, wherein The one or more SBP pixels include a first group of pixels and a second group of pixels, wherein the first group of pixels has a reduced light sensitivity, and wherein the second group of pixels has an increased light sensitivity.

11. The method according to claim 1, further comprising: identifying the one or more pixel values corresponding to the one or more SBP pixels, wherein the one or more pixel values are identified based on at least one of a difference between the one or more pixel values and the one or more saturated pixel values, a position of the one or more SBP pixels, or a position of the one or more pixel values within an image array including the plurality of pixel values.

12. An apparatus for processing image data, the apparatus comprising: at least one memory including instructions; and one or more processors configured to execute the instructions to cause the apparatus to: receive a plurality of pixel values of an image captured by a photosensor pixel array of an image sensor, wherein one or more of the plurality of pixel values correspond to one or more sensitivity-biased photosensor SBP pixels in the photosensor pixel array, and wherein one or more of the plurality of saturated pixel values correspond to one or more other photosensor pixels in the photosensor pixel array; determine an estimated actual pixel value of at least one of the one or more pixel values or the one or more saturated pixel values among the plurality of pixel values based on the one or more pixel values corresponding to the one or more SBP pixels and based on an amount of light that the one or more SBP pixels are configured to filter; determine an adjustment factor for at least one pixel value among the plurality of pixel values based on the estimated actual pixel value and a target exposure value; and correct at least one of an exposure setting associated with the image sensor or the at least one pixel value among the plurality of pixel values based on the adjustment factor.

13. The device according to claim 12, wherein, To determine the estimated actual pixel value, the one or more processors are configured to cause the apparatus to: multiply the one or more pixel values corresponding to the one or more SBP pixels by an SBP light sensitivity factor, the SBP light sensitivity factor being calculated based on an amount of light that the one or more SBP pixels are configured to filter.

14. The device according to claim 12, wherein, To determine the adjustment factor, the one or more processors are configured to cause the apparatus to divide the target exposure value by the estimated actual pixel value.

15. The device according to claim 12, wherein, Each of the one or more SBP pixels includes the photosensor pixel having a mask configured to filter a portion of light before the portion of light reaches the photosensor pixel, the photosensor pixel having an aperture different from one or more other photosensor pixels in the photosensor pixel array, or the photosensor pixel configured to convert photons into charges at a modified rate, or at least one of the photosensor pixels.

16. The device according to claim 12, wherein, The one or more SBP pixels are located at one or more boundaries of the photosensor pixel array, and the one or more boundaries include at least one of a bottom row, a top row, a left column, or a right column.

17. The apparatus according to claim 12, wherein, The one or more SBP pixels are located in one or more non-boundary regions of the photosensor pixel array.

18. The device according to claim 12, wherein, The one or more processors are configured to cause the device to: Generate the image based on at least one of the estimated actual pixel values, the one or more saturated pixel values, or at least a portion of the plurality of pixel values modified based on the adjustment factor.

19. The device according to claim 12, wherein, The one or more saturated pixel values include over-saturated pixel values, and wherein the one or more SBP pixels have reduced light sensitivity.

20. The apparatus according to claim 12, wherein, The one or more SBP pixels include at least one of the SBP photosensor pixels without a light filter, the SBP pixels having a larger aperture than other photosensor pixels in the photosensor pixel array, or the SBP photosensor pixels configured to convert photons into charges at an increased rate, and wherein the one or more saturated pixel values include one or more under-saturated pixel values.

21. The apparatus according to claim 12, wherein, The one or more SBP pixels include a first group of pixels and a second group of pixels, wherein the first group of pixels has reduced light sensitivity, and wherein the second group of pixels has increased light sensitivity.

22. The apparatus according to claim 12, wherein, The one or more processors are configured to cause the device to: Identify the one or more pixel values corresponding to the one or more SBP pixels, wherein the one or more pixel values are identified based on at least one of a difference between the one or more pixel values and the one or more saturated pixel values, a position of the one or more SBP pixels, or a position of the one or more pixel values within an image array including the plurality of pixel values; and Determine the estimated actual pixel values and the adjustment factor based at least in part on the one or more pixel values corresponding to the one or more SBP pixels.

23. The device according to claim 12, wherein, The device is a mobile device.

24. The apparatus according to claim 12, wherein, The device is a camera device.

25. The device according to claim 12, further comprising at least one of a display or an image sensor.

26. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Receiving a plurality of pixel values of an image captured by a photoelectric sensor pixel array of an image sensor, wherein, One or more of the plurality of pixel values correspond to one or more sensitivity-biased photoelectric sensor SBP pixels in the photoelectric sensor pixel array, and wherein one or more of the plurality of saturated pixel values correspond to one or more other photoelectric sensor pixels in the photoelectric sensor pixel array; Determine an estimated actual pixel value of at least one of the one or more pixel values or the one or more saturated pixel values among the plurality of pixel values based on the one or more pixel values corresponding to the one or more SBP pixels and based on the amount of light that the one or more SBP pixels are configured to filter; Determine an adjustment factor for at least one pixel value among the plurality of pixel values based on the estimated actual pixel value and the target exposure value; and Correct at least one of the exposure settings associated with the image sensor or at least one of the at least one pixel value among the plurality of pixel values based on the adjustment factor.

27. The non-transitory computer-readable medium according to claim 26, wherein To determine the estimated actual pixel value, the instructions, when executed by one or more processors, cause the one or more processors to multiply the one or more pixel values corresponding to the one or more SBP pixels by an SBP light sensitivity factor, the SBP light sensitivity factor being calculated based on the amount of light that the one or more SBP pixels are configured to filter.

28. The non-transitory computer-readable medium according to claim 26, wherein, To determine the adjustment factor, the instructions, when executed by one or more processors, cause the one or more processors to divide the target exposure value by the estimated actual pixel value.

29. The non-transitory computer-readable medium according to claim 26, wherein, Each SBP pixel of the one or more SBP pixels includes at least one of the photoelectric sensor pixel having a mask configured to filter a portion of the light before the portion of the light reaches the photoelectric sensor pixel, the photoelectric sensor pixel having an aperture different from one or more other photoelectric sensor pixels in the photoelectric sensor pixel array, or the photoelectric sensor pixel configured to convert photons into charges at a modified ratio.

30. The non-transitory computer-readable medium according to claim 26, wherein, The instructions, when executed by one or more processors, cause the one or more processors to: Generate the image based on at least one of the estimated actual pixel value, the one or more saturated pixel values, or at least a portion of the plurality of pixel values modified based on the adjustment factor.

31. An apparatus for processing image data, the apparatus including components for the steps of the method according to any one of claims 1-11.

32. A computer program product including instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-11.

Images