A system, a method, and a computer program product implementing an image processing pipeline for high dynamic range images

Abstract

A method comprising: Receiving image sensor data from an interleaved image sensor, the interleaved image sensor including a first portion of pixels exposed for a first exposure time and a second portion of pixels exposed for a second exposure time shorter than the first exposure time; identifying a first subset of pixels in the second portion having an intensity value above a first threshold; identifying a second subgroup of pixels in the first portion having an intensity value below a second threshold; Generating high dynamic range (HDR) data based on the first subset and the second subset, and recording the image sensor data by: resetting the pixels in the first part at a first reset time; Resetting the pixels in the second part at a second reset time; and Sampling the pixels in the first part and in the second part after a sampling time has elapsed since the first reset time, wherein the difference between the sampling time and the first reset time is equal to a first exposure time and the difference between the sampling time and the second reset time is equal to a second exposure time which is shorter than the first exposure time, wherein the interleaved image sensor includes a Bayer pattern color filter array arranged in a plurality of rows of four, and wherein the first part comprises odd-numbered rows of four of the interleaved image sensor and wherein the second part comprises even-numbered rows of four of the interleaved image sensor.

Description

FIELD OF THE INVENTION

[0001] The present invention relates to image processing and, more particularly, to an image processing pipeline coupled to an image sensor. BACKGROUND

[0002] Today, digital photographs are captured using a variety of image sensors, such as CMOS (complementary metal oxide semiconductor) image sensors and CCD (charge-coupled device) image sensors. The camera function is often included in current mobile devices. For example, many cellular phones, such as Apple's iPhone and Motorola's Droid, contain an integrated image sensor that a user can use to capture digital images for transmission or storage in the mobile device. The design of these compact camera systems is complicated by the fact that some scenes can have a high level of contrast (i.e., the difference in intensity level between pixels). In other words, in a single scene, some areas of the scene may be well-lit, while other areas of the scene are obscured by shadow.For example, if a user takes a picture outdoors, the scene may contain many objects in direct sunlight and other objects that are shaded from the sun.

[0003] CMOS image sensors used in mobile devices have a limited dynamic range. Each pixel location in the CMOS image sensor acts as a capacitor, capturing photons focused onto the image sensor by a lens during an exposure and building up a charge. The amount of charge that develops at a particular pixel location depends on the pixel sensor's well capacitance. For example, CMOS pixels approximately 1.4 µm in size can have a well capacitance of approximately 5,000 electrons. Once the pixel location has built up a charge equivalent to 5,000 electrons, the pixel location is unable to capture any further information about the scene's brightness. The upper limit of the dynamic range is determined by the well capacitance and the discrete nature of the light.Acquisition noise limits the sensor's highest signal-to-noise ratio (SNR) to the square root of the maximum signal, or approximately 36 dB in our example with 5,000 electrons. The lower limit of the dynamic range is determined by read noise and quantization. Even in the absence of read noise, the charge in the pixel is mapped, or sampled, to a discrete digital value; for example, a 10-bit value. The charge for a pixel can be digitized using a 10-bit ADC (analog-to-digital converter), producing a value between 0 and 1023.

[0004] As previously described, the image sensor is only capable of measuring light with a limited dynamic range. Thus, the information captured by the image sensor depends on the exposure time. Using a short exposure time prevents bright areas of the scene from saturating the corresponding pixel locations. However, detailed information for dark areas of the scene may be lost because the signal in these areas is weak. On the other hand, increasing the exposure time may reveal details in the darker areas of the scene, but may overexpose other areas of the scene.

[0005] One technique for creating high dynamic range (HDR) images is to capture two images of the same scene using different exposure times. Conventionally, a first image is captured at one exposure time, and then a second image is captured at a second exposure time. Once the images are captured, an image processing pipeline combines the two images to create a scene with a greater dynamic range than the image sensor could capture if it captured the scene during a single exposure. More recently, interleaved, or interleaved, image sensors have been developed that capture two images with different exposure times essentially simultaneously. In this case, the interleaved image sensor captures an image of the scene using two different but simultaneous exposure times that are interleaved, or interleaved, across the image sensor.be nested.

[0006] Some image processing algorithms for generating images using interleaved image sensors sacrifice spatial resolution to create HDR images. For example, a first image may be generated using half the pixels, and a second image may be generated using the other half. The first image and the second image are then combined to create an HDR image with half the vertical resolution. There is therefore a need to address this issue and / or other problems associated with the prior art. Further reference is made to the documents DE 100 64 184 C1, US 2009 / 0 262 215 A1, US 2012 / 0 262 600 A1 and US 7 483 058 B1. US 6 175 383 B1 discloses a method for image generation using different exposure times in a pixel matrix.DE 10 2006 055 905 A1 discloses a method for detecting the surroundings of a motor vehicle using sensors with different spectral sensitivities. One object of the invention is to generate an image with a very high dynamic range. OVERVIEW

[0007] This object is achieved by the features of the independent claims. Preferred embodiments are described in the dependent claims. A system, a method, and a computer program product for generating image data with a high dynamic range are disclosed. The method comprises, among other things, the steps of receiving image sensor data from an interleaved image sensor. The interleaved image sensor includes a first portion of pixels exposed with a first exposure time and a second portion of pixels exposed with a second exposure time shorter than the first exposure time.The method may further comprise the steps of: identifying a first subgroup of pixels in the second portion having an intensity value above a first threshold, identifying a second subgroup of pixels in the first portion having an intensity value below a second threshold, and generating high dynamic range (HDR) data based on the first subgroup and the second subgroup. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows a flowchart of a method for generating HDR images according to one embodiment; Fig. 2A shows an interleaved image sensor according to one embodiment; Fig. 2B shows the dependency between HDR data and image sensor data according to one embodiment; Fig. 3 is a scatter plot showing the dependence between pixels in odd-numbered rows of four and pixels in even-numbered rows of four of the interleaved image sensor according to one embodiment; Fig. 4 shows an image processing pipeline using the nested image sensor from Fig. 2, according to one embodiment; Fig. 5A shows a filter for generating pixel values for underexposed or overexposed pixels according to one embodiment; Fig. 5B shows a filter for generating pixel values for underexposed or overexposed pixels according to another embodiment; Fig. 6 shows a parallel processing unit according to an embodiment; Fig. 7 shows the data stream multiprocessor from Fig. 6 according to one embodiment; and Fig. 8 shows an illustrative system in which the various architectures and / or functions of the various previous embodiments may be implemented. DETAILED DESCRIPTION

[0008] The following describes in more detail an image processing pipeline for use with an interleaved image sensor. The image processing pipeline includes a preprocessing unit that converts the image sensor data received from the interleaved image sensor into HDR data, which is companded (compressed and expanded) and then passed to a conventional image signal processor (ISP) for further processing. The conventional ISP implements various functions, such as noise reduction, lens shading correction, mosaicking reduction, color space conversion, gamma correction, chroma subsampling, encoding, etc. The image processing pipeline can be implemented in software, hardware, or a combination thereof.In one embodiment, the image processing pipeline may be implemented as a hardware unit included in a system-on-a-chip (SoC), such as the NVIDIA Tegra application processor. In another embodiment, the image processing pipeline may be implemented in software executed by a processing unit, such as a central processing unit (CPU). In yet another embodiment, the image processing pipeline may be implemented in software executed by a highly parallel processing architecture, such as a graphics processing unit (GPU).

[0009] Fig. 1 shows a flowchart of a method 100 for generating HDR images according to one embodiment. In step 102, a pre-processing unit receives image sensor data from an interleaved image sensor. The image sensor data includes a first portion of pixels exposed with a first exposure time and a second portion of pixels exposed with a second exposure time. The first exposure time is longer than the second exposure time. In step 104, the pre-processing unit determines a first subgroup of pixels in the second portion with an intensity value above a first threshold. The first subgroup of pixels represents pixels in the second portion that have neighboring pixels in the first portion that may be inadmissible. In one embodiment, the pre-processing unit generates a mask having a value (e.g., 0 or 1) indicating whether the corresponding pixel is included in the first subgroup.In step 106, the pre-processing unit identifies a second subset of pixels in the first portion having an intensity value below a second threshold. The second subset of pixels represents pixels in the first portion that have neighboring pixels in the second portion that may be unacceptable. Again, in one embodiment, the pre-processing unit generates a mask containing a value (e.g., 0 or 1) indicating whether the corresponding pixel is included in the second subset. In step 108, the pre-processing unit generates HDR data based on the first subset and the second subset. It should be noted that although various optional features are listed in connection with the method for generating HDR images set forth above, such features are intended for illustrative purposes only and should not be considered limiting in any way.

[0010] Fig. 2A shows an interleaved image sensor 200 according to one embodiment. The image sensor 200 includes a plurality of pixels 210 arranged in a two-dimensional (2D) array. In one embodiment, the image sensor 201 includes a color filter array (CFA) overlaid on the plurality of pixels 210. The CFA may be configured such that a first subset of pixels is associated with a first color filter, a second subset of pixels is associated with a second color filter, and a third subset of pixels is associated with a third color filter. For example, a Bayer pattern CFA implements a recurring 2×2 pattern for a red, a green, and a blue color filter, with each 2×2 array of pixels being overlaid with two green color filters, one red color filter, and one blue color filter. As shown in Fig. 2A, the first row of pixels alternates between a green and a blue color filter, the second row of pixels alternates between a red and a green color filter, the third row of pixels alternates between a green and a blue color filter, and the fourth row of pixels alternates between a red and a green color filter, etc. Each pair of rows is referred to herein as a quad row (e.g., a first quad row 221, a second quad row 222, etc.). In other embodiments, different CFAs may be implemented as part of the interleaved image sensor 200, such as CFAs in the form of RGBE, RGBW, or CYGM.

[0011] Unlike a conventional CMOS image sensor, image sensor 201 is an interleaved image sensor. In a conventional CMOS image sensor, the rows of image sensor 200 are reset in sequential order. The image sensor is exposed to light according to an exposure time, causing charges to build up at each pixel location. The charge built up at each pixel location is inversely proportional to the intensity of the light incident on that pixel location compared to each of the other pixel locations. Once the exposure time has elapsed, the rows of the image sensor are sampled sequentially to generate an array of values representing the intensity of light for each pixel in a digital image. In contrast, interleaved image sensor 200 samples the pixels based on multiple exposure times. In one embodiment, the odd-numbered rows of four (i.e., 221, 223, 225, 227, etc.)) of the image sensor 200 are sequentially reset at a first reset time. Similarly, the integer quadruples (i.e., 222, 224, 226, 228, etc.) of the image sensor 201 are sequentially reset at a second reset time. The odd quadruples and the even quadruples of the image sensor 200 are read out at a sampling time. The difference between the sampling time and the first reset time is equal to a first exposure time, and the difference between the sampling time and the second reset time is equal to a second exposure time shorter than the first exposure time.Consequently, the pixels 210 contained in the odd-numbered rows of four form a first portion 231 of the pixels 210 contained in the image sensor 200, corresponding to a long exposure time, and the pixels 210 contained in the even-numbered rows of four form a second portion 232 of the pixels 210 contained in the image sensor 200, corresponding to a short exposure time. The pixels in the first portion 231 capture more detailed information about the darker areas of the scene, and the pixels in the second portion 232 capture more detailed information about the brighter areas of the scene.Note that in other embodiments, all pixels may be reset at a reset time, the pixels in the even-numbered rows of four may be read out after a second exposure time has elapsed since the reset time, and the pixels in the odd-numbered rows of four may be read out after a first exposure time has elapsed since the reset time, wherein the first exposure time is longer than the second exposure time.

[0012] Fig. 2B shows the relationship between the HDR data 292 and the image sensor data 290 according to one embodiment. As previously described, the interleaved image sensor 200 generates the image sensor data 290 comprising a first portion 231 determined from odd-numbered rows of four and a second portion 232 determined from the even-numbered rows of four, both having the same dynamic range. At certain positions in the image, neighboring pixels capture light from the same object with different intensity values corresponding to different exposure times. For example, the first pixel in the third row of the image sensor (i.e., pixel 291) may capture a green object with an intensity level of 102 (out of 1023) due to the short exposure time of the even-numbered rows of four. However, the first pixel in the fifth row of the image sensor (i.e.,Pixel 295) captures the same green object with an intensity level of approximately 816 (out of 1023) due to the long exposure time of the odd-numbered rows of four. Assuming that both pixels are neither underexposed nor overexposed, the neighboring pixels capture details of the object at different spatial positions, but with different intensity levels according to the exposure ratio.

[0013] The preprocessing unit samples the values from the raw image sensor data and intelligently filters them to generate the HDR data 292, which is a combination of values from the first portion 231, scaled values from the second portion 232, filtered values based on one or more samples in the first portion 231, and filtered values based on one or more samples in the second portion 232. For each pixel in the HDR data 292, the preprocessing unit generates an intensity value for the pixel based on an intensity value of a corresponding pixel in the image sensor data 290. If the corresponding pixel is included in the first portion 231, then the intensity value of the pixel in the HDR data 292 is set equal to the intensity value of the corresponding pixel.If the corresponding pixel is included in the second portion 232, then the intensity value of the pixel in the HDR data 292 is set equal to a scaled version of the intensity value of the corresponding pixel. In one embodiment, the intensity value of the corresponding pixel in the second portion 232 is scaled according to the exposure ratio (i.e., the ratio of the first exposure time to the second exposure time). Note that scaling with the exposure ratio may require additional bits in the HDR data 292. For example, with an exposure ratio of 8, an additional 3 bits are required to scale the intensity values of the second portion 232 by 8.

[0014] In another embodiment, instead of scaling the intensity values of corresponding pixels in the second portion 232 with the exposure ratio, the preprocessing unit scales corresponding pixels in the first portion 231 with the inverse of the exposure ratio. In such an embodiment, the HDR data 292 has the same bit depth as the image sensor data 290. Although some information may be lost by scaling the values down instead of up, an additional step to reduce the bit depth of the HDR data 292 (e.g., by companding) is not required to process the HDR data 292 with a conventional ISP.

[0015] The HDR data 292 may contain certain illegal values. For example, a pixel 293 in the HDR data 292 has an index that is associated with a corresponding pixel 295 in the image sensor data 290. The pre-processing unit may determine whether the value for the corresponding pixel 295 in the first portion 231 is legal based on a neighboring pixel 296 in the second portion 232. If the intensity level of a neighboring pixel 296 in the second portion 232 is above a threshold level t2 indicating that the pixel 295 is illegal due to overexposure, then the pre-processing unit may determine a new value for the pixel based on one or more neighboring pixels in the second portion. In one embodiment, the threshold value t2 is equal to a maximum threshold value t1 multiplied by the inverse of the exposure ratio (i.e., t2 = t1 / r x= 1023 / 8 = 128). Note that an intensity value for a pixel in the second portion 232, representing values based on the short exposure time, may be near neighboring pixels in the first portion that have intensity values approximately equal to the intensity level for the pixel in the second portion 232 multiplied by the exposure ratio. Since t2 multiplied by the exposure ratio equals the maximum intensity level of the image sensor 200, neighboring pixels in the first portion 231 captured using a longer exposure time may have saturated the image sensor 200. Similarly, neighboring pixels in the second portion 232 captured using a short exposure time may be underexposed if pixels in the first portion 231 are below a different threshold (such as a minimum threshold t0 multiplied by the exposure ratio).

[0016] To correct the overexposed pixels in the first portion 231, the preprocessing unit determines neighboring pixels in the second portion 232 whose intensity values are above a first threshold. To correct underexposed pixels in the second portion 232, the preprocessing unit determines neighboring pixels in the first portion 231 that have intensity values below a second threshold. Next, for each pixel in the HDR data 292, the preprocessing unit determines whether a neighboring pixel of the corresponding pixel in the image sensor data 290 is above the first threshold or below the second threshold. In other words, the preprocessing unit determines whether a neighboring pixel of the corresponding pixel is included in the first subgroup or the second subgroup.If the neighboring pixel is included in the first subset, then a new value for the pixel is generated in the HDR data 292 by filtering one or more values from neighboring pixels in the second portion 232 and scaling the reported value by the exposure ratio. If the neighboring pixel is included in the second subset, then a new value for the pixel is generated in the HDR data 292 by filtering one or more values from neighboring pixels in the first portion 231.

[0017] Note that the resulting HDR data will include low-resolution areas of the scene mixed with high-resolution areas of the scene. Low-resolution areas are those areas that contain filtered results because at least some of the pixels in the area were underexposed or overexposed in one of the exposures. High-resolution areas are those areas that contain results calculated from pixels that were neither underexposed nor overexposed during either exposure.Note that the pixels in the image sensor data 292 can be classified as part of three different groups: a first group containing pixels that have neighboring pixels in the first subgroup, indicating that a pixel of the HDR data 292 was generated by filtering one or more values from the second portion 232; a second group containing pixels that have neighboring pixels in the second subgroup, indicating that a pixel of the HDR data 292 was generated by filtering one or more values from the first portion 231; and a third group containing pixels that have neighboring pixels that are not in the first subgroup and not in the second subgroup, indicating that a pixel of the HDR data 292 was generated by scaling a pixel in the second portion 232 or by selecting a value from the first portion 232.

[0018] In a further embodiment, the pre-processing unit may determine a third subgroup of pixels in the second part 232 that are above a third threshold but below the first threshold. The third subgroup indicates neighboring pixels in the second part 232 that are close to pixels that are close to overexposure. The pre-processing unit may further determine a fourth subgroup of pixels in the first part 231 that are below a fourth threshold but above the second threshold. The fourth subgroup indicates neighboring pixels in the first part 231 that are close to pixels that are close to underexposure. The pre-processing unit determines whether a neighboring pixel of the corresponding pixel is in the third subgroup or the fourth subgroup.If the neighboring pixel is included in the third subset, then a new value for the pixel is generated in the HDR data 292 by filtering one or more values from neighboring pixels in the second portion 232 and scaling the filtered value by the exposure ratio to generate a first intermediate result. The preprocessing unit then blends the first intermediate result with the intensity value of the corresponding pixel in the first portion 231. In one embodiment, blending comprises a linear interpolation between the first intermediate result and the intensity value of the corresponding pixel based on the intensity value of the neighboring pixel. Similarly, if the neighboring pixel is included in the fourth subset, then a new value for the pixel is generated in the HDR data 292 by filtering one or more values from neighboring pixels in the first portion 231 to generate a first intermediate result.The preprocessing unit then mixes the first intermediate result with a scaled version of the intensity value of the corresponding pixel in the second part 232.

[0019] It should be noted that in such an embodiment, the pixels in the image sensor data 290 can be classified as part of five different groups: a first group containing pixels that have neighboring pixels in the first subgroup, indicating that a pixel of the HDR data 292 is generated by filtering one or more values from the second part 232; a second group containing pixels that have neighboring pixels in the second subgroup, indicating that a pixel of the HDR data 292 is generated by filtering one or more values from the first part 231; a third group containing pixels that have neighboring pixels in the third subgroup, indicating that a pixel of the HDR data 292 is generated by mixing a scaled and a filtered value from the second part 232 and a value from the first part 231;a fourth group containing pixels that have neighboring pixels in the fourth subgroup, indicating that a pixel of the HDR data 292 is generated by mixing a filtered value from the first portion 232 and a scaled value from the first portion 232; and a fifth group containing pixels that have neighboring pixels that are not in the first subgroup, the second subgroup, the third subgroup, or the fourth subgroup, indicating that a pixel of the HDR data 292 is generated by scaling a pixel in the second portion 232 or by selecting a value from the first portion 231;

[0020] Fig. 3 is a scattergram 300 showing the relationship between pixels in odd-numbered rows of four and pixels in even-numbered rows of four of the interleaved image sensor 200, according to one embodiment. As previously described, pixels in the odd-numbered rows of four (e.g., 221, 223, 225, 227, etc.) are associated with a first exposure time, and pixels in even-numbered rows of four (e.g., 222, 224, 226, 228, etc.) are associated with a second exposure time. The scattergram 300 relates the intensity level of pixels of a particular channel to neighboring pixels belonging to the same channel (i.e., the same color). A different scattergram 300 may be recorded for each channel in the CFA of the interleaved image sensor 200.

[0021] As in Fig. 3, the scattergram 300 shows an exposure ratio (r x) of approximately 8 (i.e., the first exposure time is approximately 8 times longer than the second exposure time). The dependence between the intensity levels of neighboring pixels is approximately linear (i.e., y = ax + b). For example, as shown in the scatter plot 300 of Fig. 3, the slope of a line fit to the sample points in the scatter plot 300 is approximately equal to the exposure ratio. In the scatter plot 300, the minimum intensity level is approximately 45 and the maximum intensity level is approximately 1023. Further, an intensity value above approximately 167 (i.e., 45 + (1023 - 45) / r x ) in an even-numbered row of four (i.e., the second part 232) indicates that there is probably an overexposed pixel in a neighboring odd-numbered row of four (i.e., the first part 231), and an intensity value of less than approximately 360 (i.e., 45*r x) in an odd-numbered row of four (i.e., the first region 231) indicates that there is likely to be an underexposed pixel in an adjacent even-numbered row of four (i.e., the second part 232) for a similar object.

[0022] In one embodiment, the relationship between intensity values for pixels in odd-numbered rows of four and neighboring pixels in even-numbered rows of four, as shown in scatter plot 300, is used to define an exposure ratio for the interleaved image sensor 200. The image sensor 200 can be calibrated during manufacturing by capturing images of scenes with standard illumination. For example, a digital camera with the image sensor 200 can be placed in a light box and photographed with a uniformly illuminated surface with different colors. The image sensor 200 is exposed using two different exposure times for the odd-numbered rows of four and the even-numbered rows of four. The values of the various sampled pixels are then fed into a linear regression algorithm to determine an exposure ratio for the image sensor 200.

[0023] Fig. 4 shows an image processing pipeline 400 associated with the nested image sensor 200 of Fig. 2 according to one embodiment. The image processing pipeline 400 includes a preprocessing unit 410, a companding unit 420, a conventional ISP 430, a tone correction unit 440, an image scaling unit 450, and an encoding unit 460. The interleaved image sensor 200 generates image sensor data sampled based on two different exposure times, i.e., a short exposure and a long exposure. The preprocessing unit 410 receives the image sensor data and generates HDR data, as previously described.

[0024] In one embodiment, the image processing pipeline 400 includes a companding unit 420. The companding unit 420 reduces the number of bits used per intensity value in the HDR data 292 in a non-linear manner so that a conventional ISP 430 can be implemented downstream in the processing of the HDR data 292. In other words, more bits are used to distinguish between lower levels of the signal than bits are used to distinguish between higher levels of the signal. Conceptually, the companding unit 420 is implemented so that a conventional ISP 430 can be used in the image processing pipeline 400. In other words, if the companding unit 420 were not implemented, then an ISP designed to process, for example, 10-bit data would not be able to process the HDR data 292 in the extended dynamic range, for example 13-bit.Instead of scaling the HDR data back to the 10-bit dynamic range, which could cause a loss of information, the companding unit 420 is configured to compress the HDR data 292 in a non-linear manner, thus avoiding unnecessary loss of information. The companding unit 420 can downscale the HDR data 292 to the original LDR dynamic range for further processing by a conventional ISP 430. In another embodiment, the companding unit 420 is not included in the image processing pipeline 400, and the ISP 430 is configured to process the HDR data 292 with a larger bit width.

[0025] The ISP 430 may implement a number of functions typically implemented in a conventional ISP. For example, the ISP 430 may perform functions for performing noise reduction, color conversion, gamma correction, and the like. Since the image processing pipeline 400 processes data that has been processed via the companding unit 420 in a non-linear manner, the image processing pipeline 400 may include a hue correction unit 440 that compensates for the non-linearity of the compression.

[0026] The image processing pipeline 400 further includes an image scaling unit 450, which may be connected to a view-finding unit 490. The image scaling unit 450 may be configured to generate scaled versions of the HDR data at resolutions that differ from the full resolution of the image sensor 200. The view-finding unit 490 may display the HDR image in real time. The image scaling unit 450 is further connected to an encoding unit 460, which is configured to encode the uncompressed image data for storage in a memory. The encoding unit 460 may implement any number of codecs or encoder / decoders for image compression known in the art, including the JPEG (Joint Image Expert Group) codec.

[0027] Fig. 5A shows a filter 500 for generating pixel values for underexposed or overexposed pixels according to one embodiment. If the pre-processing unit 410 determines that a corresponding pixel 501 in the image sensor data 290 is included in the first portion 231, then the pre-processing unit 410 examines a neighboring pixel 503 to determine whether the neighboring pixel 503 is included in the first subset (i.e., has an intensity value above a threshold). If the neighboring pixel 503 is included in the first subset, then an intensity value for the pixel is generated in the HDR data 292 by filtering one or more values in the second portion 232. In one embodiment, the filtered value is based on a single sample in the second portion 232, for example, pixel 503, since the corresponding pixel 501 is located in the first odd-numbered row of four of the image sensor 200.

[0028] If, in a similar way as in Fig. 5A, the pre-processing unit 410 determines that a corresponding pixel 503 in the image sensor data 290 is included in the second portion 232, the pre-processing unit 410 examines a neighboring pixel 501 to determine whether the neighboring pixel 501 is included in the second subset (i.e., has an intensity value below a threshold). If the neighboring pixel 501 is included in the second subset, then the pre-processing unit generates an intensity value for the pixel in the HDR data 292 by filtering one or more values in the first portion 232. In one embodiment, the pre-processing unit 410 implements a filter 500 by interpolating between two samples for neighboring pixels in rows of four that are immediately above and below the corresponding pixel 503.For example, for a pixel in HDR data 292 corresponding to pixel 503, preprocessing unit 410 would check to determine whether the neighboring pixel 501 is below a threshold. If the intensity value for pixel 501 is below the threshold, then preprocessing unit 410 generates an intensity value for the pixel based on an interpolation between pixel 501 and pixel 502.

[0029] Fig. 5B shows a filter 510 for generating pixel values for underexposed or overexposed pixels according to another embodiment. Unlike the filter 500 shown in Fig. 5A, the filter 510 samples more than two values in adjacent rows of four to generate the intensity value for the pixel. As shown in Fig. 5B, for a corresponding pixel 503, when the preprocessing unit determines that a neighboring pixel 506 is included in the second subgroup, the preprocessing unit generates an intensity value for the corresponding pixel 502 in the HDR data 292 by filtering four neighboring pixels (e.g., 501, 502, 504, and 505) in adjacent rows of four. The intensity value is generated by sampling the intensity value of the four neighboring pixels and averaging the threshold values. Note that other types of filtering may be applied to generate an intensity value for pixels that have neighboring pixels in the first subgroup or the second subgroup. For example, a filter implementing a Gaussian convolution kernel may be implemented that samples multiple intensity values of pixels within a filter window surrounding the corresponding pixel.In another embodiment, a filter may select the next adjacent pixel in the row of four directly above or below the corresponding pixel.

[0030] Again, the image processing pipeline 400 described above, and in particular the preprocessing unit 410, may be implemented in software, hardware, or a combination thereof. In one embodiment, portions of the image processing pipeline 400 may be implemented as a shading program configured to execute a parallel processing unit, such as a GPU. An illustrative parallel processing unit is described later. In one embodiment, the GPU is a general-purpose graphics processing unit (GPGPU) configured to perform computations conventionally performed by a CPU. Although the parallel processing unit may be Fig. 6 together with a number of features, such features are given for illustrative purposes only and should not be considered limiting in any way.

[0031] Fig. 6 shows a parallel processing unit (PPU) 600 according to one embodiment. Although a parallel processor is provided herein as an example of the PPU 600, it should be strongly appreciated that such a processor is provided for illustrative purposes only, and any processor may be employed to supplement and / or replace this processor. In one embodiment, the PPU 600 is configured to execute multiple threads concurrently in two or more stream multiprocessors (SMs) 650. A thread (i.e., an execution thread) is an instance of a group of instructions executing in a particular SM 650. Each SM 650, described in more detail below in connection with Fig. 7 may include, without limitation, one or more processing cores, one or more load / store units (LSUs), a level one (L1) cache, shared memory, and the like.

[0032] In one embodiment, PPU 600 includes an input / output (I / O) unit 605 configured to send and receive communication events (i.e., commands, data, etc.) from a central processing unit (CPU) (not shown) via system bus 602. I / O unit 605 may implement a Peripheral Component Interconnect Express (PCIe) interface for communication over a PCIe bus. In alternative embodiments, I / O unit 605 may implement other types of well-known bus interfaces.

[0033] The PPU 600 further includes a main interface unit 610 that decodes the instructions and forwards the instructions to the grid management unit 615 or other units of the PPU 600 (e.g., the memory interface 680) as specified in the instructions. The main interface unit 610 is configured to forward communication events between and among the various logic units of the PPU 600.

[0034] In one embodiment, a program encoded as an instruction stream is written by the CPU to a buffer. The buffer is an area in memory, such as memory 604 or system memory, that is accessible (i.e., read / write) to both the CPU and PPU 600. The CPU writes the instruction stream to the buffer and sends a pointer to the beginning of the instruction stream to PPU 600. Main interface unit 610 passes the pointers to the one or more streams to grid management unit (GMU) 615. GMU 615 selects one or more streams and is configured to manage the selected streams as a group of pending grids. The group of pending grids may include new grids that have not yet been selected for execution and may include grids that are partially executed and have been interrupted.

[0035] A work distribution unit 620, located between the GMU 615 and the SM 650, manages a group of active grids, selects active grids for execution by the SM 650, and issues them. Pending grids are transferred from the GMU 615 to the group of active grids when a pending grid is eligible for execution, i.e., has no unresolved data dependencies. An active grid is transferred to the pending group when the execution of the active grid is blocked by a dependency. When the execution of a grid is complete, the grid is removed from the group of active grids by the work distribution unit 620. In addition to receiving grids from the main interface unit 610 and the work distribution unit 620, the GMU 610 also receives grids dynamically generated by the SM 650 during the execution of a grid.These dynamically generated grids are added to the other pending grids in the pending grid group.

[0036] In one embodiment, the CPU executes a driver kernel that implements an application programming interface (API) that enables one or more applications executing on the CPU to schedule operations for execution on the PPU 600. An application may include instructions (i.e., API calls) that cause the driver kernel to create one or more grids for execution. In one embodiment, the PPU 600 implements a SIMD (Single Instruction Multiple Data) architecture in which each thread block (i.e., chain) in a grid executes concurrently with a different data set from the different threads in the thread block. The driver kernel defines thread blocks formed from k related threads so that the threads in the same thread block can exchange data via shared memory.In one embodiment, a thread block comprises 32 related threads and a grid is an array of one or more thread blocks that execute the same stream, and the different thread blocks can exchange data via global memory.

[0037] In one embodiment, the PPU 600 includes X SMs 650(X). For example, the PPU 600 may include 15 separate SMs 650. Each SM 650 is multi-threaded and configured to execute multiple threads (e.g., 32 threads) from a particular thread block simultaneously. Each of the SMs 650 is connected to a level two (L2) cache 665 via an intersection unit 660 (or other type of interconnect network). The L2 cache 665 is connected to one or more memory interfaces 680. The memory interfaces 680 implement 16-, 32-, 64-, 128-bit data buses, or the like, for high-speed data transfer. In one embodiment, the PPU 600 includes U memory interfaces 680(U), with each memory interface 680(U) connected to a corresponding memory device 604(U).For example, the PPU 600 may be connected to up to 6 memory devices 604, such as synchronized dynamic random access memories in the form of double data rate version 5 graphics memory (GDDR5 SDRAM).

[0038] In one embodiment, PPU 600 implements a multi-level memory hierarchy. Memory 604 is located off-chip in SDRAM connected to PPU 600. Data from memory 604 can be fetched and stored in L2 cache 665, which is located on-chip and shared by the various SMs 650. In one embodiment, each of the SMs 650 also implements an L1 cache. The L1 cache is private memory associated with a specific SM 650. Each of the L1 caches is connected to the shared L2 cache 665. Data from L2 cache 665 can be fetched and stored in any of the L1 caches for processing in the functional units of SM 650.

[0039] In one embodiment, the PPU 600 includes a graphics processing unit (GPU). The PPU 600 is configured to receive instructions specifying shading programs for processing graphics data. Graphics data may be defined as a group of primitives, such as points, lines, triangles, squares, triangle strips, and the like. Typically, a primitive contains data defining the number of vertices for the primitive (e.g., in a model-space coordinate system) and also contains attributes associated with each vertex of the primitive. The PPU 600 may be configured to process graphics primitives to generate a frame buffer (i.e., pixel data for each of the pixels of the display). The driver kernel implements a graphics processing pipeline, such as the graphics processing pipeline defined by the OpenGL API.

[0040] An application writes model data for a scene (i.e., a collection of vertices and attributes) into memory. The model data defines each of the objects that can be visible on a display. The application then makes an API call to the driver kernel requesting that the model data be used to generate and display an image. The driver kernel reads the model data and writes instructions to the buffer to perform one or more operations to process the model data. The instructions can encode different shading programs, including one or more of the following: vertex shading, hull shading, geometry shading, pixel shading, etc. For example, the GMU 615 can configure one or more SM 650s to execute a vertex shading program that processes a number of vertices defined by the model data.In one embodiment, GMU 615 may configure different SMs 650s to concurrently execute different shading programs. For example, a first subset of SMs 650 may be configured to execute a vertex shading program, while a second subset of SMs 650 may be configured to execute a pixel shading program. The first subset of SMs 650 processes vertex data to generate processed vertex data and writes the processed vertex data to L2 cache 665 and / or memory 604.After the processed vertex data is divided into rasters (i.e., converted from three-dimensional data to two-dimensional data in screen space) to generate fragment data, the second subgroup at SM 650 performs pixel shading to generate processed fragment data, which is then merged with other processed fragment data and written to the block buffer in memory 604. The vertex shading program and the pixel shading program can be executed concurrently, thereby processing different data from the same scene in a pipeline-like manner until all model data for the scene is stored as an image in the block buffer.

[0041] The PPU 600 may be included in a desktop computer, a mobile computer, a tablet computer, a smart phone (e.g., a wireless handset), a personal digital assistant (PDA), a digital camera, a handheld electronic device, and the like. In one embodiment, the PPU 600 is formed as a single semiconductor substrate. In another embodiment, the PPU 600 is included in a system-on-a-chip (SoC) along with one or more other logic units, such as a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), and the like.

[0042] In one embodiment, the PPU 600 may be included in a graphics card that includes one or more memory devices 604, such as a GDDR5 SDRAM. The graphics card may be configured to connect to a PCIe slot on a desktop computer motherboard, including, for example, a northbridge chipset and a southbridge chipset. In yet another embodiment, the PPU 600 may be integrated into an integrated graphics processing unit (iGPU) included in the chipset (e.g., northbridge) of the motherboard.

[0043] Fig. 7 shows the data stream multiprocessor 650 from Fig. 6 according to one embodiment. As in Fig. 7, the SM 650 includes an instruction cache 705, one or more scheduling units 710, a register file 720, one or more processing cores 750, one or more double precision units (DPU) 751, one or more special function units (SFU) 752, one or more load / store units (LSU) 753, an interconnect network 780, a shared memory / L1 / cache 770, and one or more texture units 790.

[0044] As previously described, the work distribution unit 620 issues active grids for execution in one or more SMs 650 of the PPU 600. The scheduler 710 receives the grids from the work distribution unit 620 and manages instruction scheduling for one or more thread blocks of each active grid. The scheduler 710 schedules threads for execution in groups of parallel threads, where each group is referred to as a chain. In one embodiment, each chain includes 32 threads. The scheduler 710 may manage many different thread blocks, assigning the thread blocks to chains for execution and then scheduling instructions for the multiple different chains for execution in the various functional units (i.e., the cores 750, DPU 751, SFU 752, and LSU 753) during each clock cycle.

[0045] In one embodiment, each scheduling unit 710 includes one or more command issuing units 715. Each issuing unit 715 is configured to send commands to one or more of the functional units. Fig. In the embodiment shown in Figure 7, the scheduling unit 710 includes two issue units 715, allowing two different instructions from the same chain to be issued during each clock cycle. In alternative embodiments, each scheduling unit 710 may include a single issue unit 715 or additional issue units 715.

[0046] Each SM 650 includes a register file 720 that provides a set of registers for the functional units of the SM 650. In one embodiment, the register file 720 is partitioned among the individual functional units such that each functional unit is assigned a specific portion of the register file 720. In another embodiment, the register file 720 is partitioned among the different chains executed by the SM 650. The register file 720 provides temporary storage for operands associated with the data paths of the functional units.

[0047] Each SM 650 includes L processing cores 750. In one embodiment, the SM 650 includes a large number (e.g., 192, etc.) of separate processing cores 750. Each core 750 is a fully pipelined single-precision processing unit that includes a floating-point arithmetic logic unit and an integer arithmetic logic unit. In one embodiment, the floating-point arithmetic logic units implement the IEEE 754-2008 standard for floating-point arithmetic. Each SM 650 further includes M DPUs 751 that implement double-precision floating-point arithmetic, N SFUs 752 that perform special functions (e.g., copying a rectangle, performing pixel shuffling operations, and the like), and P LSUs 753 that implement load and store operations between the shared memory / 1 cache 770 and the register file 720. In one embodiment, the SM 650 includes 64 DPU 751, 32 SFU 752, and 32 LSU 753.

[0048] Each SM 650 includes an interconnect network 780 that connects each of the functional units to the register file 720 and the shared memory / L1 cache 770. In one embodiment, the interconnect network 780 is a crossover unit that may be configured to connect any of the functional units to any register in the register file 720 or any memory location in the shared memory / L1 cache 770.

[0049] In one embodiment, the SM 650 is implemented in a GPU. In such an embodiment, the SM 650 includes J texture units 790. The texture units 790 are configured to load texture maps (i.e., a 2D array of text elements) from the memory 604 and sample the texture map to generate sampled texture values for use in shading programs. The texture units 790 implement texture operations, such as debugging operations using mip maps (i.e., texture maps with varying degrees of detail). In one embodiment, the SM 650 includes 16 texture units 790.

[0050] The PPU 600 described above can be configured to perform highly parallel computations significantly faster than conventional CPUs. Parallel computation has advantages in graphics processing, data compression, biometric operations, stream processing algorithms, and the like.

[0051] Fig. 8 shows an illustrative system 800 in which the various architectures and / or functions of the various previous embodiments may be implemented. As shown, a system 800 is provided that includes at least one central processor 801 connected to a communications bus 802. The communications bus 802 may be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI Express, AGP (Accelerated Graphics Port), HyperTransport, or one or more other bus protocols or point-to-point communications protocols. The system 800 further includes a main memory 804. Control logic (software) and data are stored in the main memory 804, which may take the form of random access memory (RAM).

[0052] The system 800 further includes input devices 812, a graphics processor 806, and a display 808, e.g., a conventional CRT (cathode ray tube), an LCD (liquid crystal display), an LED (light-emitting diode), a plasma display, or the like. User input may be received from the input devices 812, e.g., a keyboard, mouse, touch-sensitive panel, microphone, and the like. In one embodiment, the graphics processor 806 may include multiple shading programs, a raster module, etc. Each of the foregoing modules may even be arranged on a single semiconductor platform to form a graphics processing unit (GPU).

[0053] In this specification, a single semiconductor platform may refer to a single standalone semiconductor-based integrated circuit or chip. It should be noted that the term "single semiconductor platform" may also refer to multi-chip modules with an enlarged interconnect structure that simulate on-chip functionality and represent a significant improvement over a conventional central processing unit (CPU) and bus implementation. Of course, the various modules may also be arranged separately or may be provided in various combinations of semiconductor platforms according to the user's needs.

[0054] System 800 may further include secondary storage 810. Secondary storage 810 includes, for example, a hard disk drive and / or a removable drive, which may be represented by a floppy disk drive, a magnetic tape drive, a compact disk drive, a digital versatility disk (DVD) drive, a recorder, or a universal serial bus (USB) flash memory drive. The removable storage drive reads from and / or writes to a removable storage device in a well-known manner.

[0055] Computer programs or computer control logic algorithms may be stored in main memory 804 and / or secondary memory 810. Such computer programs, when executed, provide system 800 with the ability to perform various functions. Memory 804, memory 810, and / or other memory are possible examples of computer-readable media.

[0056] In one embodiment, the architecture and / or functions of the various preceding figures may be implemented in conjunction with the central processor 801, the graphics processor 806, an integrated circuit (not shown) having the capabilities of at least a portion of the central processor 801 and the graphics processor 806, a chipset (e.g., a group of integrated circuits designed to perform relevant functions as a unit and sold as such, etc.), and / or in conjunction with another integrated circuit for that purpose.

[0057] Furthermore, the architecture and / or functions of the various preceding figures may be implemented in connection with a general purpose computer system, a printed circuit board system, a game console system intended for entertainment purposes, an application-specific system, and / or any other desired system. For example, system 800 may take the form of a desktop computer, a mobile computer, a service computer, a workstation computer, game consoles, an embedded system, and / or other type of logic. Furthermore, system 800 may take the form of various other devices, including, but not limited to, a personal digital assistant (PDA) device, a mobile phone device, a television, etc.

[0058] Furthermore, although not shown, the system 800 may be connected to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN), such as the Internet, a device-to-device network, a wired network, or the like) for communication purposes.

Claims

[1] A method comprising: Receiving image sensor data from an interleaved image sensor, the interleaved image sensor including a first portion of pixels exposed for a first exposure time and a second portion of pixels exposed for a second exposure time shorter than the first exposure time; identifying a first subset of pixels in the second part having an intensity value that is above a first threshold; identifying a second subgroup of pixels in the first portion having an intensity value that is below a second threshold; Generating high dynamic range (HDR) data based on the first subset and the second subset, and recording the image sensor data by: resetting the pixels in the first part at a first reset time; Resetting the pixels in the second part at a second reset time; and Sampling the pixels in the first part and in the second part after a sampling time has elapsed since the first reset time, wherein the difference between the sampling time and the first reset time is equal to a first exposure time and the difference between the sampling time and the second reset time is equal to a second exposure time that is shorter than the first exposure time, wherein the interleaved image sensor includes a Bayer pattern color filter array arranged in a plurality of rows of four, and wherein the first part comprises odd-numbered rows of four of the interleaved image sensor and wherein the second part comprises even-numbered rows of four of the interleaved image sensor. [2] The method of claim 1, wherein identifying the first subset of pixels in the second part comprises generating a first mask that identifies the pixels in the second part that have an intensity value greater than the first threshold, and wherein identifying the second subset of pixels in the first portion comprises generating a second mask that identifies the pixels in the first portion having an intensity value less than the second threshold. [3] The method of claim 1, wherein generating the high dynamic range data comprises generating an intensity value for each pixel in the HDR data by: Determining whether a corresponding pixel associated with an index for the pixel is included in the first part or the second part; and if the corresponding pixel is contained in the first part, then: Determine whether a neighboring pixel of the corresponding pixel is included in the first subgroup, and if the neighboring pixel is included in the first subgroup, then generating the intensity value for the pixel by filtering one or more sampled values in the second part, or if the neighboring pixel is not included in the first subgroup, then generating the intensity value for the pixel by selecting the intensity value for the corresponding pixel; or if the corresponding pixel is contained in the second part, then: Determine whether the neighboring pixel is included in the second subgroup, and if the neighboring pixel is included in the second subgroup, then generating the intensity value for the pixel by filtering one or more sampled values in the first part to produce a filtered value and scaling the filtered value, or if the neighboring pixel is not included in the second subset, then generating the intensity value for the pixel by scaling the intensity value for the corresponding pixel using an exposure ratio. [4] The method of claim 3, wherein determining whether the neighboring pixel is included in the first subset comprises scanning a first mask, and wherein determining whether the neighboring pixel is included in the second subset comprises scanning a second mask. [5] The method according to claim 4, wherein the first mask is generated by performing a comparison operation between the intensity level of the neighboring pixel and the first threshold, and wherein the second mask is generated by performing a comparison operation between the intensity level of the neighboring pixel and the second threshold. [6] The method of claim 3, wherein filtering comprises performing a linear interpolation between two sampled values. [7] The method of claim 3, wherein filtering comprises generating a weighted sum of a plurality of sampled values. [8] The method of claim 3, wherein the filtering is combined with demosaic processing based on the weighted sum of a plurality of sampled values. [9] The method of claim 1, further comprising modifying the HDR data by companding the HDR data. [10] The method of claim 9, further comprising sending the modified HDR data to an image signal processor configured to perform at least one of the functions of noise reduction, demosaicing, color conversion, and gamma correction. [11] A non-transitory computer-readable storage medium that stores instructions that, when executed by a processor, cause the processor to perform steps including: Receiving image sensor data from an interleaved image sensor, the interleaved image sensor including a first portion of pixels exposed for a first exposure time and a second portion of pixels exposed for a second exposure time shorter than the first exposure time; identifying a first subset of pixels in the second part having an intensity value above a first threshold; identifying a second subgroup of pixels in the first part having an intensity value below a second threshold; and Generating high dynamic range (HDR) data based on the first subset and the second subset, and capturing the image sensor data by: resetting the pixels in the first part at a first reset time; Resetting the pixels in the second part at a second reset time; and Sampling the pixels in the first part and in the second part after a sampling time has elapsed since the first reset time, wherein the difference between the sampling time and the first reset time is equal to a first exposure time, and the difference between the sampling time and the second reset time is equal to a second exposure time that is shorter than the first exposure time. wherein the interleaved image sensor includes a Bayer pattern color filter array arranged in a plurality of rows of four, and wherein the first part comprises odd-numbered rows of four of the interleaved image sensor, and wherein the second part comprises even-numbered rows of four of the interleaved image sensor. [12] The non-transitory computer-readable storage medium of claim 11, wherein generating the high dynamic range data comprises generating an intensity value for each pixel in an HDR image by: Determining whether a corresponding pixel associated with an index for the pixel is included in the first part or in the second part; and if the corresponding pixel is contained in the first part, then: Determine whether a neighboring pixel of the corresponding pixel is included in the first subgroup, and if the neighboring pixel is included in the first subgroup, then generating the intensity value for the pixel by filtering one or more values in the second part, or if the neighboring pixel is not included in the first subgroup, then generating the intensity value for the pixel by scaling the intensity value for the corresponding pixel using an exposure ratio; or if the corresponding pixel is contained in the second part, then: Determine whether the neighboring pixel is included in the second subgroup, and if the neighboring pixel is included in the second subgroup, then generating the intensity value for the pixel by filtering one or more values in the first part, or if the neighboring pixel is not included in the second subset, then generating the intensity value for the pixel by selecting the intensity value for the corresponding pixel. [13] The non-transitory computer-readable storage medium of claim 12, wherein filtering comprises performing a linear interpolation between two sampled values. [14] A system comprising: an interleaved image sensor comprising a first portion of pixels exposed for a first exposure time and a second portion of pixels exposed for a second exposure time shorter than the first exposure time, wherein the interleaved image sensor includes a Bayer pattern color filter array arranged in a plurality of rows of four, and wherein the first portion comprises odd-numbered rows of four of the interleaved image sensor and wherein the second portion comprises even-numbered rows of four of the interleaved image sensor; and an image processing pipeline connected to the nested image sensor and configured to: to receive image sensor data from the nested image sensor, to identify a first subgroup of pixels in the second part with an intensity value above a first threshold, to identify a second subgroup of pixels in the first part with an intensity value below a second threshold, and to generate high dynamic range (HDR) data based on the first subset and the second subset, wherein the image sensor is configured to capture the image sensor data by: resetting the pixels in the first part at a first reset time; Resetting the pixels in the second part at a second reset time; and Sampling the pixels in the first part and in the second part after a sampling time has elapsed since the first reset time, wherein the difference between the sampling time and the first reset time is equal to a first exposure time and the difference between the sampling time and the second reset time is equal to a second exposure time which is shorter than the first exposure time. [15] The system of claim 14, wherein generating high dynamic range data comprises generating an intensity value for each pixel in an HDR image by: Determining whether a corresponding pixel associated with an index for the pixel is included in the first part or the second part; and if the corresponding pixel is contained in the first part, then: Determine whether a neighboring pixel of the corresponding pixel is included in the first subgroup, and if the neighboring pixel is included in the first subgroup, then generating the intensity value for the pixel by filtering one or more values in the second part, or if the neighboring pixel is not included in the first subset, then generating the intensity value for the pixel by scaling the intensity value for the corresponding pixel using an exposure ratio; or if the corresponding pixel is contained in the second part, then: Determine whether the neighboring pixel is included in the second subgroup, and if the neighboring pixel is included in the second subgroup, then generating the intensity value for the pixel by filtering one or more values in the first part, or if the neighboring pixel is not included in the second subset, then generating the intensity value for the pixel by selecting the intensity value for the corresponding pixel. [16] The system of claim 14, wherein the image processing pipeline is implemented as a shading program configured to be executed by a graphics processing unit (GPU).

Images