METHOD AND DEVICE FOR IMAGE CORRECTION

The method and device correct image artifacts in CCD sensors by generating a diagonal reference and using a compensation image to address non-vertical light source movements, enhancing image quality through comprehensive artifact correction.

DE112010003748B4Active Publication Date: 2024-11-28QUALCOMM INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
DE112010003748
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2010-09-22
Filing Date
2010-09-23
Publication Date
2024-11-28
Estimated Expiration
2030-09-23

AI Technical Summary

Technical Problem

Conventional methods for correcting image artifacts due to overexposed pixels in CCD image sensors fail to address non-vertical movements of light sources, leading to incomplete or erroneous corrections.

Method used

A method and device for correcting image data that includes generating a diagonal reference based on overexposed pixels and using a reference compensation image to provide both vertical and non-vertical compensation for image artifacts such as smear and bloom, accounting for global and local motion of light sources.

Benefits of technology

Improves image quality by effectively correcting image artifacts caused by non-vertical movements of light sources, providing comprehensive compensation that addresses both vertical and non-vertical smearing and blooming issues.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Methods for correcting image data from an image sensor, comprising: Receiving (405) image data and smear-sensitive data for a first single image; Detect (410) one or more overexposed pixels in the image data; Generate (415) a diagonal reference based on the one or more overexposed pixels and the smear-sensitive data; Generating (420) a reference compensation image, RCI - Reference Compensation Image, based on the diagonal reference estimation and at least a part of the smear-sensitive data, wherein the smear-sensitive data include at least Optical Black, OB, pixel data of the image sensor, wherein the RCI takes into account an acceleration with respect to time in at least one of three dimensions over at least two frames of the image data associated with the part of the smear-sensitive data; Generating (425) a compensation factor image, CFI - Compensation Factor Image, based on the RCI and correcting (430) image data for one or more pixels based on the CFI.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is based on, and claims priority from, Provisional Application No. 61 / 244,974, filed September 23, 2009, and U.S. Patent Application No. 12 / 888,296, filed September 22, 2010. FIELD OF INVENTION

[0002] The present disclosure relates generally to image processing and, more particularly, to correcting image data for one or more image artifacts due to overexposed pixels.

[0003] The present disclosure relates generally to image processing and, more particularly, to correcting image data for one or more image artifacts due to overexposed pixels. GENERAL STATE OF THE ART

[0004] Many imaging devices use charge-coupled devices (CCDs) as image sensors to detect image data. Conventional methods and devices for CCD image detection typically use an array of photoactive areas, or pixels, to detect image data. However, these conventional methods and devices can be susceptible to pixel overexposure due to one or more light sources detected by the imaging array. As such, these conventional methods and devices generate image data containing one or more artifacts, such as smear.

[0005] Now with reference to Fig. 1, a conventional CCD image array 100 is shown, comprising multiple columns. Conventional methods and apparatus may read CCD data by sequentially clocking each of a plurality of columns to the top of the column, reading the value from the last CCD cell of the column into a register from which the data corresponding to a particular row of the CCD image sensor is then read. Alternatively, data from the CCD image array may be read in a raster scan through multiple vertical shift registers (e.g., one vertical shift register for each column of pixels), followed by a horizontal shift register. Each time the vertical shift registers are clocked, a new row is shifted into the horizontal shift register. Each time the horizontal shift register is clocked, a new pixel is generated from the horizontal shift register and from the sensor as a whole.Consequently, an overexposed pixel affects all pixels shifted into register after the affected pixel and appears as a smear artifact in the image.

[0006] US 2006 / 0274173 A1 describes a method in which a horizontal evaluation data generation unit calculates an average value of the pixel signals in a vertical optical black area based on given pixel data in the horizontal optical black area and outputs the calculated average value to an evaluation section.

[0007] US 2002 / 0149674 A1 describes an electro-optical frame camera with forward motion compensation (FMC) system comprising a movable shutter and a full-frame focal plane array detector.

[0008] US 2008 / 0231735 A1 describes an activity-based system and method for reducing Gr-Gb gain imbalances and a digital camera incorporating this system or method.

[0009] AU Xing Yu Qi, et al.: "Motion Deblurring for Optical Character Recognition". In: Eights international proceedings on document analysis and recognition; 31st Aug. 2005. IEEE pp. 289-393, doi: 10.1109 / ICDAR.2005.162 investigates the problem of motion deblurring in image acquisition of text documents.

[0010] During video and preview capturing, image scenes containing a bright light source can contaminate one or more vertical shift registers, causing pixels below and above the light source to become brighter. The column of brighter pixels is referred to as "smear." It is also possible for an overexposed area of ​​a CCD device to affect neighboring pixels to the left and right, further obscuring the effect of the overexposed area. In addition, overexposed pixels can affect pixels all the way to the read register. Movement of light sources within a detection area of ​​the CCD can further affect overexposure of various pixels horizontally across array 100.

[0011] Conventional smear correction methods use vertical correction of pixel data. However, these conventional methods do not address non-vertical movement of a light source across the plane of a detection array. The movement of a light source with respect to the CCD plane can be complex. Vertical movement (i.e., movement in the direction of data movement as the read image is read out by the sensor's vertical shift register) is addressed in the prior art. However, there is no compensation for the non-vertical movements of the light source(s) and / or the image sensor, and therefore the correction is partial at best and sometimes undetectable, and erroneous at worst.

[0012] Thus, there is a desire to address artifact correction for CCD devices. BRIEF DESCRIPTION OF THE EMBODIMENTS

[0013] Methods and apparatus for correcting image data from an image sensor are disclosed and claimed herein. The subject matter of the invention is defined by the independent claims.

[0014] Other aspects, features, and techniques of this document will become apparent to those skilled in the art in light of the following detailed description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The features, objects, and advantages of this document will become apparent from the detailed description set forth below, taken in conjunction with the drawings, in which like reference numerals identify the invention accordingly. In the drawings: Fig. 1 a graphic representation of a conventional CCD (charge coupled device) image array; Fig. 2 is a simplified block diagram of a device for correcting image data according to one or more embodiments; Fig. 3 is a simplified illustration of a CCD array according to one embodiment; Fig. 4 illustrates a process for image correction according to an embodiment; Fig. 5A - Fig. 5B illustrates graphical representations of diagonal references according to one or more embodiments; Fig. 6 is a graphical representation of a plurality of overexposure regions according to one embodiment; Fig. 7 illustrates a process for image correction according to an embodiment; Fig. 8 a process for image correction and Fig. 9 a simplified representation of a CCD array according to another embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] One aspect relates to correcting image data from an image sensor. In one embodiment, the image correction may be provided for vertical and / or non-vertical correction of the image data. According to another embodiment, a method is provided for correcting image data from an image sensor based on one or more overexposed pixels of the image sensor. As such, one or more image artifacts such as smear and / or bloom may be corrected. Furthermore, the image correction may be provided for one or more global and local motion of one or more artifacts.

[0017] In one embodiment, a method is provided that includes generating a diagonal reference based on the one or more overexposed pixels and generating a reference compensation image (RCI) based on the diagonal reference estimate and smear-sensitive data (e.g., reference data), including optical black (OB) pixel data and dummy line data, detected by the image sensor. As used herein, references to dummy line data may refer to smear-sensitive dummy line data. The method may further include generating a compensation factor image (CFI) based on the RCI for correcting one or more of the overexposed pixels.

[0018] According to another embodiment, an apparatus is provided that includes an image sensor, such as a charge-coupled device (CCD), configured to allow non-vertical or vertical correction. The apparatus may be configured to identify non-vertical movement of overexposed pixels and provide compensation relative to such movement in addition to traditional vertical compensation. Consequently, addressing the non-vertical movement aspects of one or more light sources and / or the image sensor may provide improved image quality. The apparatus may additionally be configured to provide compensation for overexposed pixels associated with one or more global and local movements associated with the imaging array. Description of the embodiments

[0019] Now referring to the figures, Fig. 2 shows a simplified block diagram of an imaging device according to one or more embodiments. According to one embodiment, the device 200 may be configured to perform vertical correction and / or non-vertical correction of one or more pixels of detected image data. In certain embodiments, determining whether to perform vertical and / or non-vertical correction may be based on the movement of one or more overexposed pixels. Accordingly, the device 200 may be configured to identify non-vertical movement of one or more overexposed pixels and provide compensation relative to such movement in addition to the traditional vertical compensation, if applicable.Consequently, the apparatus 200 may treat one or more artifacts, such as smearing and / or blooming, due to non-vertical movement of one or more of the light sources to provide improved image quality. According to another embodiment, the apparatus 200 may be configured to identify one or more of the global and local movements of the one or more overexposed pixels to provide vertical and / or non-vertical compensation.

[0020] As in Fig. 2, the device 200 includes an image sensor 205 configured to detect incident light energy, shown as 210. In one embodiment, the image sensor 205 refers to a charge-coupled device (CCD) configured to output an array of pixel data. Based on the detected light energy, one or more of the pixels of the image sensor 205 may be addressed to generate a digital image. According to another embodiment, the image sensor 205 may further include one or more OB (Optical Black) sections of the image array. The OB sections may be masked prior to receiving external light energy to output a black level. As described below with respect to Fig. 3, the image sensor 205 may include an upper OB section (TOB) and a lower OB section (BOB). The device 200 may be configured to detect, via the image sensor 205, image data relating to image data for one or more individual images. Image data detected by the device 200 may relate to digital images, video images, etc. A buffer 215 may be configured to provide image data detected by the image sensor 205 to a processor 220 and may be further configured to temporarily store image data. In one embodiment, the buffer 215 may relate to a shift register.

[0021] The processor 220 may be configured to detect one or more overexposed pixels and determine one or more correction factors for the one or more overexposed pixels from detected image data. By way of example, the processor 220 may be configured to provide a motion estimate of each light source associated with the overexposed pixel data to generate a diagonal reference, as described with reference to Fig. 5A - Fig. 5B below. The processor 220 may then generate a reference compensation image (RCI) based on pixels associated with the diagonal reference. The processor 220 may then calculate a correction factor image (CFI) for correcting image data based on one or more RCIs. The processor 220 may relate to an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a processor in general. As shown in Fig. 2, the processor 220 is coupled to a memory 225. The memory 225 may be configured to store one or more executable instructions and intermediate results (e.g., processed / unprocessed image data) for the processor 220 and may be a RAM or ROM memory. In certain embodiments, the device 200 may cooperate with an external memory and / or an interface with a removable memory (in Fig. 2 not shown).

[0022] An input / output (I / O) interface 230 of device 200 may be configured to receive one or more user commands and / or output data. For example, I / O interface 230 may include one or more buttons for receiving user commands for the imaging device. Alternatively, or in combination, I / O interface 230 may include one or more connectors or ports for receiving and / or outputting data. In certain embodiments, device 230 includes an optional display 235 configured to display detected image data, a user interface, menu applications, etc. I / O interface 230 may output corrected image data.

[0023] As in Fig. 2, elements of device 200 are shown as modules, but it should be equally understood that one or more modules may refer to hardware and / or software components. Furthermore, functions of the modules could refer to computer-executable program code segments.

[0024] Now with reference to Fig. 3 is a simplified block diagram of a CCD array of the image sensor of Fig. 1 according to one embodiment. The CCD array includes a pixel array 300 configured to detect incident light energy for one or more imaging applications. A shift register 305 can access pixel data. According to another embodiment, the array 300 may include one or more optically black pixels. As shown in Fig. 3, array 300 includes top OB pixels (TOB) 310 and bottom OB pixels (BOB) 315. OB pixels may be masked from incident light energy to provide a black level to a processor of an imaging device. In certain embodiments, pixel array 300 may include a single OB section, such as only one of TOB 310 and BOB 315. Pixel data associated with pixel array 300, TOB 310, and BOB 315 may be output from shift register 305, as shown as 320. According to another embodiment, image correction of artifacts may be associated with pixel values ​​of TOB 310 and BOB 315.

[0025] According to one embodiment, when it is time to output an image captured by pixels of array 300, one or more pixels are shifted vertically to shift register 305, one shift per row. The output 320 of shift register 305 can be read either serially or in parallel to output the pixel data of a respective row. As such, array 300 can include a series of vertical shift registers below the capture plane of array 300 to capture pixel data.

[0026] Fig. 3 additionally shows one or more pixels that can be addressed to provide vertical correction according to one or more embodiments. For example, to read pixel 325ij, the pixel data first moves to row i-1, then row i-2, and finally through TOB 310 rows until the pixel data reaches shift register 305 for reading. If pixel 325 is overexposed, the following pixel will be overexposed after the overexposed pixel is shifted to pixel 325i-1,j. Overexposed pixels may result from saturation due to overexposure of the pixel to one or more light sources. Overexposed pixels may be detected by comparing a pixel output level to a saturation threshold. In one embodiment, only OB pixel output levels are compared to the saturation threshold.Shifting the overexposed pixel data can result in overexposed pixel data for an entire vertical row. Consequently, a smear of the overexposed pixel is read as an entire column of the image.

[0027] According to one embodiment, a correction of the pixel array 300 may be performed by measuring the values ​​of the same column at TOB 310 and optionally at BOB 315 and reducing the value of the overexposed pixel from the black levels to obtain a vertical correction. However, the vertical correction typically only corrects vertical smearing. Because a vertical correction cannot accurately account for a moving light source or a moving image sensor, and particularly when multiple light sources are provided, a non-vertical correction may be provided according to one embodiment. Analogously, a correction performed on image data detected by the pixel array 300 may be based on global and / or local movement of one or more light sources.Local motion can refer to the movement of the light source itself, and therefore is not necessarily the same for all light sources in the image (each light source may have its own motion characteristics). Local motion of a light source can affect a portion of the image data, such as a region of pixels within the frame, based on a comparison of a first and second frame (e.g., past, present, future). Global motion is motion typically generated by moving the camera, and therefore affects all light sources in the image.

[0028] Now with reference to Fig. 4 illustrates a process for image correction according to one or more embodiments. Process 400 may be performed by a processor (e.g., processor 220) of an imaging device to provide artifact correction according to one or more embodiments. In one embodiment, process 400 relates to non-vertical correction of one or more pixels of an image sensor, such as a CCD device. Process 400 may be initiated by receiving image data at block 405. In certain embodiments, the processor may determine whether correction is needed based on the detection of one or more overexposed pixels. Accordingly, the processor may be configured to detect one or more overexposed pixels at block 410.In one embodiment, process 400 may be performed for a sequence of frames, where the index of the current frame is n, the index of the previous frame is n-1, the index of the next frame is n+1, and so on. Each frame contains a TOB line and, in certain cases, a BOB line, since not all CCDs contain a BOB line.

[0029] Detecting the one or more overexposed pixels may include determining a movement associated with the one or more overexposed pixels. According to one embodiment, after detecting one or more overexposed pixels, process 400 may include correcting the overexposed pixels. Correcting the overexposed pixels may be based on vertical and / or non-vertical correction. In certain embodiments, a processor of the imaging device may determine the type of correction based on an estimate of the movement of one or more light sources associated with the overexposure, since each light source creates a smear gap in the image. As such, the shape of the smear gap (e.g.,vertical / diagonal / other) depend on the motion characteristics of a particular light source, from the time the top line of the image is read out to the time the bottom line of the image is read out. The processor may be configured to estimate motion in three dimensions: horizontal, vertical, and distance from the camera (with the light source moving closer to or farther away from the camera). The speed of the light source may not necessarily be constant in every direction. For example, a light source approaching the camera will produce a trapezoidal smear gap (e.g., narrower at the top and wider at the bottom), while a light source with constant horizontal motion may produce a diagonal smear gap.Similarly, a horizontally accelerating light source can create a diagonal smear gap with a certain curvature by making the gap angle moderate at the top and more pronounced at the bottom. Other types of motion can be related to changes in the type of light source. For example, a light source with variable intensity, variable angle, variable size, variable shape, etc.

[0030] Process 400 may use motion estimation based on an image itself and / or optical black (OB) lines. In one embodiment, the motion estimation is based on one or more OB lines, i.e., TOB lines, BOB lines, or both. Furthermore, the motion estimation may be based on a current frame, one or more previous frames, and / or one or more future (e.g., subsequent) frames. In one embodiment, motion estimation may be performed by comparing BOB lines of a current frame with TOB lines of a current frame. In the absence of BOB lines, motion estimation may be performed by comparing one or more TOB lines of a subsequent frame with TOB lines of a current frame.Similarly, motion estimation can be based on a comparison of a TOB line of a subsequent frame with a TOB line of a current frame. If on-the-fly correction needs to be made (e.g., when future information is unavailable), motion estimation can be based on a comparison of TOB lines of a current frame with TOB lines of the previous frame, using OB lines further in the past and / or future. OB data further in the past (or future) can be incorporated into the estimation to estimate acceleration, deceleration, rather than simply constant motion. It is also understood that other types of motion estimation can be used.

[0031] To correct the overexposed pixel data, the processor may generate a diagonal reference at block 415. In one embodiment, reference TOB lines (RTOB) may be generated. For example, the RTOB lines may be used as TOB lines. Alternatively, the RTOB lines may be generated by using the TOB lines and performing noise reduction. The noise reduction may be spatial, temporal, or both. The processor may be configured to provide a temporal filter that is motion-compensated. Temporal filtering may include pixel data from other OB lines in addition to the TOB lines of the current frame. Consequently, the RTOB lines may be generated by taking a subset of TOB / BOB lines from current / future / past frames and performing processing on that subset to generate the RTOB lines for the current frame.

[0032] At block 420, a reference compensation image (RCI) may be generated based on the RTOB lines and the motion characteristics of each light source. Alternatively, or in combination, the RCI may be generated based on dummy line data detected by the image sensor. In one embodiment, motion may be described by a horizontal motion vector, and there is only one RTOB line. In this case, for each line in the image from top to bottom, the RTOB line may be shifted slightly more to the right or left (depending on the direction of motion), creating a diagonal reference image. The number of pixels the RTOB line needs to be moved for each line in the image can be calculated by dividing the horizontal motion vector by the number of lines in the image. Interpolation may be used to move the RTOB line with sub-pixel accuracy.

[0033] If multiple light sources are detected, the calculation of the reference image can be repeated for each light source. The results of the reference image calculation can be summed to produce a single reference image (RCI). A constant offset may need to be subtracted from the last reference image. The differentiation into multiple light sources is described below with respect to Fig. 6 discussed.

[0034] Alternatively to, or in combination with, calculating RTOB lines, process 400 may include generating reference BOB lines (RBOB lines) at block 415. The RBOB lines may be generated in a similar manner to the RTOB lines. For example, the RBOB lines may be used as the BOB lines themselves. If no BOB lines exist, the TOB lines of the next frame may be used. Thus, the exemplary and non-limiting RBOB lines may be generated by taking a subset of TOB / BOB lines from current / future / past frames and performing any type of processing on this subset to generate RBOB lines for the current frame.

[0035] In one embodiment, an RCI can be generated based on RBOB lines and the motion characteristics of each light source. In this case, the RBOB lines are taken instead of the RTOB lines, and the reference image is generated from bottom to top using flipped motion vectors.

[0036] At block 425, a compensation factor image (CFI) may be generated based on the RCI. In one embodiment, a weighted average may be performed between the RCI generated from RTOB lines and the RCI generated from RBOB lines to determine a compensation image (CI). Weight changes may be calculated for each line in the image by the processor. According to another embodiment, the weight of the reference image generated from RTOB lines will be stronger for the upper image lines, and the weight of the reference image generated from RBOB lines will be stronger for the lower image lines.

[0037] The processor may be configured to correct one or more overexposed pixels based on the CFI at block 430 by subtracting a last reference image from the current image to correct one or more smear artifacts. According to another embodiment, the correction is not a simple subtraction, but rather a selective and weighted subtraction. For example, in certain embodiments, saturated image pixels (e.g., pixels assigned the maximum value according to a sensor's dynamic range) are not corrected according to the CFI to prevent overcorrection. In other embodiments, saturated and / or overexposed pixels may be compensated by the CFI.

[0038] Now with reference to the Fig. 5A - Fig. 5B, graphical representations of a diagonal reference are shown according to one or more embodiments. According to one embodiment, a diagonal reference may be used for a non-vertical correction. Referring first to Fig. 5A shows a pixel array 505 that includes a light source moving horizontally (e.g., from left to right) with respect to the sensor, causing a diagonal effect, shown as 510. Compensation based solely on vertical movement cannot correct for this type of movement by a light source on the array 505. Thus, in accordance with one or more principles of this document, movement of a light source may be compensated for by creating a diagonal reference. It should be understood that opposite movement of the image sensor and a light source, or any combination thereof, may also be compensated for by creating a diagonal reference, for example, from top left to bottom right.

[0039] In one embodiment, the diagonal reference 510 may be generated based on an upper OB row (TOB) 515 for a first row of the CCD. Each row in the array 505 may be shifted to the right (e.g., from top to bottom), for example, using interpolation. Alternatively, a diagonal reference 510 may be generated starting from the lower OB row (BOB) 520 and then shifted to the left for each row in the image (e.g., from bottom to top), for example, using interpolation. A reference compensation image (RCI) may then be generated by the processor of the imaging device corresponding to the diagonal reference 510.

[0040] According to another embodiment, two RCIs may be generated and then merged pixel by pixel, using a factor that gives more value to the compensation of the TOB 525 for rows closer to the top and a higher compensation to the BOB for rows closer to the bottom, BOB 530. The compensation factor may be calculated as follows: CF = BVt:α + BVb · (1 - α) Where α is a factor proportional to the position of the row (e.g., for the first row closest to the TOB 525, the value of α is approximately 1, and for the last row of the array 505, α is approximately 0). BVt refers to a black level for compensation at the TOB row 515 for a respective pixel, while BVb is the value for the BOB row 520 for a respective pixel. Accordingly, the compensation factor (CF) to be used with respect to a pixel of each line depends on its position in the sensor, both vertically and horizontally.In this way, the deficiencies of conventional methods and devices can be overcome. It should also be noted that while the diagonal reference 510 is being determined, compensation may be provided to pixels within the diagonal reference path and pixels that require compensation.

[0041] Now with reference to Fig. 5B, a movement of the light source is shown such that it not only moves horizontally but also approximates a CCD plane. According to one embodiment, the processor may provide stretching or upscaling for rows from which the RCI can be determined. Similarly, contraction or downscaling may be provided for TOB and BOB rows. For example, as shown in array 550, one or more of the TOB rows 565 and BOB rows 570 may be stretched or contracted. As shown in Fig. 5B, BOB lines 570 may be contracted to generate an RCI.

[0042] According to another embodiment, if two RCIs are produced for an array, a combined RCI can be generated. It should also be noted that other non-vertical movements possible for one or more light sources relative to the image sensor plane can be corrected analogously. In one embodiment, the processor can generate a CFI based on the RCI data and, pixel by pixel, subtract the corresponding CFI value from the pixel.

[0043] Now with reference to Fig. 6 shows a graphical representation of multiple overexposed pixels according to one embodiment. The graphical representation 600 includes overexposed pixel data 605 associated with a TOB line and overexposed image data associated with a BOB line 615. In certain cases, more than one light source may cause pixels to be overexposed. As shown in Fig. 6, overexposure of pixels may first be identified at positions 610-1, 610-2, and 610-3 of the TOB 605. Similarly, overexposure of pixels may be identified at positions 620-1, 620-2, and 620-3 of the BOB 615. Each light source may cause a different effect due to a different movement relative to an image sensor. Accordingly, diagonal references 630-1, 630-2, and 630-3 may be determined to provide correction to the pixels in the path of each of the light sources by a processor (e.g., processor 220) of the imaging device. Furthermore, operating in correction stripes ensures that the correction process is performed only on the stripes requiring correction, thereby reducing DRAM access and bandwidth, which also reduces power consumption for this operation.

[0044] Now with reference to Fig. 7 shows a process for image correction according to one or more embodiments. According to one embodiment, process 700 may be performed by the processor (e.g., processor 220) of an imaging device. Process 700 may be initiated by receiving image data at block 705. Image data received at block 705 may refer to image data received from a CCD (e.g., image sensor 205). At decision block 710, the processor may determine whether one or more pixels of an image array are overexposed. If the processor detects overexposed pixel data ("NO" path from decision block 710), the processor checks for additional image data at decision block 735.If the processor detects overexposed pixel data ("YES" path from decision block 710), the processor may compare pixel data to one or more previous rows at block 715 to determine if the overexposed pixels are in the same position (implying, for example, that only vertical movement is present). At decision block 720, the processor may determine if the overexposed pixel data is in the same position by comparing pixel data to one or more previous rows. If the overexposed pixel data is in the same row ("YES" path from decision block 720), the processor may perform a vertical correction at block 730. Alternatively, if the overexposed pixel data is not in the same row ("NO" path from decision block 720), the processor may perform a non-vertical correction at block 725.The non-vertical correction at block 725 may include multiple light source corrections. In certain embodiments, corrections may only be performed after an entire captured image frame has been stored in memory. As such, the lower and upper correction values ​​are known. It should also be understood that the correction may be based on identifying global and / or local movement of overexposed pixel data. At decision block 735, the processor may check for additional image data. If additional data exists ("YES" path from decision block 735), the processor may check for overexposed pixels at block 710. Alternatively, if no additional data exists ("NO" path from decision block 735), the correction process ends at block 740.

[0045] According to another embodiment, a temporal filter is added to address the noise associated with the correction line of the light source(s), especially when low light levels are also involved. However, such a temporal filter for reducing noise for low-level lights is problematic when used with overexposed pixels. Therefore, in areas where overexposure compensation is performed, the temporal filter for reducing noise for low-level lights is suspended and only reapplied for areas where no overexposure correction is performed.

[0046] Now with reference to Fig. 8 shows another process for image correction, which does not fall within the scope of the invention but serves to better understand the invention. The process 800 may be performed by a processor (e.g., processor 220) of an imaging device to provide artifact correction according to one or more embodiments. In one embodiment, the process 800 relates to the correction of one or more smear artifacts associated with one or more light sources in the image. In movie and / or preview modes, if the image contains a very bright light source, it may contaminate one or more vertical shift registers, causing pixels located below and above the light source to become brighter. This column of brighter pixels is referred to as "smear." If the light source is moving with respect to the camera (or vice versa), the smear artifact may not appear as a vertical column.Instead, it may appear as a diagonal column, or it may exhibit some curvature depending on the type of movement. Accordingly, process 800 can respond to both vertical and non-vertical movement.

[0047] The process 800 may be initiated by receiving sensor data (e.g., image sensor data) at block 805. In one embodiment, the sensor data may include image data associated with a pixel array output (e.g., pixel array 300) and smear-sensitive data, which may include one or more of optical black (OB) data and dummy line data. The processor may determine whether correction is needed based on the detection of one or more artifacts, such as smear. Accordingly, the processor may detect movement of one or more light sources associated with the artifact at block 810. Following detection of movement of one or more artifacts, the process 800 may characterize movement of the one or more light sources at block 815.In certain embodiments, a processor of the imaging device may determine motion characteristics associated with vertical and / or non-vertical motion. Characterizing light source motion may be based on smear-sensitive data (e.g., optical black (OB) data) such as top and bottom OB data. In certain embodiments, characterizing light source motion may be based on dummy line data and / or OB data. In another embodiment, motion may be described by a horizontal motion vector. If multiple light sources are detected, the calculation of a reference may be repeated for each light source, and then all references are combined to produce the final reference image. Characterizing motion may be based on current, past, and / or future image frames detected by the image sensor.

[0048] The processor may be configured to correct image data at block 820. In one embodiment, the correction of image data may be based on motion characteristics of one or more light sources, as well as OB data and / or dummy line data. Smear-sensitive data such as OB data and dummy line data may be processed prior to characterizing motion and / or correcting image data for spatial / temporal noise reduction, removal of defective pixels, etc. According to another embodiment, correcting the image data may include correcting a portion or correcting the entire image. At least a portion of the image data may be corrected by subtracting pixels associated with the reference image from the pixels associated with the image data.

[0049] According to another embodiment, the correction of the image data may be based on a reference. To correct the artifact, the processor may generate a reference, such as a reference pixel. For example, when performing correction of the image data on a pixel-by-pixel basis, a pixel may be corrected based on a pixel reference. Correcting on a pixel-by-pixel basis may involve calculating a reference pixel for each image pixel based on TOB, calculating another reference pixel based on BOB, and merging the two references into a final reference pixel. The pixel may then be corrected according to the reference pixel before moving on to another pixel. In this way, pixel corrections can be calculated on the fly on a pixel-by-pixel basis. The reference may be sensitive to vertical motion and non-vertical motion.

[0050] In another embodiment, reference images may be generated based on OB data and the motion characteristics of each light source to correct at least one artifact. Generating a reference image may include generating multiple reference images associated with multiple light sources, wherein the reference image is generated based on a combination (e.g., a weighted average) of the multiple reference images. The results of the reference image calculations may be summed to generate a single reference image. Alternatively to, or in combination with, calculating a reference image, process 800 may include generating a reference OB (RBOB and / or RTOB) row. In one embodiment, the reference image may include diagonal or vertical columns of overexposed pixels such that the diagonal columns represent non-vertical motion of a light source.According to a further embodiment, the reference image may be generated based on data associated with one or more dummy rows.

[0051] Now with reference to Fig. 9 is a simplified block diagram of a CCD array of the image sensor of Fig. 1 according to another embodiment. In certain embodiments, an image sensor may be configured to generate one or more dummy lines. A dummy line may refer to a line of image data generated by the sensor as a single-frame readout. For example, dummy line data may not actually be detected by sensors of the image sensor as physical sensor lines, such as image lines or OB lines. In contrast, dummy lines may be generated as part of a single-frame readout. Unlike OB data, dummy line data does not contain black level information (e.g., data detected when shielded from a light source). Dummy line data may be used as smear-sensitive data and may be used alone or in addition to OB data for correcting one or more artifacts.

[0052] The image sensor of Fig. 9 includes a pixel array 900 configured to detect incident light energy for one or more imaging applications. Shift register 905 can access pixel data. According to another embodiment, the image sensor may include one or more optically black pixels, shown as 910 and 915. However, in certain embodiments, it is understood that optically black pixels 910 and 915 are optional. As such, image data may be corrected based on dummy line data. As shown in Fig.9, the image sensor further includes dummy lines 920 and 925. Pixel data associated with pixel array 900, TOB 910 and BOB 915, and dummy lines 920 and 925 may be output from shift register 905, shown as 930. According to another embodiment, image artifact correction may be based on pixel values ​​associated with dummy lines 920 and 925. In certain embodiments, dummy line data may be used to generate one or more of a reference, a reference image, and a reference compensation image (RCI). It should be understood that dummy lines shown as 920 and 925 may refer to lines of image data generated by the sensor as a frame readout, but dummy lines 920 and 925 may not correspond to actual lines of a sensor array.

[0053] One or more embodiments described herein may refer to or concern OB data. However, it should be understood that references to one or more embodiments may be made analogously with reference to OB data and / or may use dummy row data (e.g., smear-sensitive dummy row data) in addition to, or alternatively to, OB data.

[0054] Although specific embodiments have been described and shown in the accompanying drawings, it should be understood that such embodiments are merely illustrative and not restrictive of the broad embodiments, and that the embodiments are not intended to be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art. Trademarks and copyrights referred to herein are the property of their respective owners.

Claims

[1] A method for correcting image data of an image sensor, comprising: Receiving (405) image data and smear-sensitive data for a first frame; Detecting (410) one or more overexposed pixels in the image data; Generating (415) a diagonal reference based on the one or more overexposed pixels and the smear-sensitive data; Generating (420) a reference compensation image, RCI - Reference Compensation Image, based on the diagonal reference estimate and at least a portion of the smear-sensitive data, wherein the smear-sensitive data includes at least optical black (OB) pixel data of the image sensor, wherein the RCI takes into account an acceleration with respect to time in at least one of three dimensions over at least two frames of the image data associated with the portion of the smear-sensitive data; Generating (425) a compensation factor image, CFI - Compensation Factor Image, based on the RCI and correcting (430) image data for one or more pixels based on the CFI. [2] The method of claim 1, wherein the one or more overexposed pixels are associated with one or more light sources in the first image frame. [3] The method of claim 1, wherein the diagonal reference is associated with a non-vertical image correction. [4] The method of claim 1, wherein the RCI is generated based on one or more of an upper reference OB line, RTOB - Reference Top Optical Black, a lower reference OB line, RBOB - Reference Bottom Optical Black, and dummy line data. [5] The method of claim 1, further comprising stretching and / or contracting OB lines to determine the RCI. [6] The method of claim 1, further comprising determining a plurality of reference compensation images, RCIs, associated with a plurality of light sources, wherein the CFI is based on a comparison of the plurality of reference compensation images. [7] The method of claim 6, wherein comparing the plurality of reference compensation images comprises a weighted comparison of the plurality of reference compensation images to determine the CFI. [8] The method of claim 1, wherein correcting image data based on the CFI comprises weighting pixel values ​​for one or more overexposed pixels. [9] Apparatus (200) for processing image data, the apparatus comprising: an image sensor (205) for detecting image data and smear-sensitive data for a first frame; a memory (225) and a processor (220), wherein the processor (220) is designed to Detecting one or more overexposed pixels in the image data; generating a diagonal reference based on the one or more overexposed pixels and the smear-sensitive data; Generating a reference compensation image, RCI, based on the diagonal reference estimate and at least a portion of the smear-sensitive data, wherein the smear-sensitive data includes at least OB pixel data of the image sensor, wherein the RCI takes into account an acceleration with respect to time in at least one of three dimensions across at least two frames of the image data associated with the portion of the smear-sensitive data; Generating a compensation factor image, CFI, based on the RCI and correcting image data for one or more pixels based on the CFI. [10] The device according to claim 9, wherein the image sensor is a charge-coupled device (CCD). [11] The apparatus of claim 9, wherein the one or more overexposed pixels are associated with one or more light sources in the first image frame. [12] The apparatus of claim 9, wherein the diagonal reference is associated with a non-vertical image correction. [13] The apparatus of claim 9, wherein the RCI is generated based on one or more of an upper reference OB line, RTOB - Reference Top Optical Black, a lower reference OB line, RBOB - Reference Bottom Optical Black, and dummy line data. [14] The apparatus of claim 9, wherein the processor is further configured to expand and / or contract OB lines to determine the RCI. [15] The apparatus of claim 9, wherein the processor is further configured to determine a plurality of reference compensation images, RCIs, associated with a plurality of light sources, wherein the CFI is based on a comparison of the plurality of reference compensation images. [16] The apparatus of claim 15, wherein comparing the plurality of reference compensation images comprises a weighted comparison of the plurality of reference compensation images to determine the CFI. [17] The apparatus of claim 9, wherein the processor is further configured to correct image data based on the CFI by weighting pixel values ​​for one or more overexposed pixels. [18] A method for correcting image data of an image sensor, comprising: Receiving (705) image data and smear-sensitive data for a first frame; detecting (710) one or more overexposed pixels in the image data; Determining, based on the one or more overexposed pixels, whether one or more of a vertical (730) and non-vertical (725) correction should be performed on the image data, wherein performing a non-vertical (725) correction comprises: generating (415) a diagonal reference based on the one or more overexposed pixels and the smear-sensitive data; Generating (420) a reference compensation image, RCI, based on the diagonal reference estimate and at least a portion of the smear-sensitive data, wherein the smear-sensitive data includes at least OB pixel data of the image sensor, wherein the RCI takes into account an acceleration with respect to time in at least one of three dimensions across at least two frames of the image data associated with the portion of the smear-sensitive data; Generating (425) a compensation factor image, CFI, based on the RCI and correcting (430) image data for one or more pixels based on the CFI. [19] The method of claim 18, wherein the one or more overexposed pixels are associated with one or more light sources in the first image frame. [20] The method of claim 18, wherein the RCI is generated based on one or more of an upper reference OB line, RTOB - Reference Top Optical Black, a lower reference OB line, RBOB - Reference Bottom Optical Black, and dummy line data. [21] The method of claim 18, further comprising stretching and / or contracting OB lines to determine the RCI. [22] The method of claim 18, further comprising determining a plurality of reference compensation images, RCIs, associated with a plurality of light sources, wherein the CFI is based on a comparison of the plurality of reference compensation images. [23] The method of claim 18, wherein comparing the plurality of reference compensation images comprises a weighted comparison of the plurality of reference compensation images to determine the CFI. [24] The method of claim 18, wherein vertical correction of the one or more pixels comprises comparing upper OB lines, TOB, and lower BOB lines, BOB.

Citation Information

Patent Citations

  • Electro-optical reconnaissance system with forward motion compensation

    US20020149674A1

  • Digital camera comprising smear removal function

    US20060274173A1

  • Activity-Based System and Method for Reducing Gain Imbalance in a Bayer Pattern and Digital Camera Employing the Same

    US20080231735A1

  • US12888296B2

  • WO61244974A1