Correction method and device for processing column fixed pattern noise of image sensor

Abstract

A correction device comprises a deviation value updating unit, a calculation unit and a gain extension unit. The offset value updating unit is used for generating a plurality of updated offset values according to a plurality of basic offset values, and each of the plurality of updated offset values corresponds to a column of pixel values in a plurality of columns of pixel values generated by an image sensor. The calculation unit is coupled to the offset value updating unit and is used for generating a plurality of column corrected pixel values according to the plurality of updated offset values and the plurality of column pixel values. The gain extension unit is coupled to the calculation unit, and is used for determining a compensation gain according to the plurality of updated offset values, and generating a plurality of column output pixel values according to the compensation gain and the plurality of column corrected pixel values.

Description

Technical Field

[0001] The present invention relates to image sensing, and in particular to a method and apparatus for correcting column fixed pattern noise of an image sensor and related image sensing devices. Background Art

[0002] Fixed pattern noise (FPN) is a common phenomenon in image sensors, mainly caused by process variations, temperature changes, and aging of electronic components. The specific manifestation of fixed pattern noise is: significant differences in brightness or color between certain fixed pixels and their adjacent pixels, and these differences are consistent between multiple image frames.

[0003] Column fixed pattern noise (CFPN) is a specific form of FPN, which is commonly present in image sensors using CMOS technology. The main cause of CFPN is the slight differences in the processes of the components in the readout circuit. These differences cause different degrees of voltage or digital value offsets during the readout of each column of pixels by the readout circuit, thereby forming vertical stripe patterns in the image. A common method to solve the CFPN problem is to measure the offset values of the readout circuit in the later manufacturing stage of the sensor. This includes accurately measuring the offset values of each readout circuit to obtain correction data, and storing the correction data in a correction device. During subsequent use, these correction data will be used to dynamically adjust the readout data of each column of pixels, thereby reducing or eliminating the influence of CFPN.

[0004] However, the existing CFPN correction methods encounter problems under certain conditions. For example, in high-brightness scenes, the pixel values are originally expected to be close to or reach saturation, but using the existing CFPN correction methods may overly or incorrectly reduce some pixel values to obtain non-saturated pixel values. This will result in unnatural stripes or spots in the image. Obviously, in this case, the existing CFPN correction methods have their limitations in use. Therefore, a more intelligent mechanism needs to be added to CFPN correction to adapt to different shooting conditions. Summary of the Invention

[0005] In view of the above, an object of the present invention is to provide a method and apparatus for correcting column fixed pattern noise (CFPN) of an image sensor. Embodiments of the present invention rely on a gain stretching mechanism to provide a compensation gain for CFPN correction. The compensation gain can dynamically adjust the corrected pixel values generated by the CFPN correction method, especially for pixel values close to or reaching saturation. By amplifying the corrected pixel values, embodiments of the present invention can effectively prevent the situation where the existing method of CFPN correction based on the offset value of the readout circuit excessively reduces the pixel values, thereby avoiding the appearance of unnatural stripes or spots in the image.

[0006] An embodiment of the present invention provides a method for correcting column fixed pattern noise of an image sensor, the correction method comprising: receiving a plurality of column pixel values generated by an image sensor; generating a plurality of updated offset values according to a plurality of basic offset values, wherein each of the plurality of updated offset values corresponds to one of the plurality of column pixel values; generating a plurality of column corrected pixel values according to the plurality of updated offset values and the plurality of column pixel values; determining a compensation gain according to the plurality of updated offset values; and generating a plurality of column output pixel values according to the compensation gain and the plurality of column corrected pixel values.

[0007] An embodiment of the present invention provides an apparatus for correcting column fixed pattern noise of an image sensor, the correction apparatus comprising: an offset value update unit, a calculation unit, and a gain stretching unit. The offset value update unit is configured to generate a plurality of updated offset values according to a plurality of basic offset values, wherein each of the plurality of updated offset values corresponds to one of the plurality of column pixel values generated by the image sensor. The calculation unit is coupled to the offset value update unit and is configured to generate a plurality of column corrected pixel values according to the plurality of updated offset values and the plurality of column pixel values. The gain stretching unit is coupled to the calculation unit and is configured to determine a compensation gain according to the plurality of updated offset values, and generate a plurality of column output pixel values according to the compensation gain and the plurality of column corrected pixel values.

[0008] An embodiment of the present invention provides an image sensing device, which includes an image sensor and a calibration device. The image sensor is used to provide a plurality of column pixel values. The calibration device is used to process the column fixed pattern noise of the image sensor and includes an offset value updating unit, a calculation unit, and a gain extension unit. The offset value updating unit is used to generate a plurality of updated offset values according to a plurality of basic offset values, wherein each of the plurality of updated offset values corresponds to one column pixel value among the plurality of column pixel values generated by the image sensor. The calculation unit is coupled to the offset value updating unit and is used to generate a plurality of column calibrated pixel values according to the plurality of updated offset values and the plurality of column pixel values. The gain extension unit is coupled to the calculation unit and is used to determine a compensation gain according to the plurality of updated offset values, and generate a plurality of column output pixel values according to the compensation gain and the plurality of column calibrated pixel values. Description of the Drawings

[0009] Figure 1 Schematic diagram showing the architecture of the calibration device and the corresponding image sensing device according to an embodiment of the present invention.

[0010] Figure 2 Schematic diagram showing the data structure of the basic offset values stored in the storage unit according to an embodiment of the present invention.

[0011] Figure 3 Schematic diagram showing the look-up table between the gain and the basic offset value correction factor according to an embodiment of the present invention.

[0012] Figure 4 Schematic diagram showing the architecture of the calibration device with a dithering unit according to an embodiment of the present invention.

[0013] Figure 5 Flowchart showing the calibration method for processing the column fixed pattern noise of the image sensor according to an embodiment of the present invention.

[0014] Description of the Reference Numerals:

[0015] 10 Image sensing device

[0016] 100 Calibration device

[0017] 110 Offset value updating unit

[0018] 120 Calculation unit

[0019] 130 Gain extension unit

[0020] 140 Storage unit

[0021] 150 Dithering unit

[0022] 200 Image sensor

[0023] 210 Pixel Array

[0024] 220_1 to 220_K Readout Circuits

[0025] Steps S110 to S150 Specific Embodiment

[0026] In the following text, many specific details are described to provide the reader with a thorough understanding of the embodiments of the present invention. However, those skilled in the art will understand how to implement the present invention in the absence of one or more specific details, or by using other methods, elements, or materials, etc. In other cases, well-known structures, materials, or operations will not be shown or described in detail to avoid obscuring the core concepts of the present invention.

[0027] As used herein, "an embodiment" means that the specific features, structures, or characteristics described in that embodiment may be included in at least one embodiment of the present invention. Therefore, the phrase "in an embodiment" that appears throughout this specification does not necessarily refer to the same embodiment. Additionally, the aforementioned specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] Please refer to Figure 1 , which shows a calibration device and a corresponding image sensing device according to an embodiment of the present invention. As shown in the figure, the calibration device 100 is used to process the column fixed pattern noise (CFPN) of the image sensor 200 in the image sensing device 10. The calibration device 100 includes an offset value update unit 110, a calculation unit 120, a gain extension unit 130, and a storage unit 140. On the other hand, the image sensor 200 includes a pixel array 210 and readout circuits 220_1 to 220_K. The pixel array 210 includes a plurality of pixels arranged in a matrix form, which may include M columns and N rows of pixels (i.e., M×N pixels).

[0029] The readout circuits 220_1 to 220_K are used to convert the charges sensed by the pixels on the pixel array 210 into a plurality of (digital) pixel values, that is, grayscale values. Generally speaking, each of the readout circuits 220_1 to 220_K is coupled to one or more columns of pixels in the pixel array 210 and correspondingly reads out one or more columns of pixel values from the pixel array 210. Note that although in the above embodiment, every four columns of pixels in the pixel array 210 share one readout circuit, this is not a limitation of the present invention. Furthermore, through the readout circuits 220_1 to 220_K, a plurality of column pixel values PV_1_1 to PV_1_N (the first column), PV_2_1 to PV_2_N (the second column), …, and PV_M_1 to PV_M_N (the Mth column) will be output to the calibration device 100 and corrected by the calibration device 100.

[0030] Based on a plurality of basic offset values, an offset value updating unit 110 is configured to generate a plurality of updated offset values (M updated offset values in this embodiment) respectively corresponding to a plurality of column pixel values PV_1_1 to PV_1_N (the first column), PV_2_1 to PV_2_N (the second column), …, and PV_M_1 to PV_M_N (the Mth column). Specifically, the offset value updating unit 110 is configured to read a plurality of basic offset values from a storage unit 140, and obtain a plurality of updated offset values by selecting specific basic offset values from the plurality of basic offset values.

[0031] Please refer to Figure 2 , which shows the data structure of the basic offset values in the storage unit 140 in an embodiment of the present invention. As shown in the figure, the basic offset values BO_1_1 to BO_1_L, BO_2_1 to BO_2_L, …, and BO_K_1 to BO_K_L in the storage unit 140 are divided into a plurality of groups Group_1 to Group_K. Each of the basic offset value groups Group_1 to Group_K corresponds to the CFPN correction data of a specific one of the readout circuits 220_1 - 220_K. The CFPN correction data of the readout circuits 220_1 to 220_K can be obtained through a correction data collection program for collecting the offset values of the readout circuits. During the correction data collection program, the image sensor 200 acquires a test image with a uniform brightness level under a specific operating environment (i.e., specific gain, aperture, and / or shutter speed settings). Subsequently, the data read out by the readout circuits 220_1 to 220_K of the image sensor is compared with the pixel data of these test images, so as to determine the offset values of each of the readout circuits 220_1 to 220_K, which represent the deviation from the expected output under a given operating environment. Then, the above process is repeated under different operating environments, so as to comprehensively collect the offset values of each of the readout circuits 220_1 to 220_K in a series of scenarios. Through the above program, the basic offset values BO_1_1 to BO_1_L, BO_2_1 to BO_2_L, …, and BO_K_1 to BO_K_L can be determined.

[0032] In addition, each group of basic offset value groups Group_1 to Group_K contains a plurality of basic offset values corresponding to various operating environments of the image sensor 200. Taking the basic offset values BO_1_1 to BO_1_L included in the basic offset value group Group_1 as an example, the basic offset value BO_1_1 may correspond to the operating environment OP1 of the first gain (i.e., sensitivity setting) of the image sensor 200. This means that when the image sensor 200 operates with the first gain, the basic offset value BO_1_1 can be used to reduce the CFPN of the readout circuit 220_1 of the image sensor 200. In addition, the basic offset value BO_1_4 may correspond to the operating environment OP4 of the first shutter speed (i.e., exposure setting) of the image sensor 200. This means that when the image sensor 200 operates with the first shutter speed, the basic offset value BO_1_4 can be used to reduce the CFPN of the readout circuit 220_1 of the image sensor 200. In different embodiments of the present invention, the various operating environments OP1 to OPL may involve a combination of at least one or more of the gain, aperture, shutter speed, and output resolution settings of the image sensor 200. For example, the basic offset value BO_1_5 may correspond to the operating environment OP5 including a combination of multiple settings of the image sensor 200, which is composed of the first gain, the first shutter speed, and the first aperture.

[0033] In view of this, when processing the pixel values of a specific column, the offset value update unit 110 first selects a corresponding group of basic offset values from the basic offset value groups Group_1 to Group_K. For example, if the offset value update unit 110 is processing a column of pixel values read out by the readout circuit 220_2, the offset value update unit 110 can select the basic offset value group Group_2. In addition, in the pixel-binning readout method, since the values read out by more than one readout circuit 220_1 to 220_K are used to obtain the single-column pixel values. Therefore, the offset value update unit 110 will use multiple basic offset value groups Group_1 to Group_K to generate an updated offset value to process the single-column pixel values obtained by using the pixel-binning readout method.

[0034] Once one or more of the basic offset value groups Group_1 to Group_K are selected (for the pixel merging readout mode), the offset value update unit 110 further determines the current operating environment of the image sensor 200 (i.e., gain, aperture, and / or shutter speed settings). Based on the determined operating environment, the offset value update unit 110 selects a corresponding basic offset value from the selected basic offset value group. For example, after selecting the basic offset value group Group_2, the offset value update unit 110 will select one from the basic offset values BO_2_1 to BO_2_L included in the basic offset value group Group_2 according to the operating environment.

[0035] In some embodiments, due to the limited capacity of the storage unit 140, it is not cost-effective and impractical to store the basic offset values corresponding to all possible operating environments of the image sensor 200. Therefore, the basic offset values BO_1_1 to BO_1_L, BO_2_1 to BO_2_L... and BO_K_1 to BO_K_L stored in the storage unit 140 may only correspond to a limited number of operating environments of the image sensor 200. In other words, the selected basic offset value may not exactly match the current operating environment of the image sensor 200.

[0036] Therefore, the offset value update unit 110 may need to modify the selected basic offset value through a mapping relationship. Please refer to Figure 3 for further understanding. Figure 3 A lookup table in an embodiment of the present invention is shown. This lookup table indicates the correction factors corresponding to the possible gains of the image sensor 200. For example, the basic offset values BO_2_1 to BO_2_L included in the basic offset value group Group_2 may only relate to the operating environment of the image sensor 200 at gain A and have nothing to do with gains B to F. Through the Figure 3 shown lookup table, the offset value update unit 110 modifies the basic offset values BO_2_1 to BO_2_L according to the correction factor related to the actual operating gain of the image sensor 200, so as to obtain the updated offset values for subsequent CFPN correction. Note that if the basic offset values stored in the storage unit 140 can cover all operating environments of the image sensor 200, the above modification is not necessary.

[0037] Through the above correction process, the offset value update unit 110 will generate updated offset values (which may or may not be processed by the correction factor) for correcting each column of pixel values. Accordingly, the calculation unit 120 is used to add or subtract an updated offset value from the corresponding column of pixel values to generate multiple column-corrected pixel values.

[0038] In some embodiments, the calibration device 100 may also include a dithering unit 150. Please refer to Figure 4 for further understanding. The dithering unit 150 is used to control the calculation unit 120 to perform CFPN calibration according to a plurality of updated offset values and a plurality of column pixel values. Specifically, the basic offset values stored in the storage unit 140 and the updated offset values generated by the offset value update unit 110 may be non-integers, while the pixel values generated by the image sensor 200 may be integers. The basic offset values stored in the storage unit 140 and the updated offset values can be represented by (n + m + 1) bits, where n bits can be used to represent the integer part of each basic offset value and each updated offset value, m bits can be used to represent the fractional part of each basic offset value and each updated offset value, and 1 bit is used to represent the sign (positive or negative) of each basic offset value and each updated offset value. In addition, the integer part of each pixel value generated by the image sensor 200, each calibrated pixel value generated by the calculation unit 120, each output pixel value generated by the gain extension unit 130, and each dithering value applied to the calculation unit 120 can be represented by n bits, and 1 bit is used to represent the sign (positive or negative) of each dithering value applied to the calculation unit 120.

[0039] When the calculation unit 120 applies the non-integer updated offset value to a column of pixel values to generate a column of calibrated pixel values. Among them, according to the fractional part of the non-integer updated offset value, the dithering unit 150 will control the calculation unit 120 to add one or more dithering values only to a part of the column of pixel values; or, subtract one or more dithering values only from a part of the column of pixel values. For example, assuming that the updated offset value generated by the offset value update unit 110 for a column of pixel values is "+0.5" (i.e., a non-integer updated offset value), the dithering unit 150 will control the calculation unit 120 to add "1" (i.e., the dithering value) to 50% of the pixel values in the column of pixel values and add "0" to the other 50% of the pixel values in the column of pixel values. In addition, assuming that the updated offset value generated by the offset value update unit 110 for a column of pixel values is "-1.75" (i.e., a non-integer updated offset value), the dithering unit 150 will control the calculation unit 120 to subtract "2" (i.e., the dithering value) from 75% of the pixel values in the column of pixel values and subtract "1" (i.e., the dithering value) from the other 25% of the pixel values in the column of pixel values. However, if all the updated offset values generated by the offset value update unit 110 are integers, the dithering unit 150 may not be necessary.

[0040] After the computing unit 120 generates multiple column-corrected pixel values, the gain extension unit 130 determines a compensation gain to compensate for the multiple column-corrected pixel values. Specifically, the gain extension unit 130 determines the compensation gain according to the maximum value among multiple updated offset values. In one embodiment, the compensation gain can be determined by (PV_Max) / (PV_Max - UO_Max). Wherein, the value PV_Max is the maximum possible pixel value generated by the readout circuits 220_1 - 220_K, and the value UO_Max is the maximum value among the multiple updated offset values generated by the offset value update unit 110. The maximum possible pixel value PV_Max may vary with the bit depth of the image sensor 200. For an image sensor 200 with a 10-bit depth, the maximum possible pixel value PV_Max can be 1023 (i.e., 2 10 - 1). The gain extension unit 130 generates multiple column output pixel values OPV_1_1 to OPV_1_N (corresponding to the first column), OPV_2_1 to OPV_2_N (corresponding to the second column), …, and OPV_M_1 to OPV_M_N (corresponding to the Mth column) by multiplying the multiple column-corrected pixel values by the compensation gain. Therefore, the output pixel values OPV_1_1 to OPV_1_N (corresponding to the first column), OPV_2_1 to OPV_2_N (corresponding to the second column), …, and OPV_M_1 to OPV_M_N (corresponding to the Mth column) can be regarded as pixel values not affected by CFPN.

[0041] Figure 5 A flowchart showing a method for correcting CFPN of a processing image sensor according to an embodiment of the present invention. As shown, the correction method of the present invention includes the following simplified processes:

[0042] S110: Receive multiple column pixel values generated by an image sensor;

[0043] S120: Generate multiple updated offset values according to multiple basic offset values, wherein each of the multiple updated offset values corresponds to a column pixel value among the multiple column pixel values;

[0044] S130: Generate multiple column-corrected pixel values according to the multiple updated offset values and the multiple column pixel values;

[0045] S140: Determine a compensation gain according to the multiple updated offset values; and

[0046] S150: Generate multiple column output pixel values according to the compensation gain and the multiple column-corrected pixel values.

[0047] Since the principles and specific details of the above steps have been described in detail in previous embodiments, they will not be elaborated here. It should be noted that the flow of the above correction method can be improved by adding other additional steps or making appropriate modifications and adjustments to better improve the image quality of the image sensor and further enhance the effect and efficiency of CFPN correction.

[0048] In summary, the present invention proposes an innovative CFPN correction method and related device. The present invention utilizes a sophisticated gain extension mechanism to provide a compensation gain for dynamic compensation of CFPN correction. This compensation gain can fine-tune the corrected pixel values generated by CFPN correction, with particular emphasis on compensating pixel values that are close to or reach the saturation level. Since existing CFPN correction methods rely on the offset values of the readout circuit for correction, it may lead to erroneously reducing pixel values, ultimately generating artificial and visually disruptive elements in the output image, such as stripes or spots. The gain extension mechanism of the present invention effectively avoids this problem.

[0049] Embodiments of the present invention can be specifically implemented as devices, methods, or computer program products. Accordingly, embodiments of the present invention can take the form of entities implemented entirely in hardware, entities implemented entirely in software (including firmware, resident software, microcode, etc.), or entities combining software and hardware aspects, which can be collectively referred to as "modules" or "systems". In addition, embodiments of the present invention can take the form of a computer program product embodied in any tangible expression medium and having computer-usable program code embodied in the medium. In terms of hardware, the present invention can be implemented by applying any of the following technologies or related combinations: the individual operational logic of logic gates capable of performing logical functions according to data signals, and application-specific integrated circuits (ASICs), programmable gate arrays (PGAs), or field-programmable gate arrays (FPGAs) with appropriate combinational logic.

[0050] Flowcharts and block diagrams illustrate the architecture, functionality, and operation of systems, methods, and computer program products according to various possible embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing a particular logical function. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a special-purpose hardware-based system, or a combination of special hardware and computer program instructions. These computer program instructions can be stored in a readable computer medium to direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the readable computer medium result in an apparatus that implements the functions / operations specified in the blocks or combinations of blocks of the flowchart and / or block diagram.

[0051] The above are only the preferred embodiments of the present invention, and all equivalent changes and modifications made according to the claims of the present invention shall fall within the scope of the present invention.

Claims

1. A method for correcting column fixed pattern noise of an image sensor, comprising: Receiving a plurality of column pixel values generated by an image sensor; Generating a plurality of updated offset values according to a plurality of basic offset values, wherein each of the plurality of updated offset values corresponds to a column of pixel values among the plurality of column pixel values; Generating a plurality of column corrected pixel values according to the plurality of updated offset values and the plurality of column pixel values; Determining a compensation gain according to the plurality of updated offset values; and Generating a plurality of column output pixel values according to the compensation gain and the plurality of column corrected pixel values.

2. The correction method according to claim 1, wherein the step of generating the plurality of updated offset values comprises: For each column of pixel values: Selecting a specific group of basic offset values for the column of pixel values from the plurality of basic offset values according to a label of a readout circuit for reading out the column of pixel values; and Determining a selected basic offset value from the specific group of basic offset values of the column of pixel values according to at least one operating environment of the image sensor; and Generating the plurality of updated offset values according to the selected basic offset value of each column of pixel values.

3. The correction method according to claim 2, wherein the step of generating the plurality of updated offset values comprises: Determining a correction factor according to the at least one operating environment of the image sensor; and Generating each of the plurality of updated offset values according to the correction factor and the selected basic offset value of each column of pixel values.

4. The correction method according to claim 2, wherein the at least one operating environment corresponds to at least one of a gain, an aperture, a shutter speed, and an output resolution setting of the image sensor.

5. The correction method according to claim 1, wherein the step of generating the plurality of column corrected pixel values comprises: Determining one or more dither values corresponding to each of the plurality of updated offset values according to a fractional part of each of the plurality of updated offset values; and Generating a corrected pixel value for each column according to each column of pixel values and the one or more corresponding dither values.

6. The correction method according to claim 5, wherein the step of generating a corrected pixel value for each column comprises: Applying the one or more dither values only to a part of the pixel values of each column of pixel values according to the fractional part of each of the plurality of updated offset values.

7. The correction method according to claim 1, wherein the step of determining the compensation gain comprises: Determining the compensation gain according to a maximum value among the plurality of updated offset values and a maximum possible pixel value generated by the image sensor.

8. An apparatus for correcting column fixed pattern noise of an image sensor, comprising: An offset value updating unit for generating a plurality of updated offset values according to a plurality of basic offset values, wherein each of the plurality of updated offset values corresponds to a column of pixel values among the plurality of column pixel values generated by the image sensor; A calculation unit coupled to the offset value updating unit for generating a plurality of column corrected pixel values according to the plurality of updated offset values and the plurality of column pixel values; And A gain extension unit, coupled to the computing unit, is configured to determine a compensation gain according to the plurality of updated offset values, and generate a plurality of column output pixel values according to the compensation gain and the plurality of column-corrected pixel values.

9. The correction device according to claim 8, wherein the offset value updating unit is configured to: For each column of pixel values: According to a label of a readout circuit that reads out the column of pixel values, select a specific group of basic offset values from the plurality of basic offset values for the column of pixel values; and According to at least one operating environment of the image sensor, determine a selected basic offset value from the specific group of basic offset values of the column of pixel values; and Generate the plurality of updated offset values according to the selected basic offset value of each column of pixel values.

10. The correction device according to claim 9, wherein the offset value updating unit is configured to: Determine a correction factor according to the at least one operating environment of the image sensor; and Generate each of the plurality of updated offset values according to the correction factor and the selected basic offset value of each column of pixel values.

11. The correction device according to claim 9, wherein the at least one operating environment corresponds to at least one of the gain, aperture, shutter speed, and output resolution setting of the image sensor.

12. The correction device according to claim 8, further comprising: A dithering unit, coupled to the computing unit and the offset value updating unit, is configured to determine one or more dithering values corresponding to each of the plurality of updated offset values according to a fractional part of each of the plurality of updated offset values, and control the computing unit to generate each column of corrected pixel values according to each column of pixel values and the one or more corresponding dithering values.

13. The correction device according to claim 12, wherein the dithering unit is configured to apply the one or more dithering values only to a part of the pixel values of each column of pixel values according to the fractional part of each of the plurality of updated offset values.

14. The correction device according to claim 8, wherein the gain extension unit is configured to determine the compensation gain according to a maximum value among the plurality of updated offset values and a maximum possible pixel value generated by the image sensor.

Images