Image sensing system and method of operating the same

Abstract

The present disclosure relates to an image sensing system and an operation method thereof. An image sensing system is disclosed, the image sensing system comprising: an image sensor configured to generate a first actual brightness value corresponding to a captured image; and an image processor configured to compensate the first actual brightness value for each brightness period in a single frame period based on compensation information corresponding to a linear error.

Description

Technical Field

[0001] Various embodiments of the present disclosure relate to semiconductor design technology, and more particularly, to an image sensing system and an operating method thereof. Background Art

[0002] An image sensing system is a device that captures an image using the properties of a semiconductor that reacts to light. Image sensing systems are broadly classified into charge coupled device (CCD) image sensing systems and complementary metal oxide semiconductor (CMOS) image sensing systems. Recently, CMOS image sensing systems have become widely used because they allow both analog control circuits and digital control circuits to be directly implemented on a single integrated circuit (IC). Summary of the invention

[0003] Various embodiments of the present disclosure are directed to an image sensing system having a linear sensitivity characteristic and a method of operating the image sensing system.

[0004] In addition, various embodiments of the present disclosure are directed to an image sensing system for compensating for sensitivity characteristics using a simple calculation method and an operating method of the image sensing system.

[0005] According to an embodiment of the present disclosure, an image sensing system may include: an image sensor configured to generate a first actual brightness value corresponding to a captured image; and an image processor configured to compensate the first actual brightness value for each brightness period in a single frame period based on compensation information corresponding to a linear error.

[0006] According to an embodiment of the present disclosure, an image sensing system may include: a compensator configured to determine each brightness period and to compensate differently for a first actual brightness value generated by an image sensor for each brightness period based on a slope difference between an actual slope representing a relationship between an exposure time and an actual brightness value and a reference slope representing a relationship between an exposure time and a reference brightness value; and a controller configured to calculate the slope difference for each brightness period based on a second actual brightness value generated by the image sensor.

[0007] According to an embodiment of the present disclosure, an image sensing system may include: an image sensor, which is configured to generate a first actual brightness value and a second actual brightness value corresponding to a captured image during a first time period and a second time period; a controller, which is configured to determine an actual slope based on an exposure time and the second actual brightness value, determine a reference slope based on the exposure time and the actual slope, and determine a slope difference between the reference slope and the actual slope; and a compensator, which is configured to divide the slope difference into multiple linear functions of multiple brightness time periods and use a corresponding linear function among the multiple linear functions to compensate for the first actual brightness value of each brightness time period.

[0008] According to an embodiment of the present disclosure, an operating method of an image sensing system may include the following steps: generating one or more test images; calculating an actual slope representing the relationship between exposure time and a second actual brightness value and a reference slope representing the relationship between some exposure times and some second actual brightness values ​​based on a second actual brightness value corresponding to the test image, and generating a compensation function based on a slope difference between the actual slope and the reference slope; and differently compensating the first actual brightness value of each output period based on the compensation function.

[0009] According to an embodiment of the present disclosure, an operating method of an image sensing system may include the following steps: generating compensation information representing the sensitivity of an image sensor during a first time period; and compensating a first actual brightness value generated by the image sensor for each brightness period based on the compensation information during a second time period after the first time period.

[0010] The compensation information may represent a slope difference between an actual slope and a reference slope related to a linear error for each brightness period.

[0011] The slope difference may be defined as a linear function of each brightness period.

[0012] The actual slope may represent a relationship between the exposure time and the second actual brightness value, and the second actual brightness value may be generated by the image sensor during the test mode period.

[0013] The reference slope may represent a relationship between some exposure times and some second actual brightness values. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram illustrating an image sensing system according to an embodiment of the present disclosure.

[0015] Figure 2 The present invention is shown in the embodiment of the present invention. Figure 1 Block diagram of the image sensor shown.

[0016] Figure 3 The present invention is shown in the embodiment of the present invention. Figure 2 A diagram of an example of a pixel array is shown.

[0017] Figure 4 The present invention is shown in the embodiment of the present invention. Figure 1 A block diagram of the image processor is shown.

[0018] Figure 5 The present invention is shown in the embodiment of the present invention. Figure 4 The block diagram of the controller is shown.

[0019] Figure 6 The present invention is shown in the embodiment of the present invention. Figure 4 The block diagram of the compensator is shown in FIG.

[0020] Figure 7 The present invention is shown in the embodiment of the present invention. Figure 1 Flowchart of the operation of the image sensing system shown.

[0021] Figure 8 The embodiment according to the present disclosure is shown in more detail. Figure 7 The flowchart of generating compensation information is shown.

[0022] Fig. 9 is a graph showing an actual slope and a reference slope according to an embodiment of the present disclosure.

[0023] Fig.10 is a graph showing the slope difference according to an embodiment of the present disclosure.

[0024] Fig.11 The embodiment according to the present disclosure is shown in more detail. Figure 7 The flowchart of compensating the actual brightness value is shown. DETAILED DESCRIPTION

[0025] Various embodiments of the present disclosure are described below with reference to the accompanying drawings in order to describe the present disclosure in detail so that a person having ordinary skills in the art to which the present disclosure belongs can easily implement the technical spirit of the present disclosure.

[0026] It will be understood that when an element is referred to as being "connected to" or "coupled to" another element, the element may be directly connected to or coupled to the other element, or electrically connected to or coupled to the other element, with one or more elements interposed therebetween. In addition, it will be understood that, unless otherwise mentioned, the terms "include" and "comprise" when used in this specification do not exclude the presence of one or more other elements, but may also include or have one or more other elements. In the description throughout the specification, some components are described in the singular, but the present disclosure is not limited thereto, and it will be understood that components may be formed in the plural.

[0027] Figure 1is a block diagram illustrating an image sensing system according to an embodiment of the present disclosure.

[0028] Reference Figure 1 , the image sensing system may include an image sensor 100 and an image processor 200 .

[0029] The image sensor 100 may generate an actual brightness value DPXs corresponding to a captured image. The image sensor 100 may generate a first actual brightness value DPX1s corresponding to an actual image during a frame period, and generate a second actual brightness value DPX2s corresponding to a specific image during a specific period. For example, the specific period, which is a period before the frame period, may be a test mode period. The specific image may include one or more test images generated during the test mode period.

[0030] The image processor 200 may generate or calculate compensation information corresponding to the linear error based on the second actual brightness value DPX2s. In addition, the image processor 200 may compensate the first actual brightness value DPX1s differently for each brightness period or each output period based on the compensation information. The linear error may refer to the nonlinear degree of the actual brightness value relative to the linearly increased exposure time. The compensation information as information for correcting the linear error may refer to the slope difference DS indicated for each brightness period. Figure 4 ) describes the slope difference DS.

[0031] Figure 2 The present invention is shown in the embodiment of the present invention. Figure 1 A block diagram of an image sensor 100 is shown.

[0032] Reference Figure 2 , the image sensor 100 may include a pixel array 110 and a signal converter 120 .

[0033] The pixel array 110 may include a plurality of pixels (not shown) arranged in row and column directions. The pixel array 110 may generate pixel signals VPXs for each row. For example, the pixel array 110 may generate pixel signals VPXs from pixels arranged in the first row during a first row period of a single frame period, and may generate pixel signals VPXs from pixels arranged in the nth row during an nth row period, where "n" is an integer greater than 2. Each pixel signal VPXs may be an analog pixel signal.

[0034] The signal converter 120 may convert the analog pixel signal VPXs into a digital brightness value DPXs. For example, the signal converter 120 may include an analog-to-digital converter.

[0035] Figure 3 The present invention is shown in the embodiment of the present invention. Figure 2 A diagram of an example of a pixel array 110 is shown.

[0036] Reference Figure 3 , the pixel array 110 may be arranged in a predetermined pattern. For example, the pixel array 110 may be arranged in a Bayer pattern. The Bayer pattern may be composed of repeating units each having 2×2 pixels. In each unit, two pixels Gb and Gr each having a green filter may be arranged diagonally facing each other at their corners, and a pixel B having a blue filter and a pixel R having a red filter may be arranged at their other corners. The four pixels B, Gb, Gr, and R are not necessarily limited to Figure 3 The arrangement shown may instead be arranged differently according to the Bayer pattern described above.

[0037] Figure 4 The present invention is shown in the embodiment of the present invention. Figure 1 A block diagram of an image processor 200 is shown.

[0038] Reference Figure 4 , the image processor 200 may include a controller 210 , a storage unit 220 , and a compensator 230 .

[0039] The controller 210 may calculate the slope difference DS of each brightness period based on the second actual brightness value DPX2s. The slope difference DS may refer to the deviation of each brightness period between the actual slope RS and the reference slope IS related to the linear error. The actual slope RS and the reference slope IS may be calculated based on the exposure time and the second actual brightness value DPX2s (refer to Figure 5 ).

[0040] The storage unit 220 may store the slope difference DS calculated for each brightness period. For example, the storage unit 220 may include a register or the like.

[0041] The compensator 230 may compensate the first actual brightness value DPX1s of each brightness period based on the stored slope difference DS′ output from the storage part 220 .

[0042] Figure 5 The present invention is shown in the embodiment of the present invention. Figure 4 A block diagram of the controller 210 is shown.

[0043] Reference Figure 5 , the controller 210 may include a first calculator 211, a second calculator 213, and a third calculator 215. The first calculator 211, the second calculator 213, and the third calculator 215 include all circuits, systems, software, firmware, and devices required for their respective operations and functions.

[0044] The first calculator 211 may calculate an actual slope RS indicating a relationship between the exposure time and the second actual luminance value DPX2s. For example, the first calculator 211 may represent a unit increment of the second actual luminance value DPX2s relative to a unit increment of the exposure time as the actual slope RS.

[0045] The second calculator 213 may calculate a reference slope IS indicating a relationship between the exposure time and the reference brightness value. The reference brightness value may be a second actual brightness value included in some brightness periods among the second actual brightness values ​​DPX2s. In this embodiment, some second actual brightness values ​​DPX2s that may represent a linear output for a linear input may be used as reference brightness values ​​to compensate for the first actual brightness value DPX1s. The reference slope IS may be calculated according to the following “Formula 1”.

[0046] [Formula 1]

[0047]

[0048] Here, "PXCODE1" may refer to the maximum brightness value among the second actual brightness values ​​included in the some brightness periods, "PXCODE2" may refer to the minimum brightness value among the second actual brightness values ​​included in the some brightness periods, "INTGTIME1" may refer to the exposure time corresponding to the maximum brightness value, and "INTGTIME2" may refer to the exposure time corresponding to the minimum brightness value. That is, the second calculator 213 may calculate the reference slope IS by dividing the increment of the second actual brightness value corresponding to the some brightness periods by the increment of the exposure time corresponding to the some brightness periods.

[0049] The third calculator 215 may calculate the slope difference DS of each brightness period based on the actual slope RS and the reference slope IS. For example, the third calculator 215 may calculate the deviation of the brightness value between the actual slope RS and the reference slope IS, and represent the slope difference DS of each brightness period as a compensation function based on the deviation. The compensation function may include a linear function. When the deviation is divided into the first to fourth brightness periods, the first to fourth linear functions corresponding to the first to fourth brightness periods are calculated, and the first to fourth linear functions may be defined as the following "Formulas 2 to 5", respectively.

[0050] [Formula 2]

[0051] DS=a1x+b1

[0052] Here, "x" may refer to a brightness value, and each of "a1" and "b1" may refer to a constant corresponding to the first linear function. However, each of "a1" and "b1" may be a real number.

[0053] [Formula 3]

[0054] DS=a2x+b2

[0055] Here, "x" may refer to a brightness value, and each of "a2" and "b2" may refer to a constant corresponding to the second linear function. However, each of "a2" and "b2" may be a real number.

[0056] [Formula 4]

[0057] DS=a3x+b3

[0058] Here, "x" may refer to a brightness value, and each of "a3" and "b3" may refer to a constant corresponding to the third linear function. However, each of "a3" and "b3" may be a real number.

[0059] [Formula 5]

[0060] DS=a4x+b4

[0061] Here, "x" may refer to a brightness value, and each of "a4" and "b4" may refer to a constant corresponding to the fourth linear function. However, each of "a4" and "b4" may be a real number.

[0062] Figure 6 The present invention is shown in the embodiment of the present invention. Figure 4 A block diagram of the compensator 230 is shown.

[0063] Reference Figure 6 , the compensator 230 may include a brightness period determiner 231 and a corrector 233. The compensator 230, determiner 231, and corrector 233 include all circuits, systems, software, firmware, and devices required for their respective operations and functions.

[0064] The brightness period determiner 231 may sort the first actual brightness values ​​DPX1s of the respective brightness periods. For example, the brightness period determiner 231 may determine to which brightness period each first actual brightness value DPX1s belongs, and generate a result signal AA corresponding to the determination result.

[0065] The corrector 233 may correct the first actual brightness value DPX1s based on the result signal AA and the slope difference DS. For example, the corrector 233 may correct the first actual brightness value DPX1s according to the following “Formula 6”.

[0066] [Formula 6]

[0067] y=x+a#x+b#

[0068] Here, "y" may refer to the corrected first actual brightness value, "x" may refer to the first actual brightness value, and "a#" and "b#" may refer to constants calculated for each brightness period. For example, the corrector 233 may include a multiplier and an adder.

[0069] Hereinafter, the operation of the image sensing system having the above-described configuration according to the embodiment is described.

[0070] Figure 7 The present invention is shown in the embodiment of the present invention. Figure 1 Flowchart of the operation of the image sensing system shown.

[0071] Reference Figure 7 , the operating method of the image sensing system may include an operation S100 of generating compensation information indicating the sensitivity of the image sensor 100 during a first period, and an operation S200 of compensating the first actual brightness value DPXs generated by the image sensor 100 for each brightness period based on the compensation information during a second period after the first period. The compensation information may indicate a slope difference between an actual slope RS related to a linear error and a reference slope IS for each brightness period.

[0072] The sensitivity of the image sensor 100 may comply with the standard of the International Organization for Standardization and is related to the light source, gain, integration time, etc. The first period may be a test mode period, and the second period may be a frame period.

[0073] Figure 8 The present invention is shown in the embodiment of the present invention. Figure 7 Flowchart of operation S100 of generating compensation information is shown.

[0074] Reference Figure 8 In operation S101, the image sensor 100 may generate a second actual brightness value DPX2s in a predetermined environment. The predetermined environment may include the same light source conditions and conditions in which the exposure time is increased in a uniform unit. For example, the image sensor 100 may capture a plurality of images while increasing the exposure time in a uniform unit under the same light source conditions, and generate a second actual brightness value DPX2s corresponding to each of the plurality of captured images.

[0075] In operation S103, the image processor 200 may generate an actual slope RS and a reference slope IS based on the second actual brightness value DPX2s. The actual slope RS may refer to an average brightness value of the second actual brightness value DPX2s relative to the exposure time. The reference slope IS may be calculated based on a portion of the actual slope RS. In some embodiments, the image processor 200 may sort the second actual brightness value DPX2s according to the first to third color channels or the first to fourth color channels corresponding to the pixels Gb and Gr, each having a green filter, the pixel B, and the pixel R, each having a red filter. In addition, the image processor 200 may generate the first to third actual slopes and the first to third reference slopes or the first to fourth actual slopes and the first to fourth reference slopes based on the second actual brightness values ​​sorted for each color channel.

[0076] In operation S105, the image processor 200 may calculate a slope difference DS between the actual slope RS and the reference slope IS for each brightness period. For example, the image processor 200 may define the slope difference DS as a linear function for each brightness period. In some embodiments, the image processor 200 may calculate a first slope difference between a first actual slope and a first reference slope for each first brightness period. In addition, the image processor 200 may calculate a second slope difference between a second actual slope and a second reference slope for each second brightness period. In addition, the image processor 200 may calculate a third slope difference between a third actual slope and a third reference slope for each third brightness period.

[0077] In operation S107, the image processor 200 may store the slope difference DS for each brightness period. In some embodiments, the image processor 200 may store the first slope difference for each first brightness period, the second slope difference for each second brightness period, and the third slope difference for each third brightness period.

[0078] Fig. 9 is a graph showing an actual slope RS and a reference slope IS according to an embodiment of the present disclosure.

[0079] Reference Fig. 9 , the reference slope IS and the actual slope RS may be expressed according to the relationship between the exposure time and the brightness value. The brightness value may refer to the average brightness value of the second actual brightness value DPX2s. The reference slope IS may have a brightness value that increases continuously in response to the increasing exposure time. The actual slope RS may have a brightness value that increases randomly compared to the increasing exposure time. That is, the actual slope RS may have a linear error.

[0080] Fig.10 is a graph showing a slope difference DS according to an embodiment of the present disclosure.

[0081] Reference Fig.10 , the image processor 200 may analyze the slope difference DS between the reference slope IS and the actual slope RS. In addition, the image processor 200 may regard the brightness values ​​showing similar trends as one brightness period according to the analysis result. For example, when the slope difference DS has four trends, the slope difference DS may be defined as first to fourth linear functions F1, F2, F3 and F4 corresponding to the first to fourth brightness periods D1, D2, D3 and D4, respectively.

[0082] Fig.11 The present invention is shown in the embodiment of the present invention. Figure 7 4. A flowchart of operation S200 of compensating the first actual brightness value DPXs is shown.

[0083] Reference Fig.11 In operation S201, the image sensor 100 may generate first actual luminance values ​​DPX1s corresponding to a captured image. Hereinafter, a process in which one of the first actual luminance values ​​DPX1s is corrected will be representatively described.

[0084] In operations S203, S207, and S211, the image processor 200 may determine to which brightness period the first actual brightness value belongs. For example, when it is determined that the first actual brightness value belongs to the first brightness period D1 (i.e., "yes" in operation S203), the image processor 200 may read constants a1 and b1 corresponding to the first linear function F1 from the storage unit 220 in operation S205. In addition, in operation S219, the image processor 200 may correct the first actual brightness value by applying constants a1 and b1 to the above-mentioned "Formula 6". When it is determined that the first actual brightness value belongs to the second brightness period D2 (i.e., "yes" in operation S207), the image processor 200 may read constants a2 and b2 corresponding to the second linear function F2 from the storage unit 220 in operation S209. In addition, in operation S219, the image processor 200 may correct the first actual brightness value by applying constants a2 and b2 to the above-mentioned "Formula 6". When it is determined that the first actual brightness value belongs to the third brightness period D3 (ie, "yes" in operation S211), the image processor 200 may read constants a3 and b3 corresponding to the third linear function F3 from the storage unit 220 in operation S213. In addition, in operation S219, the image processor 200 may correct the first actual brightness value by applying the constants a3 and b3 to the above-mentioned "Formula 6". When it is determined that the first actual brightness value does not belong to the third brightness period D3 (ie, "no" in operation S211), the image processor 200 may determine that the first actual brightness value belongs to the fourth brightness period D4 in operation S215 and read constants a4 and b4 corresponding to the fourth linear function F4 from the storage unit 220 in operation S217. In addition, in operation S219, the image processor 200 may correct the first actual brightness value by applying the constants a4 and b4 to the above-mentioned "Formula 6".

[0085] According to the above-described embodiments of the present disclosure, the image sensing system may have a linear sensitivity characteristic by compensating for an actual brightness value having a linear error, using a simple calculation method such as a linear function, and compensating for the sensitivity characteristic through firmware such as an image processor.

[0086] According to the above-described embodiments of the present disclosure, the image sensing system may have a linear sensitivity characteristic, thereby generating a linear brightness value.

[0087] In addition, according to the embodiments of the present disclosure, the image sensing system can compensate for the sensitivity characteristics through a simple calculation method, thereby minimizing the number of added circuits.

[0088] In addition, according to the embodiments of the present disclosure, the image sensing system can compensate for sensitivity characteristics through post-processing operations in the digital domain, thereby reducing development costs and design costs, and the image sensing system can be easily used compatibly in various image processing fields.

[0089] Although the present disclosure is shown and described with respect to specific embodiments, the disclosed embodiments are provided for description and are not intended to be limiting. In addition, it should be noted that those skilled in the art will recognize from this disclosure that the present disclosure can be implemented in various ways by substitutions, changes and modifications within the scope of the following claims.

[0090] CROSS-REFERENCE TO RELATED APPLICATIONS

[0091] This application claims priority to Korean Patent Application No. 10-2020-0185922 filed on December 29, 2020 and Korean Patent Application No. 10-2021-0121538 filed on September 13, 2021, the disclosures of which are incorporated herein by reference in their entirety.

Claims

1. An image sensing system, comprising: an image sensor that generates a first actual brightness value corresponding to a captured image; as well as An image processor that sorts the first actual brightness values ​​for each brightness period, generates a result signal corresponding to a result of the sorting, and compensates the first actual brightness value for each brightness period in a single frame period based on the result signal and compensation information corresponding to a linear error, wherein the compensation information represents a slope difference between an actual slope related to a linear error for each brightness period and a reference slope.

2. The image sensing system according to claim 1, wherein: The slope difference is defined as a linear function of each brightness period.

3. The image sensing system according to claim 1, wherein: The actual slope represents the relationship between the exposure time and the second actual brightness value.

4. The image sensing system according to claim 3, wherein: During the test mode period, the second actual brightness value is generated by the image sensor.

5. The image sensing system according to claim 3, wherein: The reference slope represents a relationship between some of the exposure times and some of the second actual brightness values.

6. The image sensing system according to claim 1, wherein: The image processor sorts the first actual brightness values ​​of the respective color channels and compensates the first actual brightness values ​​based on different compensation information.

7. The image sensing system according to claim 1, wherein: The image processor comprises: a compensator that compensates the first actual brightness value for each brightness period based on a slope difference between an actual slope and a reference slope, wherein the actual slope represents a relationship between an exposure time and an actual brightness value, and the reference slope represents a relationship between the exposure time and a reference brightness value; and A controller is provided to calculate the slope difference of each brightness period based on a second actual brightness value generated by the image sensor.

8. An image sensing system, the image sensing system comprising: a compensator that determines respective brightness periods and differently compensates a first actual brightness value generated by an image sensor for the respective brightness periods based on a slope difference between an actual slope and a reference slope, wherein the actual slope represents a relationship between an exposure time and an actual brightness value and is related to a linear error, and the reference slope represents a relationship between the exposure time and a reference brightness value; and A controller is provided to calculate the slope difference of each brightness period based on a second actual brightness value generated by the image sensor.

9. The image sensing system according to claim 8, wherein: The reference brightness values ​​are some of the second actual brightness values.

10. The image sensing system according to claim 8, wherein: The slope difference is defined as a linear function of each brightness period.

11. The image sensing system according to claim 8, wherein: The controller comprises: a first calculator, the first calculator calculating the actual slope based on the second actual brightness value; a second calculator that calculates the reference slope based on the reference brightness value; and A third calculator is provided to calculate the slope difference based on the actual slope and the reference slope.

12. The image sensing system according to claim 8, wherein: The compensator comprises: a brightness period determiner that sorts the first actual brightness values ​​of the respective brightness periods and generates a result signal corresponding to a result of the sorting; and A corrector is provided to correct the first actual brightness value based on the result signal and the slope difference.

13. The image sensing system according to claim 12, wherein: The corrector includes a multiplier and an adder. 14 . The image sensing system according to claim 8 , further comprising a storage section storing the slope difference for each brightness period.

15. An image sensing system, the image sensing system comprising: an image sensor that generates a first actual brightness value and a second actual brightness value corresponding to a captured image during a first time period and a second time period; a controller that determines an actual slope based on an exposure time and the second actual brightness value, determines a reference slope based on the exposure time and the actual slope, and determines a slope difference between the reference slope and the actual slope; as well as A compensator divides the slope difference into a plurality of linear functions for a plurality of brightness periods and compensates the first actual brightness value for each brightness period using a corresponding linear function among the plurality of linear functions.

16. A method for operating an image sensing system, the method comprising the following steps: generating one or more test images; calculating, based on second actual brightness values ​​corresponding to the test image, an actual slope indicating a relationship between exposure time and a second actual brightness value and a reference slope indicating a relationship between some of the exposure times and some of the second actual brightness values, and generating a compensation function based on a slope difference between the actual slope and the reference slope; as well as differently compensating the first actual brightness value of each output period based on the compensation function, Wherein, the compensation function includes a linear function.

17. The operating method according to claim 16, wherein: The respective output periods include respective brightness periods.

Images