Processing device, imaging device, and program

Abstract

This technology provides the ability to acquire white level information appropriate for image correction. [Solution] The processing device performs processing on multiple pixels of an image sensor. Each of the multiple pixels has multiple light-receiving elements. Each of the multiple light-receiving elements has a photoelectric conversion element whose current changes according to the amount of light irradiated onto the light-receiving element. The processing device includes a storage unit and a white level information generation unit. The storage unit stores black level information representing the pixel values ​​of the multiple pixels when the image sensor is not detecting light. The white level information generation unit generates white level information representing the pixel values ​​of the multiple pixels when the image sensor is detecting uniform light, based on the black level information.

Description

Technical Field

[0001] The present disclosure relates to an image sensor.

Background Art

[0002] Patent Document 1 discloses an image sensor in which each pixel has a photoelectric conversion element.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For the correction of an image obtained by an image sensor, white level information representing a plurality of pixel values obtained by the image sensor when the image sensor detects uniform light may be used.

[0005] An object of the present disclosure is to provide a technique capable of acquiring white level information suitable for image correction.

Means for Solving the Problems

[0006] One aspect of the processing device performs processing related to a plurality of pixels included in an image sensor. The plurality of pixels each have a plurality of light receiving elements. Each of the plurality of light receiving elements has a photoelectric conversion element in which the flowing current changes according to the amount of light irradiated to the light receiving element. The processing device includes a storage unit and a white level information generation unit. The storage unit stores black level information representing pixel values of a plurality of pixels when the image sensor is not detecting light. The white level information generation unit generates white level information representing pixel values of a plurality of pixels when the image sensor is detecting uniform light, based on the black level information.

[0007] Furthermore, one embodiment of the processing device performs processing on an image sensor having multiple pixels. Each of the multiple pixels has multiple light-receiving elements. Each of the multiple light-receiving elements has a photoelectric conversion element whose current changes according to the amount of light irradiated onto the light-receiving element. The processing device comprises a storage unit and a white level information adjustment unit. The storage unit stores white level information representing the pixel values ​​of the multiple pixels when the image sensor detects uniform light. The white level information adjustment unit adjusts the white level information based on the pixel values ​​obtained from the image sensor.

[0008] Furthermore, the imaging device includes an image sensor and the processing device described above, which performs processing on multiple pixels of the image sensor.

[0009] Another form of the program is one that causes a computer device to function as the processing unit described above. [Effects of the Invention]

[0010] Appropriate white level information for image correction can be obtained. [Brief explanation of the drawing]

[0011] [Figure 1] This is a schematic diagram showing an example of an imaging device. [Figure 2] This is a schematic diagram showing an example of the configuration of the processing unit. [Figure 3] This is a schematic diagram showing an example of an image sensor configuration. [Figure 4] This is a schematic diagram showing an example of pixel configuration. [Figure 5] This is a schematic diagram showing an example of the configuration of a light-receiving element. [Figure 6] This graph shows an example of the voltage-current characteristics of a photoelectric conversion element. [Figure 7] This is a schematic diagram illustrating an example of the operation of the sensor control unit. [Figure 8] This is a schematic diagram showing an example of a histogram of multiple pixel values ​​for black level information. [Figure 9]It is a schematic diagram showing an example of a histogram of a plurality of pixel values of white level information. [Figure 10] It is a schematic diagram showing an example of the configuration of the control unit. [Figure 11] It is a flowchart showing an example of the operation of the control unit. [Figure 12] It is a flowchart showing an example of the operation of the control unit. [Figure 13] It is a graph for explaining an example of the operation of the control unit. [Figure 14] It is a graph for explaining an example of the operation of the control unit. [Figure 15] It is a schematic diagram showing an example of the information stored in the storage unit. [Figure 16] It is a schematic diagram showing an example of a histogram of a plurality of pixel values of black level information. [Figure 17] It is a schematic diagram showing an example of a histogram of a plurality of pixel values of white level information. [Figure 18] It is a graph showing an example of the voltage-current characteristics of the photoelectric conversion element. [Figure 19] It is a schematic diagram showing an example of the information stored in the storage unit. [Figure 20] It is a schematic diagram showing an example of the configuration of the control unit. [Figure 21] It is a schematic diagram showing an example of the configuration of the control unit. [Figure 22] It is a schematic diagram showing an example of the configuration of the control unit.

Embodiments for Carrying Out the Invention

[0012] FIG. 1 is a schematic diagram showing an example of an imaging device 1 that captures an image. The imaging device 1 can capture an image in response to a shooting instruction from a user, for example, and display the captured image. The imaging device 1 is also called a camera, for example. The image captured by the imaging device 1 may be a color image or a grayscale image.

[0013] As shown in Figure 1, the imaging device 1 includes, for example, an image sensor 2 for capturing images, a processing unit 3 for processing related to the image sensor 2, a display unit 4, and an input unit 5 for receiving input from the user. The processing unit 3 can, for example, control the image sensor 2 and the display unit 4. The processing unit 3 can also be called, for example, a control unit, a control device, or a processing device. The image sensor 2, the processing unit 3, the display unit 4, and the input unit 5 are housed in, for example, an external case (in other words, a housing or enclosure).

[0014] The input unit 5 is capable of receiving various inputs from the user. The input unit 5 may, for example, be equipped with multiple operation buttons. The input unit 5 may also be equipped with a touch sensor that receives touch operations from the user. The processing unit 3 can identify the content of the input received by the input unit 5 based on the output signal from the input unit 5.

[0015] The display unit 4 can display various types of information under the control of the processing unit 3. The display unit 4 has a display surface for displaying various types of information. The display unit 4 may be, for example, a liquid crystal display or an organic EL (electro-luminescence) display. Furthermore, if the input unit 5 is equipped with a touch sensor, the touch sensor and the display surface of the display unit 4 may constitute a touch panel display having display and touch detection functions. In this case, the input unit 5 can detect touch operations on the display surface of the display unit 4. The input unit 5 and the display unit 4 constitute a user interface.

[0016] The image sensor 2 detects light. The imaging device 1 is equipped with, for example, a lens, and light passing through this lens illuminates the image sensor 2. The image sensor 2 has multiple pixels. The pixels of the image sensor 2 can also be called pixel circuits. The image sensor 2 outputs the pixel values ​​of the multiple pixels to the processing unit 3. The pixel values ​​of the multiple pixels constitute a single image. The processing unit 3 can display the image composed of the multiple pixel values ​​from the image sensor 2, that is, the image captured by the image sensor 2, on the display unit 4.

[0017] Hereafter, when we simply refer to a pixel, we mean the pixels of the image sensor 2. Also, when we simply refer to a pixel value and an image, we mean the pixel value and image obtained by the image sensor 2.

[0018] <Example of processing unit configuration> Figure 2 is a schematic diagram showing an example of the configuration of the processing unit 3. As shown in Figure 2, the processing unit 3 (in other words, the processing unit 3) includes, for example, a control unit 30 and a storage unit 31. The control unit 30 is capable of controlling the image sensor 2 and the display unit 4. The control unit 30 is capable of comprehensively managing the operation of the imaging device 1. The control unit 30 includes, for example, at least one processor. The control unit 30 may include, for example, a CPU (Central Processing Unit). The control unit 30 can also be called, for example, a control circuit. The processing unit 3 can also be called, for example, a computer device.

[0019] The memory unit 31 may include non-temporary recording media that can be read by the CPU of the control unit 30, such as ROM (Read Only Memory) and RAM (Random Access Memory). The memory unit 31 stores, for example, a program 130 for controlling the processing unit 3. Various functions of the control unit 30 are realized, for example, by the CPU of the control unit 30 executing the program 130 in the memory unit 31.

[0020] In addition to the program 130, the memory unit 31 also includes, for example, black level information 140 and white level information 150 used for correcting images generated by the image sensor 2. The black level information 140 represents the pixel values ​​of multiple pixels when the image sensor 2 is not detecting light. The white level information 150 represents the pixel values ​​of multiple pixels when the image sensor 2 is detecting uniform light.

[0021] In the black level information 140, a pixel value is associated with each pixel. For example, in the black level information 140, for each pixel, the identification information of that pixel and the pixel value of that pixel are associated. In the white level information 150, similar to the black level information 140, a pixel value is associated with each pixel. The black level information 140 can be called, for example, a black level table, and the white level information 150 can be called, for example, a white level table.

[0022] As will be described later, the control unit 30 corrects the image based on the black level information 140 and the white level information 150. Then, the control unit 30 displays the corrected image on the display unit 4.

[0023] The configuration of the control unit 30 is not limited to the above example. For example, the control unit 30 may have multiple CPUs. The control unit 30 may also have at least one DSP (Digital Signal Processor). Furthermore, all or some of the functions of the control unit 30 may be implemented by hardware circuits that do not require software to implement those functions. The storage unit 31 may also have a computer-readable non-temporary recording medium other than ROM and RAM. The storage unit 31 may include, for example, a small hard disk drive and an SSD (Solid State Drive). Furthermore, the processing unit 3 may be composed of a microcomputer that includes, for example, a CPU, input / output circuits, memory circuits, and peripheral circuits.

[0024] <Example of image sensor configuration> Figure 3 is a schematic diagram showing an example of the configuration of the image sensor 2. As shown in Figure 3, the image sensor 2 includes, for example, a pixel array 200 in which a plurality of pixels 201 are arranged in a matrix, a sensor control unit 250 that controls the pixel array 200, a voltage generation unit 270, and an AD converter 280. The image sensor 2 also includes a plurality of row signal lines 202 that extend in the row direction (left-right direction in Figure 3) and are arranged in the column direction (up-down direction in Figure 3), and a plurality of column signal lines 203 that extend in the column direction and are arranged in the row direction. A single pixel 201 is located at the intersection where a row signal line 202 and a column signal line 203 intersect, and is connected to the row signal line 202 and the column signal line 203.

[0025] Figure 4 is a schematic diagram showing an example of the configuration of each pixel 201. Each pixel 201 includes, for example, a light-receiving element 210, a first selection switch 222, and a second selection switch 223. In addition, each pixel 201 includes, for example, an exposure time setting switch 231, a reset switch 232, and a capacitor 235.

[0026] The on / off states of the first selection switch 222, the second selection switch 223, the exposure time setting switch 231, and the reset switch 232 are controlled by the sensor control unit 250.

[0027] The light-receiving element 210 includes a photoelectric conversion element 211 whose current changes according to the amount of light irradiated onto the light-receiving element 210. The photoelectric conversion element 211 is, for example, a diode; hereafter, the photoelectric conversion element 211 that is a diode may be referred to as diode 211.

[0028] Figure 5 is a schematic diagram showing an example of the configuration of the photodetector 210. The photodetector 210 is a semiconductor device. As shown in Figure 5, the photodetector 210 comprises, for example, a p-type semiconductor region 216 and an n-type semiconductor region 217. The p-type semiconductor region 216 functions as the anode of the diode 211, and the n-type semiconductor region 217 functions as the cathode of the diode 211. The n-type semiconductor region 217, which functions as the cathode, is supplied with a voltage VR output by the voltage generation unit 270. Voltage VR can be said to be the reverse voltage supplied to the diode 211. The configuration between the p-type semiconductor region 216 and the n-type semiconductor region 217 shown in Figure 5 represents a depletion layer.

[0029] When light is shone on the image sensor 2, the light is shone on the junction between the p-type semiconductor region 216 and the n-type semiconductor region 217 in each light-receiving element 210. Due to the photovoltaic effect, electrons and holes are generated at the junction, with the holes moving to the p-type semiconductor region 216 and the electrons moving to the n-type semiconductor region 217. As a result, the current flowing through the diode 211 changes according to the amount of light shone on the junction, that is, the amount of light detected by the light-receiving element 210. For example, the absolute value of the current I in the diode 211 increases as the amount of light detected by the light-receiving element 210 increases. The diode 211 can also be called a light-receiving diode or photodiode. Generally, the value of the current I in the diode 211 is considered positive in the direction from the anode to the cathode. Therefore, the value of the current I in the diode 211 is a negative value.

[0030] Figure 6 is a schematic diagram showing an example of the voltage-current characteristics (also called IV characteristics) of diode 211. The horizontal axis of Figure 6 shows the value of the forward voltage VF of diode 211, and the vertical axis shows the value of the current I. There is variation in the IV characteristics of diode 211 of multiple pixels 201.

[0031] Figure 6 shows curve 100 representing the IV characteristics when the photodetector 210 is not detecting light. Figure 6 also shows curve 101 representing the IV characteristics when the photodetector 210 is detecting a relatively small amount of light (first intensity). Furthermore, Figure 6 shows curve 102 representing the IV characteristics when the photodetector 210 is detecting a relatively large amount of light (second intensity). Hereafter, the curves representing the IV characteristics may be referred to as IV curves.

[0032] In this example, the voltage VR (in other words, the reverse voltage) is set so that a current I flows through the diode 211 when the photodetector 210 is not detecting light. In Figure 6, the negative value of the forward voltage VF is the value of the voltage VR. As shown in Figure 6, when the value of the voltage VR is V1, the value of the current I in the diode 211 of the photodetector 210 that is not detecting light is I1. That is, a current I1 flows through the photodetector 210 that is not detecting light. Also, when the value of the voltage VR is V1, the value of the current I in the diode 211 of the photodetector 210 that is detecting a relatively small first-intensity light is I2, which is smaller than I1. And when the value of the voltage VR is V1, the value of the current I in the diode 211 of the photodetector 210 that is detecting a relatively large second-intensity light is I3, which is smaller than both I1 and I2.

[0033] In this example, current flows through the light-receiving element 210, which is not detecting light. The current flowing through the light-receiving element 210 that is not detecting light is also called dark current. The current I in the diode 211 of the light-receiving element 210 that is not detecting light is the dark current. The current flowing through the light-receiving element 210 changes according to the amount of light detected by the light-receiving element 210. Specifically, the current flowing through the light-receiving element 210 increases as the amount of light detected by the light-receiving element 210 increases.

[0034] The light detectable by the light-receiving element 210 includes visible light. In addition to visible light, the light detectable by the light-receiving element 210 may also include, for example, near-infrared light. In this disclosure, near-infrared light is defined as light with a wavelength of 800 nm or more and 2500 nm or less.

[0035] Returning to Figure 4, the first selection switch 222 and the second selection switch 223 of the pixel 201 may be composed of, for example, NMOS transistors. The first selection switch 222 and the second selection switch 223 of the pixel 201 are used to select the pixel 201. When the first selection switch 222 and the second selection switch 223 of the pixel 201 are turned ON, the pixel 201 is selected.

[0036] The on / off state of the first selection switch 222 is controlled by a control signal output from the sensor control unit 250 to the row signal line 202. For example, when the control signal output to the row signal line 202 is at a high level, the first selection switch 222 is turned on, and when the control signal output to the row signal line 202 is at a low level, the first selection switch 222 is turned off. The on / off state of the second selection switch 223 is controlled by a control signal output from the sensor control unit 250 to the column signal line 203. For example, when the control signal output to the column signal line 203 is at a high level, the second selection switch 223 is turned on, and when the control signal output to the column signal line 203 is at a low level, the second selection switch 223 is turned off.

[0037] One end of the first selection switch 222 is connected to the anode of the diode 211. The other end of the first selection switch 222 is connected to one end of the second selection switch 223. The other end of the second selection switch 223 is connected to one end of the exposure time setting switch 231.

[0038] The exposure time setting switch 231 is used to set the exposure time of the pixel 201. The other end of the exposure time setting switch 231 is connected to one end of the reset switch 232 and one end of the capacitor 235. The other end of the reset switch 232 and the other end of the capacitor 235 are connected to ground potential. The on / off state of the exposure time setting switch 231 is controlled by the exposure setting signal es output from the sensor control unit 250. For example, when the exposure setting signal es is at a high level, the exposure time setting switch 231 is in the ON state, and when the exposure setting signal es is at a low level, the exposure time setting switch 231 is in the OFF state. The exposure time setting switch 231 may be composed of, for example, an NMOS transistor.

[0039] The current I flowing through diode 211 causes charge to accumulate in capacitor 235. The reset switch 232 is used to discharge the charge accumulated in capacitor 235 and initialize capacitor 235.

[0040] The on / off state of the reset switch 232 is controlled by the reset signal rs output from the sensor control unit 250. For example, when the reset signal rs is at a high level, the reset switch 232 is in the ON state, and when the reset signal rs is at a low level, the reset switch 232 is in the OFF state. The reset switch 232 may be composed of, for example, an NMOS transistor.

[0041] In the pixel 201 having the above configuration, when the reset switch 232 is in the off state and the first selection switch 222, the second selection switch 223, and the exposure time setting switch 231 are in the on state, the current I flowing through the photodetector 210 flows through the capacitor 235, and charge is accumulated in the capacitor 235. Then, the voltage at one end of the capacitor 235 changes according to the amount of charge accumulated in the capacitor 235. The voltage at one end of the capacitor 235 becomes the pixel value. The voltage at one end of the capacitor 235, i.e., the pixel value, is converted from analog format to digital format by the AD converter 280 and then input to the control unit 30.

[0042] Figure 7 is a schematic diagram illustrating an example of the operation of the sensor control unit 250. Figure 7 shows an example of the time variation of the reset signal rs and the exposure setting signal es output by the sensor control unit 250. The sensor control unit 250 controls each pixel 201 of the image sensor 2 to output a pixel value from each pixel 201. The pixel value output from each pixel 201 is input to the control unit 30. Hereafter, a single pixel 201 of interest will be referred to as the "pixel of interest 201" in this explanation.

[0043] When the sensor control unit 250 wants to output a pixel value from the pixel of interest 201, it first selects the pixel of interest 201 by setting the first selection switch 222 and the second selection switch 223 of the pixel of interest 201 to the ON state.

[0044] Next, as shown in Figure 7, the sensor control unit 250 changes the reset signal rs and the exposure setting signal es from a low level to a high level, turning on the reset switch 232 and the exposure time setting switch 231. After a predetermined time has elapsed, the sensor control unit 250 changes the reset signal rs to a low level, turning off the reset switch 232. This initializes the capacitor 235 and the state of the path through which the current I flows. If the exposure time setting switch 231 remains on and the reset switch 232 is turned off, the current I begins to flow into the capacitor 235, and the accumulation of charge in the capacitor 235 begins. After the accumulation of charge in the capacitor 235 has begun and a predetermined time has elapsed, the sensor control unit 250 changes the exposure setting signal es from a high level to a low level, turning off the exposure time setting switch 231. As shown in Figure 7, the time from the moment the reset signal rs changes to a low level to the moment the exposure setting signal es changes to a low level is the exposure time T. During the exposure time T, charge is accumulated in the capacitor 235 by the current I. After the exposure setting signal es changes to a low level, the AD converter 280 of the image sensor 2 converts the voltage at one end of the capacitor 235 from analog to digital and outputs it to the control unit 30 of the processing unit 3 as the pixel value of the pixel of interest 201. The pixel value can also be called the pixel signal. In this example, the exposure time T is fixed.

[0045] As described above, each pixel 201 is controlled by the sensor control unit 250, allowing the control unit 30 of the processing unit 3 to acquire the pixel value of each pixel 201 from the image sensor 2. Since the current I changes according to the amount of light detected by the light-receiving element 210 (also called the detected light amount), the amount of charge accumulated in the capacitor 235 during the exposure time T also changes according to the detected light amount. Therefore, the pixel value changes according to the detected light amount. Specifically, the larger the detected light amount, the larger the pixel value.

[0046] The AD converter 280 has a resolution of, for example, 10 bits. Therefore, the minimum and maximum digital pixel values ​​received by the control unit 30 from the image sensor 2 are "0" and "1023," respectively. Note that the resolution of the AD converter 280 may be other than 10 bits. Hereafter, when simply referring to a pixel value, it means the digital pixel value output from the AD converter 280.

[0047] The sensor control unit 250 may, for example, include at least one processor, similar to the control unit 30. The sensor control unit 250 may also include a CPU, for example. The sensor control unit 250 can also be called a control circuit, for example. Furthermore, all or some of the functions of the sensor control unit 250 may be implemented by hardware circuits that do not require software to realize those functions.

[0048] The voltage generation unit 270 generates a voltage VR and supplies the generated voltage VR to the cathode of the diode 211 of each pixel 201. The voltage generation unit 270 may be configured as, for example, a step-down circuit that steps down the input voltage and outputs it. The voltage generation unit 270 can also be called, for example, a voltage generation circuit or a voltage generation circuit.

[0049] The voltage VR settings of the diodes 211 for multiple pixels 201 on the image sensor 2 are all the same. In this example, the voltage VR is fixed and does not change. The voltage VR and exposure time T can also be viewed as shooting conditions.

[0050] <Example of processing unit operation> When the input unit 5 receives a shooting instruction from the user, the control unit 30 of the processing unit 3 issues an image acquisition instruction to the sensor control unit 250 of the image sensor 2, instructing it to acquire an image from the image sensor 2. Upon receiving the image acquisition instruction, the sensor control unit 250 controls each pixel 201 of the image sensor 2 to output a pixel value from each pixel 201. The pixel values ​​output from each pixel 201 are input to the control unit 30. The control unit 30 performs image processing on the image composed of the multiple pixel values ​​received from the image sensor 2, such as correction processing using black level information 140 and white level information 150. The control unit 30 then displays the processed image on the display unit 4. Thus, the user who issued the shooting instruction to the input unit 5 can visually confirm the image displayed on the display unit 4. The user's shooting instruction may be, for example, an operation on the shutter button included in the input unit 5.

[0051] <Correction processing using black level information and white level information> The black level information 140 includes the pixel values ​​of multiple pixels 201 when the image sensor 2 is not detecting light. The black level information 140 includes the pixel values ​​actually obtained from multiple pixels 201. If we define the pixel value when the image sensor 2 is not detecting light as the black level, then the black level information 140 includes the black levels of multiple pixels 201.

[0052] On the other hand, the white level information 150 includes the pixel values ​​of multiple pixels 201 when the image sensor 2 detects uniform light of a predetermined intensity. Uniform light can also be called homogeneous light. If we call the pixel value of a pixel 201 when the image sensor 2 detects uniform light of a predetermined intensity the white level, then the white level information 150 includes the white levels of multiple pixels 201.

[0053] The white level information 150 includes, for example, multiple pixel values ​​when the image sensor 2 detects uniform light having a light intensity similar to that of relatively bright light expected in the shooting environment of the imaging device 1. For example, the white level information 150 may include multiple pixel values ​​when the image sensor 2 detects uniform light having a light intensity equal to or close to that of the maximum light intensity expected in the shooting environment. As will be described later, in this example, the white level information 150 is generated based on the black level information 140, which includes the pixel value actually obtained at pixel 201. The white level information 150 includes estimated values ​​of the pixel values ​​actually obtained from multiple pixels 201, rather than the actual pixel values ​​obtained from multiple pixels 201. Therefore, the white level included in the white level information 150 is an estimated value of the pixel value actually obtained from pixel 201.

[0054] Figure 8 is a schematic diagram showing an example of a histogram of multiple pixel values ​​included in the black level information 140. Figure 9 is a schematic diagram showing an example of a histogram of multiple pixel values ​​included in the white level information 150. In Figures 8 and 9, the horizontal axis represents the pixel value and the vertical axis represents the number of pixels.

[0055] Figure 10 is a schematic diagram showing an example of a functional block provided by the control unit 30. As shown in Figure 10, the control unit 30 includes, for example, an image correction unit 300, a black level information generation unit 310, and a white level information generation unit 320 as functional blocks. The image correction unit 300 corrects the image obtained by the image sensor 2. The black level information generation unit 310 generates black level information 140 based on multiple pixel values ​​acquired by the image sensor 2. The white level information generation unit 320 generates white level information 150 based on the black level information 140. Note that all or some of the functions of the image correction unit 300 may be implemented by hardware circuits that do not require software to realize those functions. The same applies to the black level information generation unit 310 and the white level information generation unit 320, and also to the functional blocks described later.

[0056] In this example, the image sensor 2 is assumed to have M pixels 201, where M is an integer greater than or equal to 2. The M pixels 201 are assigned numbers from 1 to M. The pixel value of pixel 201 at number m (1 ≤ m ≤ M) is represented by PV(m). The pixel value PV(m) is the pixel value of the image captured by the image sensor 2 in response to a user's shooting instruction. Furthermore, the black level of pixel 201 at number m, included in the black level information 140, is represented by BL(m), and the white level of pixel 201 at number m, included in the white level information 150, is represented by SL(m).

[0057] The image correction unit 300 corrects the pixel value PV(m) using the following equation (1).

[0058]

number

[0059] In equation (1), CPV(m) represents the corrected pixel value, which is the corrected pixel value PV(m). Also, for sat(i) in equation (1), sat(i)=i if i is 1023 or less, and sat(i)=1023 if i is greater than 1023. Furthermore, DIFF(m) in equation (1) represents the difference between the white level SL(m) and the black level BL(m). Specifically, DIFF(m) represents the value obtained by subtracting the black level BL(m) from the white level SL(m).

[0060] The image correction unit 300 corrects each pixel value of the image captured by the image sensor 2 (also called the real-world image) in response to a shooting instruction from the user in the actual shooting environment, using equation (1). Then, the control unit 30 displays the image composed of the multiple pixel values ​​corrected by the image correction unit 300, that is, the corrected real-world image, on the display unit 4.

[0061] As described above, in this example, even when the photodetector 210 is not detecting light, a current I flows through the diode 211 of the photodetector 210. Therefore, the dark current of the photodetector 210 becomes relatively large. When the photodetector 210 is not detecting light, the pixel value is ideally "0", but when the dark current of the photodetector 210 is large, as in this example, the pixel value becomes large (see Figure 8). Therefore, the effect of the dark current on the pixel values ​​of the real-world image becomes large. Furthermore, due to the variation in dark current among multiple photodetectors 210, variation occurs even among multiple pixel values ​​when the image sensor 2 is not detecting light (see Figure 8). Therefore, variation occurs in the effect of the dark current among multiple pixel values ​​of the real-world image.

[0062] Thus, in this example, the effect of dark current is relatively large at each pixel value, and there is variation in the effect of dark current among multiple pixel values. This is a factor that degrades the quality of real-world images.

[0063] In equation (1) above, the black level BL(m) is subtracted from the pixel value PV(m), thus reducing the effect of dark current in the corrected pixel value CPV(m). Furthermore, since a black level BL(m) prepared individually for each pixel 201 is used to correct each pixel value, the variation in the effect of dark current is reduced among multiple corrected pixel values ​​CPV(m). Therefore, the quality of images in real-world environments can be improved.

[0064] Furthermore, in equation (1), the ratio of 1023, the maximum output value of the AD converter 280, to the difference between the white level SL(m) and the black level BL(m) is multiplied by the difference between the pixel value PV(m) and the black level BL(m). As a result, in the corrected real-world image, the difference between the minimum and maximum pixel values ​​increases, improving contrast.

[0065] <Examples of generating black level information and white level information> In this example, black level information 140 and white level information 150 are generated before the imaging device 1 is shipped. The black level information 140 and white level information 150 are generated, for example, by the processing unit 3 of the imaging device 1. Figure 11 is a flowchart showing an example of the operation of the processing unit 3 when it generates the black level information 140 and white level information 150 before the imaging device 1 is shipped. Hereafter, the situation in which the image sensor 2 does not detect light may be referred to as "no light detection".

[0066] In step s1, the control unit 30 of the processing unit 3 controls the sensor control unit 250 of the image sensor 2 to acquire multiple pixel values ​​when no light is detected. At this time, for example, the lens of the imaging device 1 is covered with a black material (for example, a lens cap), creating a state in which the image sensor 2 does not detect light.

[0067] Next, in step s2, the black level information generation unit 310 (also simply called the generation unit 310) of the control unit 30 generates black level information 140 that includes the multiple pixel values ​​acquired in step s1. The histogram of the multiple pixel values ​​included in the black level information 140 generated in step s2 is, for example, as shown in Figure 8. Then, in step s3, the white level information generation unit 320 (also simply called the generation unit 320) of the control unit 30 generates white level information 150 based on the black level information 140 generated in step s2. The histogram of the multiple pixel values ​​included in the white level information 150 generated in step s3 is, for example, as shown in Figure 9. The control unit 30 stores the generated black level information 140 and white level information 150 in the storage unit 31.

[0068] After the black level information 140 and white level information 150 are stored in the memory unit 31, the imaging device 1 is shipped. After shipment, the imaging device 1 takes images in response to shooting instructions from the user as described above, and corrects the captured images based on the black level information 140 and white level information 150.

[0069] In step s3, the generation unit 320 generates white level information 150 based, for example, on black level information 140 and the voltage-current characteristics of the diode 211 provided by the pixel 201. Figure 12 is a flowchart showing an example of the details of the process in step s3.

[0070] In step s3, step s31 is performed first. In step s31, the generation unit 320 identifies the pixel 201 having the largest black level among the multiple black levels included in the black level information 140 generated in step s2 as the black level maximum pixel 201.

[0071] Next, in step s32, the generation unit 320 sets the white level of the black level maximum pixel 201 to 1023, which is the maximum output value of the AD converter 280. This determines the white level of the black level maximum pixel 201 included in the white level information 150. Then, step s33 is executed.

[0072] Here, the memory unit 31 stores the IV curve of the diode 211 of each pixel 201 when the light-receiving element 210 is not detecting light. Hereafter, the IV curve of the diode 211 of the light-receiving element 210 when it is not detecting light will be called the reference IV curve.

[0073] In step s33, the generation unit 320 obtains a reference IV curve 100a of the black level maximum pixel 201 from the storage unit 31. The generation unit 320 then determines the value of current I Ia0 when the voltage VR is set to the value V1 in the obtained reference IV curve 100a. The generation unit 320 also determines the value of current I Ia (also called the first value Ia) such that the pixel value is 1023, based on the set value of exposure time T and the capacitance of capacitor 235. The generation unit 320 then determines the ratio (in other words, the ratio) of the absolute value of value Ia to the absolute value of value Ia0. This ratio is called the specific ratio. Figure 13 is a schematic diagram showing an example of values ​​Ia0 and Ia.

[0074] Once a specific ratio is determined in step s33, in step s34, the generation unit 320 determines the white level of each pixel 201 other than the black level maximum pixel 201, based on the IV characteristics of the pixel 201 and the specific ratio.

[0075] When the generation unit 320 determines the white level of a pixel 201 of interest, which is one of the pixels 201 other than the pixel 201 with the highest black level, it obtains the reference IV curve 100b of the pixel 201 of interest from the storage unit 31.

[0076] Next, the generation unit 320 determines the value Ib0 of the current I when the voltage VR is set to the set value V1 in the acquired reference IV curve 100b. Next, the generation unit 320 multiplies the value Ib0 by a specific ratio and takes the resulting value as value Ib (also called the second value Ib). Figure 14 is a schematic diagram showing an example of values ​​Ib0 and Ib. Next, the generation unit 320 determines the pixel value of the pixel of interest when the current I is the second value Ib, based on the set value of the exposure time T and the capacitance of the capacitor 235. Then, the generation unit 320 sets the determined pixel value as the white level of the pixel of interest 201.

[0077] Due to variations in the IV characteristics of the diodes 211 of multiple pixels 201, there is variation between the first value Ia for the pixel 201 with the highest black level and the second value Ib for each pixel 201 other than the pixel 201 with the highest black level.

[0078] In this way, in step s34, the generation unit 320 determines the white level of each pixel 201 except for the pixel 201 with the highest black level. This determines the white level of each pixel 201 to be included in the white level information 150. The generation unit 320 generates white level information 150 which includes the white level of each pixel 201.

[0079] As described above, in this example, the white level information 150 is generated such that the maximum white level it contains matches the maximum value of the output range of the AD converter 280. Therefore, the multiple white levels contained in the white level information 150 generated as described above can be seen as estimated values ​​(in other words, pseudo-values) of multiple pixel values ​​actually obtained by the image sensor 2 when the image sensor 2 detects uniform light with a light intensity equivalent to the relatively bright light assumed in the shooting environment of the imaging device 1. It can also be said that each white level contained in the white level information 150 is a value based on the brightness of the shooting environment.

[0080] Thus, in this example, since white level information 150 is generated based on black level information 140, it is possible to easily obtain white level information 150 suitable for image correction without actually irradiating the image sensor with uniform light.

[0081] Furthermore, as in the example above, when white level information 150 is generated based on black level information 140 and the voltage-current characteristics of the photoelectric conversion element 211 of the pixel 201, appropriate white level information 150 for image correction can be obtained.

[0082] <Changes to shooting conditions> In the example above, the voltage VR was fixed, but the voltage VR may be changeable. For example, the voltage generation unit 270 may be capable of outputting voltage VR of multiple values. The processing unit 3 may control the voltage generation unit 270 to change the value of voltage VR.

[0083] For example, the control unit 30 of the processing unit 3 may instruct the voltage generation unit 270 to change the voltage VR according to the brightness of the shooting environment. For example, if the shooting environment is dark, the control unit 30 may increase the voltage VR (in other words, the reverse voltage of the diode 211) in the voltage generation unit 270. When the voltage VR increases, the absolute value of the current I increases, and the pixel value increases. Therefore, by increasing the voltage VR when the shooting environment is dark, a bright image can be obtained without increasing the exposure time T even when the shooting environment is dark. As a result, image blur can be reduced even when the shooting environment is dark. For example, the control unit 30 may calculate the average value of multiple pixel values ​​that constitute the corrected real-world image, and if the calculated average value is below a threshold, it may determine that the shooting environment is dark and instruct the voltage generation unit 270 to increase the voltage VR.

[0084] When the voltage VR changes, the histograms of the black levels of multiple pixels 201 and the histograms of the white levels of multiple pixels 201 change according to the value of the voltage VR.

[0085] Therefore, if the voltage VR is changeable, the processing unit 3 may, for example, before the imaging device 1 is shipped, generate a plurality of black level information 140 corresponding to a plurality of values ​​of the voltage VR, and based on the generated plurality of black level information 140, generate a plurality of white level information 150 corresponding to a plurality of values ​​of the voltage VR.

[0086] For example, consider a case where there are three values ​​for the voltage VR: V1, V2, and V3. In this case, before shipment, the generation unit 310 generates black level information 140 corresponding to V1, black level information 140 corresponding to V2, and black level information 140 corresponding to V3.

[0087] If black level information 140 corresponding to V1 is generated, steps s1 and s2 are executed with the voltage VR set to V1. As a result, black level information 140 corresponding to V1 is generated and stored in the storage unit 31.

[0088] Similarly, when black level information 140 corresponding to V2 is generated, steps s1 and s2 are executed with voltage VR set to V2. As a result, black level information 140 corresponding to V2 is generated and stored in the storage unit 31. Similarly, when black level information 140 corresponding to V3 is generated, steps s1 and s2 are executed with voltage Vr set to V3. As a result, black level information 140 corresponding to V3 is generated and stored in the storage unit 31.

[0089] The generation unit 320 generates white level information 150 based on the black level information 140 corresponding to V1, in the same manner as in step s3 above, and stores it in the storage unit 31. Similarly, the generation unit 320 generates white level information 150 corresponding to V2 based on the black level information 140 corresponding to V2, and stores it in the storage unit 31. Furthermore, the generation unit 320 generates white level information 150 corresponding to V3 based on the black level information 140 corresponding to V3, and stores it in the storage unit 31.

[0090] Figure 15 is a schematic diagram showing an example of how the memory unit 31 stores black level information 140 and white level information 150 corresponding to V1, black level information 140 and white level information 150 corresponding to V2, and black level information 140 and white level information 150 corresponding to V3.

[0091] Before the imaging device 1 is shipped, black level information 140 and white level information 150, each corresponding to a specific value of voltage VR, are stored in the storage unit 31, and then the imaging device 1 is shipped. After shipment, when the image correction unit 300 corrects the real-world image captured by the image sensor 2 in response to a user's shooting instruction, it reads the black level information 140 and white level information 150 corresponding to the current value of voltage VR from the storage unit 31. Then, the image correction unit 300 corrects the real-world image in the same manner as described above, based on the read black level information 140 and white level information 150.

[0092] Thus, when multiple black level information 140 corresponding to multiple values ​​of voltage (e.g., voltage VR) applied to the photoelectric conversion element 211 of the pixel 201 are used to generate multiple white level information 150 corresponding to each of those values, it is possible to easily obtain appropriate white level information 150 that corresponds individually to each value of voltage applied to the photoelectric conversion element 211.

[0093] The processing unit 3 may generate black level information 140 corresponding to a different value of voltage VR from black level information 140 corresponding to a certain value of voltage VR, and generate white level information 150 corresponding to the other value based on the generated black level information 140.

[0094] For example, consider a case where black level information 140 and white level information 150 corresponding to V1 are stored in the storage unit 31 before the imaging device 1 is shipped. In this case, after the imaging device 1 is shipped, if the voltage VR is set to V1, the image correction unit 300 corrects the real-world image based on the black level information 140 and white level information 150 corresponding to V1 stored in the storage unit 31.

[0095] On the other hand, when the voltage VR is set to V2, the generation unit 310 generates black level information 140 corresponding to V2 based on the black level information 140 corresponding to V1 in the storage unit 31. Next, the generation unit 320 generates white level information 150 corresponding to V2 based on the black level information 140 corresponding to V2, in the same manner as above. Then, the image correction unit 300 corrects the real-world image based on the black level information 140 and white level information 150 corresponding to V2. The generation unit 310, generation unit 320, and image correction unit 300 operate similarly even when the voltage VR is set to a value different from V1 and V2.

[0096] For example, suppose that V2 is greater than V1, and the histogram of multiple black levels included in the black level information 140 corresponding to V1 is as shown in Figure 8 above, and the histogram of multiple white levels included in the white level information 150 corresponding to V1 is as shown in Figure 9 above. In such a case, the histogram of multiple black levels included in the black level information 140 corresponding to V2 is as shown in Figure 16, for example. Also, the histogram of multiple white levels included in the white level information 150 corresponding to V2 is as shown in Figure 17, for example.

[0097] The following describes an example of how to generate black level information 140 corresponding to V2. Note that black level information 140 corresponding to a different value than V2 can also be generated using the same method described below.

[0098] The generation unit 310 sets the black level of each pixel 201 according to V2, based on the black level of the pixel 201 included in the black level information 140 according to V1 (i.e., the black level of the pixel 201 according to V1). The black level information 140, which includes the black level according to V2 for each pixel 201, becomes the black level information 140 according to V2.

[0099] When the generation unit 310 generates a black level corresponding to V2 of the pixel of interest 201, it first reads the reference IV curve 100c of the pixel of interest 201 from the storage unit 31. The reference IV curve 100c is the voltage-current characteristic of the diode 211 of the pixel of interest 201 when the image sensor 2 is not detecting light. Next, the generation unit 310 identifies the current value Ic1 (see Figure 18) when the voltage VR is V1 in the reference IV curve 100c. Next, the generation unit 310 identifies the current value Ic2 (see Figure 18) when the voltage VR is V2 in the reference IV curve 100c. Next, the generation unit 310 calculates the ratio of the absolute value of Ic2 to the absolute value of Ic1 (also called the first ratio). Then, the generation unit 310 multiplies the calculated first ratio by the black level of the pixel of interest 201 contained in the black level information 140 corresponding to V1 in the storage unit 31, and sets the resulting value as the black level of the pixel of interest 201 corresponding to V2. This allows us to determine the black level of the target pixel 201 included in the black level information 140 corresponding to V2. Since the pixel value changes according to the current I, the black level corresponding to V2 can be obtained in this way.

[0100] The generation unit 310 similarly determines the black level corresponding to V2 for each pixel 201. It then generates black level information 140, which includes the black level corresponding to V2 for each pixel 201, i.e., black level information 140 corresponding to V2. The black level information 140 corresponding to V2 will include an estimated value of the pixel value that is actually obtained when no light is detected for each pixel 201 where the voltage VR is set to V2, rather than the actual pixel value that is actually obtained.

[0101] In this way, by generating black level information 140 corresponding to a different value of voltage VR based on black level information 140 corresponding to a certain value of voltage VR, it becomes unnecessary to generate the black level information 140 corresponding to that other value based on the actual pixel values ​​from the image sensor 2. Furthermore, even if it is not possible to prepare multiple black level information 140 corresponding to all values ​​of voltage VR before shipment because it is not known how voltage VR will change, black level information 140 can be generated after shipment according to the actual value of voltage VR.

[0102] Furthermore, as in the example above, if black level information 140 corresponding to a different value of voltage VR is generated based on black level information 140 corresponding to a certain value of voltage VR and the voltage-current characteristics of the photoelectric conversion element 211 when the image sensor 2 is not detecting light, then appropriate black level information 140 corresponding to that different value can be obtained.

[0103] In the example above, the black level information 140 and white level information 150 corresponding to V2 are generated after shipment, but the black level information 140 and white level information 150 corresponding to V2 may be generated and stored in the storage unit 31 before shipment.

[0104] In the example above, the exposure time T was fixed, but the exposure time T may be changeable. For example, the sensor control unit 250, which outputs the exposure setting signal es, may be able to set multiple exposure time T values ​​by changing the ON state time of the exposure time setting switch 231. The control unit 30 of the processing unit 3 may control the sensor control unit 250 to change the exposure time T.

[0105] For example, the control unit 30 may instruct the sensor control unit 250 to change the exposure time T according to the brightness of the shooting environment. For example, if the shooting environment is dark, the control unit 30 may instruct the sensor control unit 250 to increase the exposure time T. This makes it possible to obtain a bright image even when the shooting environment is dark. For example, the control unit 30 may calculate the average value of multiple pixel values ​​that make up the corrected real-world image, and if the calculated average value is below a threshold, it may determine that the shooting environment is dark and instruct the sensor control unit 250 to increase the exposure time T.

[0106] When the exposure time T changes, the pixel values ​​change according to the exposure time T. Therefore, the histograms of the black levels of multiple pixels 201 and the histograms of the white levels of multiple pixels 201 change according to the exposure time T.

[0107] Therefore, if the exposure time T changes, the processing unit 3 may, for example, generate a plurality of black level information 140 corresponding to a plurality of exposure time T values ​​before the imaging device 1 is shipped, and generate a plurality of white level information 150 corresponding to a plurality of exposure time T values ​​based on the plurality of black level information 140 generated.

[0108] For example, consider a case where the voltage VR value is fixed, and there are three values ​​for the exposure time T: T1, T2, and T3. In this case, before shipment, the generation unit 310 of the processing unit 3 generates black level information 140 corresponding to T1, black level information 140 corresponding to T2, and black level information 140 corresponding to T3.

[0109] When black level information 140 corresponding to T1 is generated, steps s1 and s2 are executed with the exposure time T set to T1. As a result, black level information 140 corresponding to T1 is generated and stored in the storage unit 31. Black level information 140 corresponding to T2 and black level information 140 corresponding to T3 are generated in the same manner.

[0110] The generation unit 320 generates white level information 150 based on the black level information 140 corresponding to T1, in the same manner as in step s3 above, and stores it in the storage unit 31. The white level information 150 stored in the storage unit 31 becomes white level information 150 corresponding to T1. Similarly, the generation unit 320 generates white level information 150 corresponding to T2 based on the black level information 140 corresponding to T2, and stores it in the storage unit 31. In addition, the generation unit 320 generates white level information 150 corresponding to T3 based on the black level information 140 corresponding to T3, and stores it in the storage unit 31.

[0111] Furthermore, the voltage VR may be changed along with the exposure time T. When the exposure time T and voltage VR change, black level information 140 and white level information 150 may be generated according to the combination of the exposure time T value and the voltage VR value (also called the first combination). Hereafter, the first combination of a certain value Tγ (γ is a variable) of the exposure time T and a certain value Vα of the voltage VR will be referred to as the first combination (Tγ, Vα).

[0112] For example, consider a case where there are two values ​​for exposure time T, T1 and T2, and two values ​​for voltage VR, V1 and V2. In this case, there are four possible first combinations: the first combination (T1, V1), the first combination (T2, V1), the first combination (T1, V2), and the first combination (T2, V2).

[0113] Before shipment, the generation unit 310 of the processing unit 3 generates four black level information 140 corresponding to four first combinations. When black level information 140 corresponding to a first combination (T1, V1) is generated, steps s1 and s2 are executed with the exposure time T set to T1 and the voltage VR set to V1. As a result, black level information 140 corresponding to the first combination (T1, V1) is generated and stored in the storage unit 31. Black level information 140 corresponding to the first combinations other than the first combination (T1, V1) is generated in the same manner.

[0114] Before shipment, the generation unit 320 generates white level information 150 based on the black level information 140 corresponding to the first combination (T1, V1) in the same manner as in step s3 above, and stores it in the storage unit 31. The white level information 150 stored in the storage unit 31 becomes the white level information 150 corresponding to the first combination (T1, V1). Similarly, the generation unit 320 generates white level information 150 corresponding to each of the other three first combinations and stores it in the storage unit 31.

[0115] Figure 19 is a schematic diagram showing an example of how the memory unit 31 stores four black level information 140 corresponding to four first combinations and four white level information 150 corresponding to four first combinations. In the example of Figure 20, focusing on the exposure time T, it can be said that the memory unit 31 stores black level information 140 and white level information 150 corresponding to T1, and black level information 140 and white level information 150 corresponding to T2.

[0116] As described above, before the imaging device 1 is shipped, black level information 140 and white level information 150, each individually corresponding to one of the multiple first combinations, are stored in the storage unit 31, and then the imaging device 1 is shipped. After shipment, when the image correction unit 300 corrects the real-world image captured by the image sensor 2 in response to a user's shooting instruction, it reads the black level information 140 and white level information 150 corresponding to the current first combination from the storage unit 31. Then, the image correction unit 300 corrects the real-world image in the same manner as described above, based on the read black level information 140 and white level information 150.

[0117] For example, consider a scenario where the exposure time T and voltage VR when the image sensor 2 captures an image of the real environment are T1 and V1, respectively, and the current first combination is the second combination (T1, V1). In this case, the image correction unit 300 reads out black level information 140 and white level information 150 corresponding to the first combination (T1, V1) from the storage unit 31. Then, the image correction unit 300 corrects the real environment image based on the read-out black level information 140 and white level information 150.

[0118] The processing unit 3 may generate black level information 140 corresponding to a different value of exposure time T from black level information 140 corresponding to a certain value of exposure time T, and generate white level information 150 corresponding to the other value based on the generated black level information 140.

[0119] For example, consider a case where the voltage VR is fixed, and there are two exposure time T values, T1 and T2, and before shipment, black level information 140 and white level information 150 corresponding to T1 are stored in the storage unit 31. In this case, after shipment, when the exposure time T is set to T1, the image correction unit 300 corrects the real-world image based on the black level information 140 and white level information 150 corresponding to T1 stored in the storage unit 31.

[0120] On the other hand, when the exposure time T is set to T2, the generation unit 310 generates black level information 140 corresponding to T2 based on the black level information 140 corresponding to T1 in the storage unit 31. For example, the generation unit 310 determines the ratio of T2 to T1 (also called the second ratio). Next, the generation unit 310 multiplies each black level included in the black level information 140 corresponding to T1 by the second ratio. Then, the generation unit 310 makes the black level information 140 containing each black level multiplied by the second ratio the black level information 140 corresponding to T2. The generation unit 320 generates white level information 150 corresponding to T2 based on the black level information 140 corresponding to T2, in the same manner as above. The image correction unit 300 corrects the real-world image based on the black level information 140 and white level information 150 corresponding to T2.

[0121] As another example, consider the case where, before shipment, black level information 140 and white level information 150 corresponding to a first combination (T1, Vα) are stored in the storage unit 31 as black level information 140 and white level information 150 corresponding to T1. In this case, after shipment, when the exposure time T and voltage VR are set to T1 and Vα respectively, the image correction unit 300 corrects the real-world image based on the black level information 140 and white level information 150 corresponding to the first combination (T1, Vα) stored in the storage unit 31.

[0122] On the other hand, when the exposure time T and voltage V are set to T2 and Vα, respectively, the generation unit 310 generates black level information 140 corresponding to the first combination (T2, Vα) based on the black level information 140 corresponding to the first combination (T1, Vα) in the storage unit 31. For example, the generation unit 310 multiplies each black level included in the black level information 140 corresponding to the first combination (T1, Vα) by a second ratio. Then, the generation unit 310 makes the black level information 140 containing each black level multiplied by the second ratio the black level information 140 corresponding to the first combination (T2, Vα). The generation unit 320 generates white level information 150 corresponding to the first combination (T2, Vα) based on the black level information 140 corresponding to the first combination (T2, Vα) in the same manner as above. The image correction unit 300 corrects the real-world image based on the black level information 140 and white level information 150 corresponding to the first combination (T2, Vα).

[0123] In this way, by generating black level information 140 corresponding to a different value of exposure time T based on black level information 140 corresponding to a certain value of exposure time T, it becomes unnecessary to generate the black level information 140 corresponding to that other value based on the actual pixel values ​​from the image sensor 2. Furthermore, even if it is not possible to prepare multiple black level information 140 corresponding to all values ​​of exposure time T before shipment because it is not known how the exposure time T will change, black level information 140 can be generated after shipment according to the actual value of exposure time T.

[0124] In the example above, the black level information 140 and white level information 150 corresponding to T2 are generated after shipment, but the black level information 140 and white level information 150 corresponding to T2 may be generated before shipment.

[0125] Furthermore, the control unit 30 may generate black level information 140 corresponding to other first combinations, which have different exposure time T and voltage V values ​​from the given first combination, based on black level information 140 corresponding to a certain first combination.

[0126] For example, consider a case where, before shipment, black level information 140 and white level information 150 corresponding to a first combination (T1, V1) are stored in the storage unit 31. In this case, the control unit 30 may generate black level information 140 corresponding to a first combination (T2, V2) based on the black level information 140 corresponding to a first combination (T1, V1). For example, the generation unit 310 multiplies each black level included in the black level information 140 corresponding to a first combination (T1, V1) by a second ratio. Then, the generation unit 310 makes the black level information 140 containing each black level multiplied by the second ratio the black level information 140 corresponding to a first combination (T2, V1). Next, the generation unit 310 generates black level information 140 corresponding to a first combination (T2, V2) based on the black level information 140 corresponding to a first combination (T2, V1). The black level information 140 corresponding to the first combination (T2, V2) is generated in the same manner as the method for generating the black level information 140 corresponding to V2, as explained with reference to Figure 18.

[0127] <Adjusting white level information> The control unit 30 of the processing unit 3 may include a white level information adjustment unit 330 as a functional block, as shown in Figure 20, which adjusts the white level information 150. The white level information adjustment unit 330 adjusts the white level information 150 based on the pixel values ​​obtained by the image sensor 2 after the imaging device 1 has been shipped. The image correction unit 300 corrects the image in the same manner as described above, based on the black level information 140 and the adjusted white level information 150.

[0128] Here, a particular real-world image that we focus on for the purpose of this explanation will be called the "focused real-world image." Similarly, a particular white level information 150 that we focus on for the purpose of this explanation will be called the "focused white level information 150."

[0129] After shipment, if the target real-world image is corrected by the image correction unit 300 based on the target white level information 150, the white level information adjustment unit 330 may adjust the target white level information 150 based on the maximum pixel value of the multiple pixel values ​​that constitute the target real-world image. For example, the white level information adjustment unit 330 obtains a difference value obtained by subtracting the maximum pixel value from 1023, which is the maximum value of the output range of the AD converter 280. Then, the white level information adjustment unit 330 adjusts the target white level information 150 by subtracting the obtained difference value for each white level included in the target white level information 150. This makes it possible to adjust the white level information 150 according to the actual brightness of the shooting environment and obtain white level information 150 appropriate for image correction. The image correction unit 300 corrects the target real-world image based on the black level information 140 and the target white level information 150 adjusted by the white level information adjustment unit 330. This makes it possible to appropriately correct the real-world image.

[0130] <Identification of defective pixels based on black level information> As shown in Figure 21, the control unit 30 of the processing unit 3 may include a defective pixel identification unit 340 as a functional block that identifies defective pixels 201 included in a plurality of pixels 201 of the image sensor 2 based on black level information 140. In the example of Figure 23, the control unit 30 does not include a white level information adjustment unit 330, but it may include a defective pixel identification unit 340 and a white level information adjustment unit 330.

[0131] The defective pixel identification unit 340 identifies defective pixels 201 based on black level information 140, for example, before the imaging device 1 is shipped. For example, the defective pixel identification unit 340 calculates the average value avg and standard deviation σ of multiple black levels included in the black level information 140. Then, the defective pixel identification unit 340 identifies pixels 201 whose pixel value exceeds the range of avg ± 4σ as defective pixels 201. The defective pixel identification unit 340 may identify multiple defective pixels 201. The identification unit 340 stores the defective pixel identification information for identifying the defective pixels 201 in the storage unit 31.

[0132] Furthermore, as shown in the example in Figure 15 above, if multiple black level information 140 is generated, the defective pixel 201 may be identified using any one of the multiple black level information 140. Alternatively, each of the multiple black level information 140 may be used to identify the defective pixel 201.

[0133] If a defective pixel 201 is identified, the control unit 30 may include, as shown in Figure 22, a black level information correction unit 350 for correcting the black level information 140 and a defective pixel correction unit 360 for correcting the pixel value of the defective pixel 201 included in the real-world image, as functional blocks.

[0134] The black level information correction unit 350 corrects the black level information 140 by correcting the black level (in other words, the pixel value) of the defective pixel 201 included in the black level information 140, for example, before shipment. When correcting the black level of the defective pixel 201 included in the black level information 140, the black level information correction unit 350 generates an interpolated value for the defective pixel 201 based on the black levels of multiple normal pixels 201 surrounding the defective pixel 201 included in the black level information 140. Then, the black level information correction unit 350 replaces the black level of the defective pixel 201 included in the black level information 140 with the generated interpolated value. This corrects the black level of the defective pixel 201 included in the black level information 140. If there are multiple defective pixels 201, the black level information correction unit 350 corrects the black level of each of the multiple defective pixels 201 included in the black level information 140 to correct the black level information 140.

[0135] The white level information generation unit 320 generates white level information 150 based on the black level information 140 corrected by the black level information correction unit 350 before shipment. In the white level information 150 generated based on the corrected black level information 140, that is, the black level of the defective pixel has been corrected, it can be seen that the white level of the defective pixel 201 has been corrected.

[0136] After shipment, the defective pixel correction unit 360 identifies a defective pixel 201 among multiple pixels 201 based on the defective pixel identification information in the storage unit 31. The defective pixel correction unit 360 then corrects the pixel value of the defective pixel 201 included in the real-world image. For example, the defective pixel correction unit 360 generates an interpolated value for the defective pixel 201 based on the pixel values ​​of multiple normal pixels 201 surrounding the defective pixel 201 included in the real-world image. The defective pixel correction unit 360 then replaces the pixel value of the defective pixel 201 included in the real-world image with the generated interpolated value. This corrects the pixel value of the defective pixel 201 included in the real-world image. If multiple defective pixels 201 exist, the defective pixel correction unit 360 corrects the pixel value of each of the multiple defective pixels 201 included in the real-world image.

[0137] The image correction unit 300 corrects the real-world image in which the pixel values ​​of the defective pixels 201 have been corrected, based on the corrected black level information 140 and the white level information 150 generated therefrom, as described above. In other words, the image correction unit 300 corrects the real-world image in which the pixel values ​​of the defective pixels 201 have been corrected, based on the black level information 140 in which the black level of the defective pixels 201 has been corrected and the white level information 150 in which the white level of the defective pixels 201 has been corrected. This improves the image quality of the real-world image.

[0138] Furthermore, as in the example shown in Figure 15 above, if multiple black level information 140s are generated, the black level information modification unit 350 modifies each of the multiple black level information 140s in the same manner. Then, the generation unit 320 generates white level information based on each modified black level information 140.

[0139] The image correction unit 300 corrects the real-world image based on black level information 140, which is uncorrected for the black level of the defective pixel 201, and white level information 150, which is generated based on that black level information. The defective pixel correction unit 360 may then correct the pixel values ​​of the defective pixels 201 included in the real-world image corrected by the image correction unit 300. Even in this case, the image quality of the real-world image can be improved in the same way as described above.

[0140] The control unit 30 may include a white level information adjustment unit 330 (see Figure 22), a defective pixel identification unit 340, a black level information correction unit 350, and a defective pixel correction unit 360 (see Figure 24). In this case, the white level information adjustment unit 330 may adjust the white level information 150 generated based on the black level information 140 corrected by the black level information correction unit 350, based on the pixel values ​​of the actual environment image, as described above. The image correction unit 300 may then correct the actual environment image, from which the pixel values ​​of the defective pixels 201 have been corrected by the defective pixel correction unit 360, based on the black level information 140 corrected by the black level information correction unit 350 and the white level information 150 adjusted by the white level information adjustment unit 330.

[0141] The white level information adjustment unit 330 may adjust the white level information 150 generated based on the black level information 140, which has not been corrected for the black level of the defective pixel 201. In this case, the image correction unit 300 may correct the real-world image based on the black level information 140, which has not been corrected for the black level of the defective pixel 201, and the white level information 150 adjusted by the white level information adjustment unit 330, and the defective pixel correction unit 360 may correct the pixel value of the defective pixel 201 included in the real-world image corrected by the image correction unit 300.

[0142] At least part of the processing performed by the processing unit 3 before shipment may be performed by a processing unit (also called a second processing unit) having a similar configuration to the processing unit 3, provided separately from the processing unit 3. For example, before shipment, the second processing unit may perform the generation of black level information 140, the generation of white level information 150, the identification of defective pixels 201 based on the black level information 140, and the correction of the black level information 140. In this case, the second processing unit may be directly connected to the image sensor 2 and acquire pixel values ​​obtained by the image sensor 2 directly from the image sensor 2, or it may be connected to the processing unit 3 and acquire pixel values ​​obtained by the image sensor 2 through the processing unit 3. In this case, the control unit of the second processing unit will have multiple functional blocks corresponding to a black level information generation unit 310, a white level information generation unit 320, a defective pixel identification unit 340, and a black level information correction unit 350, respectively. In this case, the black level information 140, white level information 150, and defective pixel identification information obtained by the second processing unit are stored in the storage unit 31 of the processing unit 3 and used by the processing unit 3 after shipment. The second processing device may, for example, be part of a manufacturing device or part of an inspection device.

[0143] Furthermore, in the above example, the generation of white level information 150, which is performed before shipment, may be performed after shipment. For example, step s3 in Figure 11 above may be performed after shipment.

[0144] Furthermore, in the above example, the white level information 150 is generated based on the black level information 140, but white level information 150 that includes multiple pixel values ​​actually obtained by the image sensor 2 when the image sensor 2 detects uniform light of a predetermined amount may also be used. In this case, for example, uniform light having an amount of light equivalent to the maximum amount of light expected in the shooting environment is irradiated onto the image sensor 2 before the imaging device 1 is shipped. Then, when the image sensor 2 detects this uniform light, white level information 150 that includes multiple pixel values ​​actually obtained by the image sensor 2 is generated.

[0145] The functions of the elements disclosed herein may be implemented using circuit configurations or processing circuit configurations, including general-purpose processors, dedicated processors, integrated circuits, ASICs ("Application-Specific Integrated Circuits"), conventional circuit configurations, and / or combinations thereof, configured to perform the disclosed elements or programmed to perform the disclosed functions. A processor is considered a processing circuit configuration or circuit configuration if it includes transistors and other circuit configurations within it. In this disclosure, a circuit configuration, unit, or means is hardware that performs the listed functions or hardware programmed to perform such functions. Hardware may be any hardware disclosed herein or other known hardware programmed to perform the listed functions or configured to perform such functions. When hardware is a processor that may be considered a type of circuit configuration, a circuit configuration, means, or unit is a combination of hardware and software, software used to configure the hardware, and / or a processor.

[0146] As described above, the imaging apparatus has been explained in detail, but the above explanation is illustrative in all respects, and this disclosure is not limited thereto. Furthermore, the various examples described above can be combined and applied insofar as they do not contradict each other. And it is understood that countless examples not illustrated can be conceived without falling outside the scope of this disclosure.

[0147] This disclosure includes the following aspects:

[0148] The first embodiment of the processing apparatus is a processing apparatus for processing an image sensor having a plurality of pixels, wherein each of the plurality of pixels has a plurality of light-receiving elements, and each of the plurality of light-receiving elements has a photoelectric conversion element whose current changes according to the amount of light irradiated onto the light-receiving element, and comprises a storage unit that stores black level information representing the pixel values ​​of the plurality of pixels when the image sensor is not detecting light, and a white level information generation unit that generates white level information representing the pixel values ​​of the plurality of pixels when the image sensor is detecting uniform light, based on the black level information.

[0149] The processing apparatus according to the second embodiment is the processing apparatus according to the first embodiment, wherein the white level information generation unit generates the white level information based on the black level information and the voltage-current characteristics of the photoelectric conversion element.

[0150] The processing apparatus according to the third embodiment is a processing apparatus according to the first or second embodiment, wherein the storage unit stores a plurality of black level information corresponding to a plurality of voltage values ​​applied to the photoelectric conversion element, and the white level information generation unit generates a plurality of white level information corresponding to the plurality of voltage values ​​based on the plurality of black level information.

[0151] The processing apparatus according to the fourth embodiment is a processing apparatus according to any one of the first to third embodiments, wherein the storage unit stores a plurality of black level information corresponding to a plurality of exposure time values ​​of the image sensor, and the white level information generation unit generates a plurality of white level information corresponding to a plurality of exposure time values ​​based on the plurality of black level information.

[0152] The processing apparatus according to the fifth embodiment is a processing apparatus according to the first embodiment or the second embodiment, wherein the storage unit stores the black level information corresponding to a first value of the voltage applied to the photoelectric conversion element, the white level information generation unit generates the white level information corresponding to the first value of the voltage based on the black level information corresponding to the first value of the voltage, and the black level information generation unit generates the black level information corresponding to a second value of the voltage based on the black level information corresponding to the first value of the voltage, and the white level information generation unit generates the white level information corresponding to a second value of the voltage based on the black level information corresponding to a second value of the voltage.

[0153] The processing apparatus according to the sixth embodiment is the processing apparatus according to the fifth embodiment, wherein the black level information generation unit generates the black level information according to the second value of the voltage based on the black level information according to the first value of the voltage and the voltage-current characteristics of the photoelectric conversion element when the image sensor is not detecting light.

[0154] The seventh embodiment of the processing apparatus is the first embodiment or the second embodiment, wherein the storage unit stores the black level information corresponding to a first value of the exposure time of the image sensor, the white level information generation unit generates the white level information corresponding to the first value of the exposure time based on the black level information corresponding to the first value of the exposure time, and the black level information generation unit generates the black level information corresponding to a second value of the exposure time based on the black level information corresponding to the first value of the exposure time, and the white level information generation unit generates the white level information corresponding to a second value of the exposure time based on the black level information corresponding to a second value of the exposure time.

[0155] The processing apparatus according to the eighth embodiment is a processing apparatus according to any one of the first to seventh embodiments, comprising an image correction unit that corrects an image having the pixel values ​​of the plurality of pixels based on the black level information and the white level information.

[0156] The processing apparatus according to the ninth embodiment is a processing apparatus according to any one of the first to eighth embodiments, and includes a white level information adjustment unit that adjusts the white level information based on the pixel values ​​obtained by the image sensor.

[0157] The processing apparatus according to the tenth embodiment is the processing apparatus according to the ninth embodiment, wherein the white level information adjustment unit adjusts the white level information based on the maximum pixel value among the pixel values ​​of the plurality of pixels.

[0158] The processing apparatus according to the 11th embodiment is a processing apparatus according to any one of the first to tenth embodiments, and includes a defective pixel identification unit that identifies defective pixels included in the plurality of pixels based on the black level information.

[0159] The processing apparatus according to the 12th embodiment is the processing apparatus according to the 11th embodiment, comprising a black level information correction unit that corrects the black level information by correcting the pixel values ​​of the defective pixels included in the black level information, and the white level information generation unit generates the white level information based on the corrected black level information.

[0160] The processing apparatus according to the 13th embodiment is a processing apparatus according to the 12th embodiment, comprising: a defective pixel correction unit that corrects the pixel values ​​of the defective pixels included in an image having the pixel values ​​of the plurality of pixels; and an image correction unit that corrects the image in which the pixel values ​​of the defective pixels have been corrected based on the corrected black level information and the white level information.

[0161] The processing apparatus according to the 14th embodiment is a processing apparatus according to the 11th embodiment, comprising: an image correction unit that corrects an image having the pixel values ​​of the plurality of pixels based on the black level information and the white level information; and a defective pixel correction unit that corrects the pixel values ​​of the defective pixels included in the pixel values ​​of the plurality of pixels in the corrected image.

[0162] The processing apparatus according to the 15th embodiment is a processing apparatus for processing an image sensor having a plurality of pixels, wherein each of the plurality of pixels has a plurality of light-receiving elements, each of the plurality of light-receiving elements has a photoelectric conversion element whose current changes according to the amount of light irradiated onto the light-receiving element, and comprises a storage unit that stores white level information representing the pixel values ​​of the plurality of pixels when the image sensor detects uniform light, and a white level information adjustment unit that adjusts the white level information based on the pixel values ​​obtained by the image sensor.

[0163] The processing apparatus according to the 16th embodiment is the processing apparatus according to the 15th embodiment, wherein the white level information adjustment unit adjusts the white level information based on the maximum pixel value among the pixel values ​​of the plurality of pixels.

[0164] The processing apparatus according to the 17th embodiment is a processing apparatus according to the 15th or 16th embodiment, wherein the storage unit stores black level information representing the pixel values ​​of the plurality of pixels when the image sensor does not detect light, and the image correction unit corrects an image having the pixel values ​​of the plurality of pixels based on the black level information and the white level information adjusted by the white level information adjustment unit.

[0165] The processing apparatus according to the 18th embodiment is the processing apparatus according to the 17th embodiment, wherein the black level information and the white level information in the storage unit have corrected pixel values ​​of defective pixels included in the plurality of pixels, the white level information adjustment unit includes a defective pixel correction unit that adjusts the white level information in which the pixel values ​​of the defective pixels have been corrected and corrects the pixel values ​​of the defective pixels included in the image, and the image correction unit corrects the image in which the pixel values ​​of the defective pixels have been corrected based on the black level information in which the pixel values ​​of the defective pixels have been corrected and the white level information adjusted by the white level information adjustment unit.

[0166] The processing apparatus according to the 19th embodiment is a processing apparatus according to the 17th embodiment, further comprising a defective pixel correction unit that corrects the pixel values ​​of defective pixels included in the pixel values ​​of the plurality of pixels in the image corrected by the image correction unit.

[0167] The imaging device according to the 20th embodiment comprises an image sensor and a processing device according to any one of the first to 19 embodiments, which performs processing on a plurality of pixels provided by the image sensor.

[0168] The program according to the 21st aspect is a program that causes a computer device to function as a processing unit according to any one of the first to 19th aspects. [Explanation of Symbols]

[0169] 3. Processing Unit 130 programs 140 Black Level Information 150 White Level Information 201 pixels 210 Photodetector 211 Photoelectric conversion element 300 Image Correction Unit 310 Black Level Information Generation Unit 320 White Level Information Generation Unit 330 White Level Information Adjustment Unit 340 Defective Pixel Identification Section 350 Black Level Information Correction Section 360 Defective Pixel Correction Unit

Claims

1. A processing apparatus for processing an image sensor having multiple pixels, Each of the aforementioned multiple pixels has multiple light-receiving elements, Each of the aforementioned plurality of light-receiving elements has a photoelectric conversion element whose current changes according to the amount of light irradiated onto the light-receiving element. A storage unit that stores black level information representing the pixel values ​​of the plurality of pixels when the image sensor is not detecting light, A white level information generation unit generates white level information representing the pixel values ​​of the plurality of pixels when the image sensor detects uniform light, based on the black level information. A processing apparatus equipped with the following features.

2. The apparatus according to claim 1, The white level information generation unit is a processing device that generates the white level information based on the black level information and the voltage-current characteristics of the photoelectric conversion element.

3. The processing apparatus according to claim 1 or claim 2, The memory unit stores a plurality of black level information corresponding to a plurality of voltage values ​​applied to the photoelectric conversion element, The white level information generation unit is a processing device that generates a plurality of white level information based on the plurality of black level information, each corresponding to the plurality of voltage values.

4. The processing apparatus according to claim 1 or claim 2, The storage unit stores a plurality of black level information corresponding to a plurality of exposure time values ​​for the image sensor, The white level information generation unit is a processing device that generates a plurality of white level pieces of information based on the plurality of black level pieces of information, each corresponding to the plurality of exposure time values.

5. The processing apparatus according to claim 1 or claim 2, The storage unit stores the black level information corresponding to the first value of the voltage applied to the photoelectric conversion element. The white level information generation unit generates the white level information corresponding to the first value of the voltage based on the black level information corresponding to the first value of the voltage, The unit comprises a black level information generation unit that generates black level information corresponding to a second value of the voltage based on the black level information corresponding to the first value of the voltage, The white level information generation unit is a processing device that generates white level information corresponding to the second value of the voltage based on the black level information corresponding to the second value of the voltage.

6. The processing apparatus according to claim 5, The black level information generation unit is a processing device that generates black level information corresponding to a second value of the voltage based on the black level information corresponding to a first value of the voltage and the voltage-current characteristics of the photoelectric conversion element when the image sensor is not detecting light.

7. The processing apparatus according to claim 1 or claim 2, The storage unit stores the black level information corresponding to a first value of the exposure time of the image sensor. The white level information generation unit generates white level information corresponding to the first value of exposure time based on the black level information corresponding to the first value of exposure time, The unit includes a black level information generation unit that generates black level information corresponding to a second value of the exposure time based on the black level information corresponding to the first value of the exposure time, The white level information generation unit is a processing device that generates white level information corresponding to the second value of exposure time based on the black level information corresponding to the second value of exposure time.

8. The processing apparatus according to claim 1 or claim 2, A processing apparatus comprising an image correction unit that corrects an image having the pixel values ​​of the plurality of pixels based on the black level information and the white level information.

9. The processing apparatus according to claim 1 or claim 2, A processing apparatus comprising a white level information adjustment unit that adjusts the white level information based on the pixel values ​​obtained from the image sensor.

10. The processing apparatus according to claim 9, The white level information adjustment unit is a processing device that adjusts the white level information based on the maximum pixel value among the pixel values ​​of the plurality of pixels.

11. The processing apparatus according to claim 1 or claim 2, A processing apparatus comprising a defective pixel identification unit that identifies defective pixels included in the plurality of pixels based on the black level information.

12. The apparatus according to claim 11, The system includes a black level information correction unit that corrects the black level information by correcting the pixel values ​​of the defective pixels included in the black level information, The white level information generation unit is a processing device that generates the white level information based on the corrected black level information.

13. The apparatus according to claim 12, A defective pixel correction unit corrects the pixel values ​​of the defective pixels included in the image having the pixel values ​​of the plurality of pixels, An image correction unit corrects the image, in which the pixel values ​​of the defective pixels have been corrected, based on the corrected black level information and white level information. A processing apparatus equipped with the following features.

14. The apparatus according to claim 11, An image correction unit that corrects an image having the pixel values ​​of the plurality of pixels based on the black level information and the white level information, A defective pixel correction unit corrects the pixel values ​​of the defective pixels included in the pixel values ​​of the plurality of pixels in the corrected image. A processing apparatus equipped with the following features.

15. A processing apparatus for processing an image sensor having multiple pixels, Each of the aforementioned multiple pixels has multiple light-receiving elements, Each of the aforementioned plurality of light-receiving elements has a photoelectric conversion element whose current changes according to the amount of light irradiated onto the light-receiving element. A storage unit that stores white level information representing the pixel values ​​of the plurality of pixels when the image sensor detects uniform light, A white level information adjustment unit adjusts the white level information based on the pixel values ​​obtained from the image sensor. A processing apparatus equipped with the following features.

16. The processing apparatus according to claim 15, The white level information adjustment unit is a processing device that adjusts the white level information based on the maximum pixel value among the pixel values ​​of the plurality of pixels.

17. The processing apparatus according to claim 15 or claim 16, The storage unit stores black level information representing the pixel values ​​of the plurality of pixels when the image sensor is not detecting light. A processing apparatus comprising an image correction unit that corrects an image having the pixel values ​​of the plurality of pixels based on the black level information and the white level information adjusted by the white level information adjustment unit.

18. The apparatus according to claim 17, In the black level information and white level information within the storage unit, the pixel values ​​of the defective pixels included in the plurality of pixels are corrected. The white level information adjustment unit adjusts the white level information, in which the pixel value of the defective pixel has been corrected. The image includes a defective pixel correction unit that corrects the pixel values ​​of the defective pixels contained in the image, The image correction unit is a processing device that corrects the image, in which the pixel values ​​of the defective pixels have been corrected, based on the black level information in which the pixel values ​​of the defective pixels have been corrected and the white level information adjusted by the white level information adjustment unit.

19. The apparatus according to claim 17, A processing apparatus comprising a defective pixel correction unit that corrects the pixel values ​​of defective pixels included in the pixel values ​​of the plurality of pixels in the image corrected by the image correction unit.

20. Image sensor and, The processing apparatus according to claim 1 or claim 15, which performs processing on a plurality of pixels provided by the image sensor, An imaging device equipped with the following features.

21. A program that causes a computer device to function as the processing device described in claim 1 or claim 15.

Images