Image processing device, imaging device, image processing method, and program

Abstract

To solve a problem that the contrast of an image decreases and the visibility is lowered.SOLUTION: An image processing device comprises: acquisition means for acquiring exposure condition information for each pixel of an image, the exposure condition information indicating that predetermined determination conditions are satisfied for each of multiple exposure conditions; condition calculation means for calculating the number of times the exposure condition information appears for each of the multiple exposure conditions as an occurrence frequency; condition control means for setting an information range for extracting an image signal from the image on the basis of the occurrence frequency; and cutting means for cutting a portion of the image signal from the image on the basis of the information range.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, and a program. [Background technology]

[0002] In recent years, imaging devices have been proposed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the count value from each pixel to perform photoelectric conversion to generate an image. Patent Document 1 discloses that accurate signal information is obtained in which the linearity between the number of photons incident on the APD and the photon count value detected by the imaging device is maintained. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-019281 [Patent Document 2] Japanese Patent Application Publication No. 2023-068379 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1, which is one of the patent documents mentioned above, can obtain an image with a wider dynamic range than an image acquired by a conventional imaging device equipped with an APD that sets only one exposure time. The technology disclosed in Patent Document 2 extracts a bit string corresponding to a specific luminance distribution range from the dynamic range image and outputs the image data. However, for example, when the output format is SDI (Serial Digital Interface), one of the conventional image output methods, the output range is 10 bits or 12 bits. Therefore, if an image with a wider range than the output format is compressed to fit the output format, the gradation of the image displayed on the image display device cannot be fully expressed, resulting in a problem of reduced image contrast and reduced visibility.

[0005] Therefore, the present invention provides an image processing device, an imaging device, an image processing method, and a program that can improve the contrast of an image and improve visibility. [Means for solving the problem]

[0006] In order to solve this problem, for example, an image processing device of the present invention has the following arrangement: an acquisition means for acquiring exposure condition information for each pixel of an image, the exposure condition information indicating whether a predetermined determination condition is satisfied for each of a plurality of exposure conditions; a condition calculation means for calculating the number of times that the exposure condition information appears for each of the plurality of exposure conditions as an appearance frequency; a condition control means for setting an information range for extracting an image signal based on the frequency of appearance; a cutting means for cutting out a part of an image signal of the image based on the information range; Equipped with. [Effects of the Invention]

[0007] According to the present invention, the contrast of an image can be improved, thereby improving visibility. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the overall configuration of an imaging device according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment. [Figure 3] FIG. 2 is a diagram showing an example of the configuration of a sensor substrate according to an embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a circuit board according to an embodiment. [Figure 5] FIG. 2 is a diagram showing an equivalent circuit of a pixel and a signal processing circuit corresponding to the pixel according to the embodiment. [Figure 6] FIG. 2 is a diagram schematically illustrating the relationship between signals in the signal processing circuit according to the embodiment. [Figure 7] FIG. 4 is a timing chart for explaining the operation of the signal processing circuit according to the embodiment. [Figure 8] 5A and 5B are diagrams showing the relationship between the exposure time and the count value of a counter circuit for each pixel included in the photoelectric conversion element according to the embodiment. [Figure 9] 5A and 5B are diagrams showing the relationship between the exposure time and the count value of a counter circuit for each pixel included in the photoelectric conversion element according to the embodiment. [Figure 10] FIG. 4 is a flowchart illustrating a bit extraction process performed by the image processing device according to the first embodiment. [Figure 11] FIG. 10 is a block diagram showing the overall configuration of an imaging device according to a second embodiment. [Figure 12] FIG. 10 is a diagram showing a central region and a peripheral region of an image according to a second embodiment. [Figure 13] FIG. 10 is a block diagram showing the overall configuration of an imaging device according to a third embodiment. [Figure 14] FIG. 11 is a flowchart illustrating a bit extraction process performed by the image processing device according to the third embodiment. [Figure 15] FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (First embodiment) DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of the present invention will be described in detail below with reference to the accompanying drawings. Fig. 1 is a block diagram showing the overall configuration of an image capturing apparatus according to the first embodiment.

[0011] An example of an image capturing device including an image processing device according to the first embodiment will be specifically described with reference to Fig. 1. The image capturing device 100 includes an image sensor 110 and an image processing device 120.

[0012] The image sensor 110 has multiple avalanche photodiodes (hereinafter referred to as APDs). The image sensor 110 photoelectrically converts light incident on the APDs, digitally counts the number of photons incident on the APDs, and outputs a count value corresponding to the number of photons from each pixel as a digital signal. The image sensor 110 calculates the amount of light incident on each pixel and controls the exposure time (an example of an exposure condition) for each pixel based on the calculated amount of light. The image sensor 110 associates exposure condition bits (also referred to as exposure time information) indicating the exposure time for each pixel with the pixel and outputs the exposure condition bits together with the image signal to the image processing device 120. The image signal information for each pixel includes exposure condition bits indicating the exposure time for each pixel and brightness information bits indicating the amount of light incident on the pixel.

[0013] The configuration and operation of the image sensor 110 will be described in detail with reference to FIGS.

[0014] FIG. 2 is a diagram showing an example of the configuration of a photoelectric conversion element of an image sensor according to this embodiment. Hereinafter, the photoelectric conversion element 200 includes a sensor substrate 21 and a circuit substrate 31. The sensor substrate 21 is stacked on the circuit substrate 31 and electrically connected to the circuit substrate 31. In this embodiment, the photoelectric conversion element 200 is described as having a stacked structure of the sensor substrate 21 and the circuit substrate 31, but is not limited to this structure. For example, the photoelectric conversion element 200 may have a so-called non-stacked structure in which the components included in the sensor substrate 21 and the components included in the circuit substrate 31 are arranged on a common semiconductor layer. The sensor substrate 21 includes a pixel region 22 including a plurality of pixels. The circuit substrate 31 includes a circuit region 32 that processes signals detected in the pixel region 22.

[0015] <Sensor board> 3 is a diagram showing an example of the configuration of the sensor substrate 21. The pixel region 22 of the sensor substrate 21 includes a plurality of pixels 201 arranged two-dimensionally across a plurality of rows and columns. Each pixel 201 has a photoelectric conversion unit 202 including an APD. The photoelectric conversion unit 202 photoelectrically converts received light to obtain charges, and outputs the charges as electrical signals to the circuit board 31. Note that the number of rows and columns of the pixel array in the pixel region 22 is not particularly limited.

[0016] <Circuit board> 4 is a diagram showing an example of the configuration of the circuit board 31. The circuit board 31 has a signal processing circuit 203, a vertical scanning circuit section 210, a horizontal scanning circuit section 211, a readout circuit 212, signal lines 213, an output circuit 214, and a control section 215.

[0017] The signal processing circuit 203 acquires and processes the electrical signals output from the photoelectric conversion unit 202 of each pixel. The signal processing circuit 203 has a counter, a memory, and the like associated with each pixel. The signal processing circuit 203 counts the number of photons and stores the count value in the memory. The signal processing circuit 203 outputs the count value based on a control pulse received from the vertical scanning circuit unit 210.

[0018] The vertical scanning circuit unit 210 receives a control pulse supplied from the control unit 215 and supplies the control pulse to the signal processing circuit 203 of each pixel.

[0019] The horizontal scanning circuit unit 211 receives control pulses supplied from the control unit 215 and supplies control pulses to each pixel column in order to sequentially select each column via the readout circuit 212 and the signal processing circuit 203. The control pulses of the horizontal scanning circuit unit 211 are pulses for reading out pixel signals including count values ​​for each pixel held in the memory of the signal processing circuit 203.

[0020] The readout circuit 212 reads out pixel signals, including count values ​​output by the signal processing circuit 203, for each column via a signal line 213 based on control pulses generated by the vertical scanning circuit unit 210. The readout circuit 212 includes a shift register and an address decoder, connecting multiple rows as a unit. Therefore, the readout circuit 212 achieves high-speed readout by simultaneously reading out pixel signals from multiple rows. In particular, in an imaging device that digitally counts the number of photons incident on an APD and outputs the count value from the pixel as a photoelectrically converted digital signal, the operation of the counter circuit that digitally counts the number of photons takes time. Therefore, high-speed readout of pixel signals is achieved by simultaneously reading out pixel signals from multiple rows. In this embodiment, the readout circuit 212 simultaneously reads out pixel signals from pixels in a first row and pixels in a second row, for example, for each column.

[0021] A pixel signal including a count value and the like is output to the signal line 213 from the signal processing circuit 203 of a pixel at which a column selected by the horizontal scanning circuit unit 211 and one or more rows selected by the vertical scanning circuit unit 210 intersect. The pixel signal output to the signal line 213 is output to the outside of the photoelectric conversion element 200 via the readout circuit 212 and the output circuit 214.

[0022] The control unit 215 controls the photoelectric conversion element 200. Specifically, the control unit 215 supplies control pulses to the vertical scanning circuit unit 210 and the horizontal scanning circuit unit 211 in order to selectively read out pixel signals from each pixel. The control unit 215 sets threshold information (count threshold) that serves as a judgment criterion and judgment time information that indicates the timing at which judgment is performed for a count judgment circuit (described later). The judgment time information may be the above-mentioned control pulse.

[0023] <Connection between sensor board and circuit board> 3 and 4, a plurality of signal processing circuits 203 are arranged in a region overlapping the pixel region 22 and overlapping any of the pixels 201 in a planar view. A vertical scanning circuit unit 210, a horizontal scanning circuit unit 211, a readout circuit 212, an output circuit 214, and a control unit 215 are arranged between an end of the sensor substrate 21 and an end of the pixel region 22 so as to overlap in a planar view. In other words, the sensor substrate 21 has the pixel region 22 and a non-pixel region arranged around the pixel region 22. The vertical scanning circuit unit 210, the horizontal scanning circuit unit 211, the readout circuit 212, the output circuit 214, and the control unit 215 are arranged so as to overlap with the non-pixel region in a planar view.

[0024] The arrangement of the signal lines 213, the readout circuits 212, and the output circuits 214 is not limited to the arrangement shown in Fig. 4. For example, the signal lines 213 may be arranged to extend in the row direction, and the readout circuits 212 may be arranged at the ends of the signal lines 213. Furthermore, the function of the signal processing circuit 203 does not necessarily need to be provided for each photoelectric conversion unit 202, and one signal processing circuit 203 may be shared by multiple photoelectric conversion units 202 to perform signal processing sequentially.

[0025] FIG. 5 is a diagram showing an equivalent circuit of the pixel 201 and the signal processing circuit 203 associated with the pixel 201. As shown in FIG.

[0026] The APD 301 functions as the photoelectric conversion unit 202 of the pixel 201, and performs photoelectric conversion on incident light to generate and output charge pairs corresponding to the incident light. One of the two nodes of the APD 301 is connected to a power supply line that supplies a drive voltage VL (first voltage). The other of the two nodes of the APD 301 is connected to a power supply line that supplies a drive voltage VH (second voltage) higher than the voltage VL. One node of the APD 301 is an anode, and the other node of the APD is a cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 301 so that the APD 301 performs avalanche multiplication. When light is incident while such a voltage is supplied to the APD 301, charges generated by the light undergo avalanche multiplication, generating an avalanche current.

[0027] The drive mode of the APD 301 is classified into two modes depending on the value of the reverse bias voltage at which the APD 301 operates. The two drive modes include Geiger mode and linear mode. In Geiger mode, the APD 301 operates in a state where the voltage difference between the anode and cathode is greater than the breakdown voltage. An APD 301 operated in Geiger mode is called a SPAD (Single Photon Avalanche Diode). In the case of a SPAD, for example, the voltage VL (first voltage) is −30 V and the voltage VH (second voltage) is 1 V. In linear mode, the APD 301 operates in a state where the voltage difference between the anode and cathode is close to or less than the breakdown voltage.

[0028] The quench element 302 is connected to a power supply line to which a drive voltage VH is supplied and to one of the anode and cathode nodes of the APD 301. The quench element 302 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication. The quench element 302 suppresses the voltage supplied to the APD 301 to suppress avalanche multiplication (quench operation). The quench element 302 returns the voltage supplied to the APD 301 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0029] The quench element 302 is, for example, a MOS (Metal Oxide Semiconductor) transistor. The on / off of the quench element 302 is controlled by a control signal CLK input via a control line 320 connected to the gate. The control signal CLK is controlled by a signal generating unit in the control unit 215.

[0030] 5, the signal processing circuit 203 has a waveform shaping unit 310, a counter circuit 311, a count determination circuit 312, and a selection circuit 313 in addition to the quench element 302. It is sufficient that the signal processing circuit 203 has at least one of the waveform shaping unit 310, the counter circuit 311, and the selection circuit 313 in addition to the quench element 302.

[0031] The waveform shaping unit 310 shapes the voltage change at the cathode of the APD 301 obtained via node A when a photon is detected, and outputs a pulse signal corresponding to the frequency of photon reception. The waveform shaping unit 310 is, for example, an inverter circuit. While FIG. 5 shows an example in which the waveform shaping unit 310 has one inverter circuit, it may also be a circuit in which multiple inverter circuits are connected in series, or may be another circuit that has a waveform shaping effect. The waveform shaping unit 310 is an example of an output means.

[0032] Counter circuit 311 counts the pulse signals output from waveform shaping unit 310 via node B and holds the count value. Furthermore, counter circuit 311 resets the held count value when control pulse RES is supplied via drive line 314. Furthermore, counter circuit 311 continues to hold the count value when control pulse STOP is supplied via drive line 317 until control pulse RES is supplied. Counter circuit 311 is an example of counter means.

[0033] The count determination circuit 312 receives the count value held by the counter circuit 311 via drive line 316 and a control pulse from the control unit 215 via drive line 318. Upon receiving the control pulse, the count determination circuit 312 compares the count value with a predetermined count threshold to determine whether the count value exceeds the count threshold. Whether the count value exceeds the count threshold is an example of a predetermined determination condition. If the count determination circuit 312 determines that the count value exceeds the count threshold as a result of the comparison, it supplies a control pulse STOP to the counter circuit 311 via drive line 317 to stop counting. Furthermore, the count determination circuit 312 outputs the determination result at the timing of the control pulse and the count value via drive line 319 to the selection circuit 313. A timing chart of the pulse signal will be described later with reference to FIG. 7. The determination result may be information indicating the exposure time, which is the exposure condition for which the count value exceeds the count threshold (exposure time information and exposure condition bits, described later). The count determination circuit 312 is an example of a determination means.

[0034] 4 to the selection circuit 313 via a drive line 315 (not shown in FIG. 4) in FIG. 5, and switches between electrical connection and disconnection between the count determination circuit 312 and the signal line 213. The selection circuit 313 includes, for example, a buffer circuit for outputting a signal, and outputs an output signal including the determination result and count value received from the count determination circuit 312 of the pixel to the signal line 213.

[0035] The electrical connections may be switched by disposing switches such as transistors between the quench element 302 and the APD 301 and between the photoelectric conversion unit 202 and the signal processing circuit 203. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 202 may be electrically switched using switches such as transistors.

[0036] 6 is a diagram schematically showing the relationship between the control signal CLK, the voltage at node A, the voltage at node B, and the output signal OUT in the signal processing circuit 203 shown in FIG. 5. The control signal CLK is a signal that switches the quench element 302, which is a switch, on and off. The voltage at node A is the voltage at one node of the waveform shaping unit 310 connected to the APD 301. The voltage at node B is the voltage at the other node of the waveform shaping unit 310 connected to the counter circuit 311. The output signal OUT is a signal output from the count determination circuit 312.

[0037] When the control signal CLK is at a high level, the drive voltage VH is not easily supplied to the APD 301, and when the control signal CLK is at a low level, the drive voltage VH is supplied to the APD 301. A high level of the control signal CLK is, for example, 1 V. A low level of the control signal CLK is, for example, 0 V. When the control signal CLK is at a high level, the quench element 302, which is a switch, is turned off. When the control signal CLK is at a low level, the quench element 302, which is a switch, is turned on. The resistance value of the quench element 302 when the control signal CLK is at a high level is higher than the resistance value of the quench element 302 when the control signal CLK is at a low level. When the control signal CLK is at a high level, a recharge operation is not easily performed even if avalanche multiplication occurs in the APD 301, so the voltage supplied to the APD 301 is a voltage equal to or lower than the breakdown voltage of the APD 301. Therefore, the avalanche multiplication operation in the APD 301 stops.

[0038] At time t1, the control signal CLK changes from high to low, turning on the quench element 302 and initiating the recharge operation of the APD 301. This causes the voltage at the cathode of the APD 301 to transition to high. The voltage difference between the voltages applied to the anode and cathode of the APD 301 allows avalanche multiplication. The voltage at the cathode of the APD 301 is the same as node A of the waveform shaping unit 310. Therefore, when the voltage at the cathode of the APD 301 transitions from low to high, the voltage at node A of the waveform shaping unit 310 exceeds the threshold at time t2. At this time, the pulse signal output from node B of the waveform shaping unit 310 is inverted, transitioning from high to low. When the recharge of the APD 301 is completed, the voltage difference (drive voltage VH - drive voltage VL) is applied to the APD 301. After that, the control signal CLK transitions to high, turning off the quench element 302.

[0039] Next, at time t3, when a photon is incident on the APD 301, avalanche multiplication occurs in the APD 301, and an avalanche multiplication current flows through the quench element 302. This causes the voltage at the cathode of the APD 301 to drop. That is, the voltage at node A of the waveform shaping unit 310 drops. If the voltage at node A of the waveform shaping unit 310 drops below the decision threshold while the voltage at node A of the waveform shaping unit 310 is dropping, the voltage at node B of the waveform shaping unit 310 changes from low to high. That is, the waveform shaping unit 310 shapes the portion of the output waveform at node A of the waveform shaping unit 310 that exceeds the decision threshold, and outputs the result as a pulse signal from node B. The counter circuit 311 counts the pulse signal and increments the count value of the counter signal it outputs by 1 LSB (Least Significant Bit).

[0040] Although photons are incident on the APD 301 between time t3 and time t4, the quench element 302 is in the off state, and the voltage difference applied to the APD 301 is not large enough to cause avalanche multiplication, so the voltage level of node A of the waveform shaping unit 310 does not exceed the decision threshold.

[0041] At time t4, the control signal CLK changes from high to low, turning on the quench element 302. As a result, a current that compensates for the voltage drop from the drive voltage VL flows through node A of the waveform shaping unit 310, and the voltage at node A of the waveform shaping unit 310 transitions to its original voltage level. At time t5, the voltage at node A of the waveform shaping unit 310 becomes equal to or greater than the determination threshold, so the pulse signal at node B of the waveform shaping unit 310 is inverted, changing from high to low.

[0042] At time t6, node A of the waveform shaping unit 310 settles to its original voltage level, and the control signal CLK changes from low to high. In principle, the period during which the control signal CLK is low should be longer than the period during which node A of the waveform shaping unit 310 transitions from low to high. In FIG. 6, the period during which the control signal CLK is low is set to be the same as the period during which node A of the waveform shaping unit 310 transitions from low to high. This allows the frequency of the control signal CLK to be set high, thereby reducing the effect of the "nonlinear relationship between the number of input signals and the number of output signals," which will be described later. Thereafter, the voltages of nodes A and B of the waveform shaping unit 310, the signal lines, and the like change in response to the control signal CLK and the incidence of photons, as described from time t1 to time t6.

[0043] However, when the recharge frequency of the APD 301 is controlled by the control signal CLK, the relationship between the number of input signals and the number of output signals is not linear. In this case, the number of input signals refers to the number of photons incident on the APD 301, and the number of output signals refers to the count value of photons detected by the imaging device 100. In a SPAD, when avalanche breakdown occurs, secondary photons are emitted, causing light emission crosstalk with adjacent pixels. However, if the effects of light emission crosstalk are ignored, the relationship between the number of input signals and the number of output signals can be theoretically derived. Specifically, when the number of input signals is Nph, the number of output signals is Nct, the frequency of the control signal CLK (number of CLKs per unit time) is f, and the length of the exposure time is T, the number of output signals Nct can be expressed by the following equation (1):

number

[0044] FIG. 7 is a timing chart illustrating the operation of the signal processing circuit 203. The exposure time for each pixel is set to a predetermined exposure time, T / (n to the (m-1) power). T represents the maximum exposure time within one frame. n is a positive integer. m is an integer greater than or equal to 1, but FIG. 7 shows a timing chart for a case where m is set to 1≦m≦4. A high-level control pulse is supplied to the drive line 318 when the exposure time determined by t=T / (n to the (m-1) power) is reached. In this case, if the count value of the counter circuit 311 reaches a predetermined count threshold when one of the four exposure times described above is reached, the avalanche of the APD 301 enters a pause state. Once the APD 301 enters a pause state, no pulse signal is output from the waveform shaping unit 310. Therefore, the count value of the counter circuit 311 remains unchanged. The count determination circuit 312 outputs T / (n to the (m-1) power), which represents the exposure time corresponding to the control pulse, and the count value to the selection circuit 313.

[0045] FIG. 8 shows the relationship between the exposure time for each pixel 201 included in the photoelectric conversion element 200 and the count value of the counter circuit 311. As in FIG. 7, the exposure time is set to T / (n to the (m-1) power), where m is set to 1≦m≦4. When the upper count limit of the counter circuit 311 is Cmax, the count threshold is set to Cmax / n for reasons that will be described later. In FIG. 8, the count value of a certain pixel 201 increases with time at a constant rate of increase. As described in FIG. 7, the count determination circuit 312 determines whether the count value exceeds the count threshold when the exposure time becomes T / (n to the (m-1) power), in ascending order of exposure time. When the count value increases as shown in FIG. 8, the count determination circuit 312 determines whether the exposure time becomes T / n 3 , and T / n 2When the exposure time is T / n, the count determination circuit 312 determines that the count value does not exceed the count threshold. On the other hand, when the exposure time is T / n, the count determination circuit 312 determines that the count value exceeds the count threshold. The count determination circuit 312 supplies a control pulse STOP to the counter circuit 311 to stop counting at T / n. The count determination circuit 312 outputs the count value Cout at T / n and information about the exposure time T / n. The count determination circuit 312 may output the exposure time T / n as is as the exposure time information, or may output exposure time information Tcode preset according to the exposure time when the count threshold is exceeded, as shown in Table 1. Table 1 shows the relationship between the exposure time when the count threshold is exceeded in the first embodiment, the exposure time information Tcode output according to the exposure time when the count threshold is exceeded, and the number of bits of the value after linearity correction when n = 8. The exposure time information Tcode shown in Table 1 is an example when the output data format is several bits (here, 3 bits). The exposure time information Tcode is the same as the exposure condition bit and is an example of exposure condition information. [Table 1]

[0046] 8 is calculated as Cest=Cout×n, assuming that the count value increases at the same rate for exposure times 0 to T / n and exposure times T / n to T. As mentioned above, when controlling the recharge frequency of APD 301 using the control signal CLK, the relationship between the number of input signals and the number of output signals is not linear, so linearity correction must be performed beforehand.

[0047] FIG. 9 shows the relationship between the exposure time for each pixel 201 included in the photoelectric conversion element 200 and the count value of the counter circuit 311. FIG. 9 also shows the increase in the count value when the count value reaches the upper count limit Cmax at each exposure time during which the count determination circuit 312 performs a count value determination based on the count threshold. In this count estimation method, to maintain the accuracy of the count estimation, the count determination circuit 312 performs a determination based on the count threshold before the count value reaches the upper count limit Cmax. Here, t1 is the timing of the determination based on the count threshold when the count value increases at a rate that will reach the upper count limit Cmax during the maximum exposure time T. In this case, it is necessary to determine the timing t2 of the determination based on the count threshold when the count value increases at a rate that will reach the upper count limit Cmax at time t1. That is, when the count threshold is Cmax / n, the time required to reach the count threshold Cmax / n when the count value increases at a rate that will reach the upper count limit Cmax during the maximum exposure time T is calculated as T × (1 / n), so t1 = T / n. Similarly, t2 = t1 × (1 / n) = T / n 2 In this way, each exposure time for making a determination based on the count threshold Cmax / n is calculated as T / (n to the (m-1)th power).

[0048] As shown in FIG. 1, the image processing device 120 includes an image signal acquisition unit 121, a condition calculation unit 122, a condition control unit 123, a threshold setting unit 124, an output setting acquisition unit 125, a data conversion unit 126, a bit extraction unit 127, an image processing unit 128, and an image output unit 129.

[0049] The image signal acquisition unit 121 receives an image signal from the imaging sensor 110, as well as exposure condition bits and brightness information bits associated with pixels. The exposure condition bits are an example of exposure condition information. The image signal acquisition unit 121 outputs the exposure condition bits and brightness information bits separately. Specifically, the image signal acquisition unit 121 extracts the exposure condition bits output together with the image signal and outputs them to the condition calculation unit 122. The image signal acquisition unit 121 outputs the brightness information bits, which are brightness information of the imaging sensor 110, and the exposure condition bits to the data conversion unit 126.

[0050] The condition calculation unit 122 counts the number of times an exposure condition bit appears for each exposure condition of the exposure condition bit and calculates the counted value as the occurrence frequency. Since the exposure condition bit is output from each pixel whose photon count value exceeds a count threshold, the occurrence frequency can be interpreted as, for example, the number of pixels that satisfy a determination condition based on the count threshold in one frame. A high occurrence frequency of pixels with short exposure times indicates that a large amount of light is incident on each pixel, and the shooting environment can be inferred to be bright. Conversely, a high occurrence frequency of pixels with long exposure times indicates that a small amount of light is incident on each pixel, and the shooting environment can be inferred to be dark. Furthermore, if the occurrence frequency of each exposure condition is uniform, it indicates that pixels with a large amount of incident light and pixels with a small amount of incident light coexist. In other words, it can be inferred that the shooting environment requires a dynamic range. In this way, by calculating the occurrence frequency of each exposure condition, the shooting environment can be determined without creating a histogram of the captured image.

[0051] The condition control unit 123 determines the number of effective bits of the image signal to be used for image processing based on the occurrence frequency calculated by the condition calculation unit 122. Specifically, the condition control unit 123 obtains a frequency threshold for the occurrence frequency from the threshold setting unit 124. If the occurrence frequency of an exposure condition bit for a certain exposure condition received from the condition calculation unit 122 does not reach the frequency threshold, the condition control unit 123 determines that no pixels were captured under that exposure condition. The condition control unit 123 outputs information on an unnecessary bit range, which is information on a bit range for reduction corresponding to the exposure condition for which it was determined that no pixels were captured. The processing by the condition control unit 123 is not limited to this. For example, the information output may not be the unnecessary bit range for reduction, but may be information on an effective bit range (hereinafter referred to as an effective bit range) for remaining bits after subtracting the unnecessary bit range. The unnecessary bit range and the effective bit range are examples of information ranges.

[0052] The threshold setting unit 124 sets a frequency threshold that the condition control unit 123 compares with the occurrence frequency. The threshold setting unit 124 may set the frequency threshold used for determination by the condition control unit 123 by acquiring it from a user or the like. For example, the threshold setting unit 124 may acquire the frequency threshold from a setting component such as a touch panel installed in the image capture device 100, or from a frequency threshold setting controller external to the image capture device 100. The threshold setting unit 124 may set the frequency threshold that the condition control unit 123 compares with the occurrence frequency for each setting for outputting an image signal (hereinafter referred to as an image output setting). For example, when outputting an image signal via SDI, the threshold setting unit 124 may set a high frequency threshold to prevent an image from having low contrast because the image signal is output at 10 bits or 12 bits. If the frequency threshold is increased, the number of exposure conditions under which the condition control unit 123 determines that the occurrence frequency does not reach the frequency threshold increases. This narrows the bit range used, improving contrast. On the other hand, if it is desired to reduce loss of information about the image in order to save the image signal in a storage device or the like, the threshold setting unit 124 may set the frequency threshold to 0. This allows the condition control unit 123 to determine that all exposure conditions exist in the image, thereby eliminating loss of information.

[0053] The output setting acquisition unit 125 acquires image output settings, which are settings (also called output formats) for the image processing device 120 to output an image signal. The output setting acquisition unit 125 outputs the image output settings to the image output unit 129. The output setting acquisition unit 125 may also output the image output settings to the threshold setting unit 124.

[0054] The data conversion unit 126 performs linearity correction on the image signal from the exposure condition bits and the luminance information bits, and outputs the corrected image signal to the bit extraction unit 127 .

[0055] 6, the relationship between the number of input signals and the number of output signals is not linear. Therefore, the data conversion unit 126 performs linearity correction based on the estimated count value calculated by the signal processing circuit 203. The linearity correction refers to finding the number of input signals Nph from the number of output signals Nct per exposure time using the following equation (2), where f is the frequency of the control signal CLK (number of CLKs per unit time) and T is the length of exposure time.

number

[0056] The bit clipping unit 127 cuts out a portion of the image signal by reducing the information of the image signal linearity-corrected by the data conversion unit 126 based on information on at least one of the unnecessary bit range and the effective bit range received from the condition control unit 123. For example, when reducing lower-order bit information based on the unnecessary bit range, the bit clipping unit 127 truncates a specified number of lower-order bits from the image signal output from the data conversion unit 126. Alternatively, when reducing upper-order bit information based on the unnecessary bit range, the bit clipping unit 127 clips values ​​equal to or greater than the maximum value that can be expressed using the number of bits other than the bits specified in the unnecessary bit range. When reducing both upper and lower-order bits based on the unnecessary bit range, the bit clipping unit 127 performs both the truncation and clipping of the lower-order bits. This process enables the image processing device 120 to prevent the final image from having low contrast due to unnecessary information.

[0057] The image processing unit 128 performs gamma conversion on the image signal output from the bit clipping unit 127. Specifically, the image processing unit 128 performs gamma conversion with a gamma value of 2.2 or gamma conversion using PQ (Perceptual Quantization) gamma, which is generally used for images with a wide dynamic range.

[0058] The image output unit 129 outputs the image processed by the image processing unit 128 from the image processing device 120 based on the image output setting acquired from the output setting acquisition unit 125. The format for outputting the image indicated by the image output setting may be an image output format such as SDI and HDMI (registered trademark) (High Definition Multimedia Interface), Ethernet, etc. The image output unit 129 may output an image signal with a signal waveform, amount of information, and bit range in accordance with the output format of each image output setting selected by the output setting acquisition unit 125.

[0059] The following describes the hardware configuration of the image processing device 120. FIG.

[0060] 15, the image processing device 120 is an example of a computer. The image processing device 120 includes a processor 151, a memory 152, a storage 153, a communication IF 154, an input IF 155, an output IF 156, and a bus 157. The processor 151, the memory 152, the storage 153, the communication IF 154, the input IF 155, and the output IF 156 are connected via the bus 157 so as to be able to transmit and receive information to and from each other.

[0061] The processor 151 is an arithmetic processing device, such as a CPU (Central Processing Unit). Note that the image processing device 120 may include other processors, such as an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or a QPU (Quantum Processing Unit), instead of or in addition to the CPU. The processor 151 reads out programs stored in the storage 153 and loads them into the memory 152, thereby realizing various functions. For example, by reading out the programs, the processor 151 realizes some or all of the functions of the image signal acquisition unit 121, the condition calculation unit 122, the condition control unit 123, the threshold setting unit 124, the output setting acquisition unit 125, the data conversion unit 126, the bit extraction unit 127, the image processing unit 128, and the image output unit 129. Some or all of the functions of the image signal acquisition unit 121, condition calculation unit 122, condition control unit 123, threshold setting unit 124, output setting acquisition unit 125, data conversion unit 126, bit extraction unit 127, image processing unit 128, and image output unit 129 may be realized by one or more circuits such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0062] The memory 152 is a high-speed readable / writable storage device such as a RAM (Random Access Memory). The memory 152 functions as a work area when the processor 151 executes a program. The memory 152 temporarily stores the program and parameters necessary for executing the program. For example, the memory 152 temporarily stores the exposure time, exposure time information, the number of bits of the brightness value, the occurrence frequency, the frequency threshold, the image being processed, and the like.

[0063] The storage 153 is a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage 153 retains programs, parameters required for executing the programs, and results of executing the programs even when power is not supplied. The storage 153 stores, for example, programs executed by the processor 151 and images processed by the image processing unit 128.

[0064] The communication IF 154 is an interface for realizing communication with an external device via a wired or wireless network, and transmits, for example, an image output by the image output unit 129 to the external device.

[0065] The input IF 155 is an interface for receiving information input from an input device, such as a mouse, a keyboard, or a touch panel.

[0066] The output IF 156 is an interface for outputting information to an external device, such as a display device.

[0067] The processing of the image processing device 120 according to this embodiment will be described with reference to the flowchart of Fig. 10. Fig. 10 is a flowchart relating to the bit extraction processing performed by the image processing device 120. The flowchart of Fig. 10 is executed by the processor 151 reading out a program and loading it into the memory 152.

[0068] This flowchart shows the case where the image sensor 110 has 11 luminance information bits, which are luminance information, and 3 exposure condition bits, which represent the exposure conditions. The combination of exposure condition bits indicating the exposure condition with the shortest exposure time is called the shortest condition bit, the combination of exposure condition bits indicating the exposure condition with the second shortest exposure time is called the second condition bit, the combination of exposure condition bits indicating the exposure condition with the second longest exposure time is called the third condition bit, and the combination of exposure condition bits indicating the exposure condition with the longest exposure time is called the longest condition bit. The value of the number of input signals Nph after linearity correction can be calculated using equation (2).

[0069] Table 1 shows the exposure time, exposure time information, and number of bits of the brightness value after linearity correction when the count threshold is exceeded when n = 8. The maximum value of the number of input signals Nph, which is the value after linearity correction, is expressed by bits 0 to 10 if the count threshold is not exceeded. In other words, the number of bits for the maximum value of the number of input signals Nph is 11 bits. Similarly, when the exposure condition is T / 8, the maximum value of the number of input signals Nph can be expressed by bits 0 to 13, and the maximum number of bits is 14 bits. When the exposure condition is T / 82, the maximum value of the number of input signals Nph can be expressed by bits 0 to 16, and the maximum number of bits is 17 bits. When the exposure condition is T / 83, the maximum value of the number of input signals Nph can be expressed by bits 0 to 22, and the maximum number of bits is 23 bits. However, because the values ​​of n and T are variable, the maximum number of bits after linearity correction is not limited to this, and the processing described below is not limited to this either.

[0070] In S101, the image signal acquisition unit 121 receives the exposure condition bit along with the image signal and the luminance information bit. In this example, the relationship between the exposure time, which is the exposure condition, and the exposure condition bit (exposure time information Tcode) is as shown in Table 1, and there are four patterns of exposure time. The image signal acquisition unit 121 outputs the exposure condition bit along with the image signal to the condition calculation unit 122. The image signal acquisition unit 121 separates the luminance information bit and the exposure condition bit and outputs them together with the image signal to the data conversion unit 126, and then proceeds to S102.

[0071] In S102, the condition calculation unit 122 recognizes the pattern of the received exposure condition bits, counts the number of times the exposure condition bits appear for each exposure condition, and calculates the counted value as the frequency of appearance. Specifically, the condition calculation unit 122 counts the number of times each exposure condition bit appears during one frame period. When one frame period ends, the condition calculation unit 122 sends the count value to the condition control unit 123, resets the count value, starts counting for the next frame, and proceeds to S1021.

[0072] In S1021, the threshold setting unit 124 sets a frequency threshold. For example, the threshold setting unit 124 sets the frequency threshold based on the output setting acquired from the output setting acquisition unit 125. The threshold setting unit 124 outputs the set frequency threshold to the condition control unit 123, and the process proceeds to S103.

[0073] In S103, the condition control unit 123 compares the appearance frequency of the exposure condition bit for the shortest exposure time calculated by the condition calculation unit 122 with the frequency threshold set by the threshold setting unit 124. If the condition control unit 123 determines as a result of the comparison that the appearance frequency reaches the frequency threshold, the process proceeds to S109. If the condition control unit 123 determines that the appearance frequency does not reach the frequency threshold, the process proceeds to S104.

[0074] Furthermore, in S104, the condition control unit 123 compares the appearance frequency of the exposure condition bit for the second shortest exposure time calculated by the condition calculation unit 122 with the frequency threshold set by the threshold setting unit 124. If, as a result of the comparison, the condition control unit 123 determines that the appearance frequency reaches the frequency threshold, the process proceeds to S108. If the condition control unit 123 determines that the appearance frequency does not reach the frequency threshold, the process proceeds to S105.

[0075] Furthermore, in S105, the condition control unit 123 compares the appearance frequency of the exposure condition bit for the third shortest exposure time calculated by the condition calculation unit 122 with the frequency threshold set by the threshold setting unit 124. If, as a result of the comparison, the condition control unit 123 determines that the appearance frequency reaches the frequency threshold, the process proceeds to S107. If the condition control unit 123 determines that the appearance frequency does not reach the frequency threshold, the process proceeds to S106.

[0076] In S106, the condition control unit 123 sets and outputs the range of unnecessary bits on the upper side of the image signal, and the process proceeds to S110. Specifically, the condition control unit 123 determines that the range of unnecessary bits on the upper side is from the 11th bit to the 22nd bit, and outputs this information to the bit clipping unit 127. Note that the condition control unit 123 may determine that the minimum value of the range of unnecessary bits on the upper side is the 11th bit, and output this information. Alternatively, the condition control unit 123 may determine that the range of effective bits is up to the 10th bit, or that the maximum value of the range of effective bits is the 10th bit, and output this information to the bit clipping unit 127.

[0077] In S107, the condition control unit 123 sets and outputs the range of unnecessary bits on the upper side of the image signal, and the process proceeds to S110. Specifically, the condition control unit 123 determines that the range of unnecessary bits on the upper side is from the 14th bit to the 22nd bit, and outputs the information to the bit clipping unit 127. Note that the condition control unit 123 may determine that the minimum value of the range of unnecessary bits on the upper side is the 14th bit, and output the information. The condition control unit 123 may also determine that the range of effective bits is up to the 13th bit, or that the maximum value of the range of effective bits is the 13th bit, and output the information to the bit clipping unit 127.

[0078] In S108, the condition control unit 123 sets and outputs the range of unnecessary bits on the upper side of the image signal, and the process proceeds to S110. Specifically, the condition control unit 123 determines that the range of unnecessary bits on the upper side is from the 17th bit to the 22nd bit, and outputs this information to the bit clipping unit 127. Note that the condition control unit 123 may determine that the minimum value of the range of unnecessary bits on the upper side is the 17th bit, and output this information. Alternatively, the condition control unit 123 may determine that the range of effective bits is up to the 16th bit, or that the maximum value of the range of effective bits is the 16th bit, and output this information to the bit clipping unit 127.

[0079] In S109, the condition control unit 123 sets and outputs the range of unnecessary bits on the upper side of the image signal, and the process proceeds to S110. Specifically, the condition control unit 123 determines that there is no range of unnecessary bits on the upper side, and outputs the information to the bit clipping unit 127. The condition control unit 123 may also determine that there is no minimum value in the range of unnecessary bits on the upper side, and output the information. The condition control unit 123 may also determine that the valid bit range is up to the 22nd bit, or that the maximum value of the valid bit range is the 22nd bit, and output the information to the bit clipping unit 127.

[0080] In S110, the condition control unit 123 compares the appearance frequency of the exposure condition bit for the longest exposure time calculated by the condition calculation unit 122 with the frequency threshold set by the threshold setting unit 124. If the condition control unit 123 determines as a result of the comparison that the appearance frequency reaches the frequency threshold, the process proceeds to S116. If the condition control unit 123 determines that the appearance frequency does not reach the frequency threshold, the process proceeds to S111.

[0081] Furthermore, in S111, the condition control unit 123 compares the appearance frequency of the exposure condition bit for the third shortest exposure time calculated by the condition calculation unit 122 with the frequency threshold set by the threshold setting unit 124. If, as a result of the comparison, the condition control unit 123 determines that the appearance frequency reaches the frequency threshold, the process proceeds to S115. If the condition control unit 123 determines that the appearance frequency does not reach the frequency threshold, the process proceeds to S112.

[0082] Furthermore, in S112, the condition control unit 123 compares the appearance frequency of the exposure condition bit for the second shortest exposure time calculated by the condition calculation unit 122 with the frequency threshold set by the threshold setting unit 124. If, as a result of the comparison, the condition control unit 123 determines that the appearance frequency reaches the frequency threshold, the process proceeds to S114. If the condition control unit 123 determines that the appearance frequency does not reach the frequency threshold, the process proceeds to S113.

[0083] In S113, the condition control unit 123 sets and outputs the range of unnecessary bits on the lower side of the image signal, and the process proceeds to S1170. Specifically, the condition control unit 123 determines that the range of unnecessary bits on the lower side is from the 0th bit to the 11th bit, and outputs the information to the bit extraction unit 127. Note that the condition control unit 123 may determine that the maximum value of the range of unnecessary bits on the lower side is the 11th bit, and output the information. The condition control unit 123 may also determine that the range of effective bits starts from the 12th bit, or that the minimum value of the range of effective bits is the 12th bit, and output the information to the bit extraction unit 127.

[0084] In S114, the condition control unit 123 sets and outputs the range of unnecessary bits on the lower side of the image signal, and the process proceeds to S1170. Specifically, the condition control unit 123 determines that the range of unnecessary bits on the lower side is from the 0th bit to the 5th bit, and outputs the information to the bit extraction unit 127. Note that the condition control unit 123 may determine that the maximum value of the range of unnecessary bits on the lower side is the 5th bit, and output the information. The condition control unit 123 may also determine that the range of effective bits starts from the 6th bit, or that the minimum value of the range of effective bits is the 6th bit, and output the information to the bit extraction unit 127.

[0085] In S115, the condition control unit 123 sets and outputs the range of unnecessary bits on the lower side of the image signal, and the process proceeds to S1170. Specifically, the condition control unit 123 determines that the range of unnecessary bits on the lower side is from the 0th bit to the 2nd bit, and outputs the information to the bit extraction unit 127. Note that the condition control unit 123 may also determine that the maximum value of the range of unnecessary bits on the lower side is the 2nd bit, and output the information. The condition control unit 123 may also determine that the range of effective bits is from the 3rd bit, or that the minimum value of the range of effective bits is the 3rd bit, and output the information to the bit extraction unit 127.

[0086] In S116, the condition control unit 123 sets and outputs the range of unnecessary bits on the lower side of the image signal, and the process proceeds to S1170. Specifically, the condition control unit 123 determines that there is no range of unnecessary bits on the lower side, and outputs the information to the bit extraction unit 127. Note that the condition control unit 123 may also determine that there is no maximum value for the range of unnecessary bits on the lower side, and output the information. The condition control unit 123 may also determine that the valid bit range starts from the 0th bit, or that the minimum value of the valid bit range is the 0th bit, and output the information to the bit extraction unit 127.

[0087] In S1170, the data conversion unit 126 performs linearity correction on the image signal from the exposure condition bits and brightness information bits acquired from the image signal acquisition unit 121. The data conversion unit 126 outputs the corrected image signal to the bit extraction unit 127, and the process proceeds to S117.

[0088] In S117, the bit extraction unit 127 extracts bits of the linearity-corrected image signal based on the bit range information output from the condition control unit 123. As described above, the condition control unit 123 determines the upper side of the unnecessary bit range in steps S106 to S109, and the lower side of the unnecessary bit range in steps S113 to S116. In this way, the condition control unit 123 determines the upper and lower sides of the unnecessary bit range, and determines the remaining range as the effective bit range. The bit extraction unit 127 extracts the image signal in the effective bit range based on these determinations by the condition control unit 123.

[0089] Specifically, if the lower-order bits of the image signal are determined to be unnecessary bits, the bit clipping unit 127 truncates the lower-order bits of the image signal by that number of bits. If the upper-order bits of the image signal are determined to be unnecessary bits, the bit clipping unit 127 performs clipping processing using the maximum number of bits other than the unnecessary bits. For example, in the darkest case, the bit clipping unit 127 clips the image signal from 0 to 10 bits based on the outputs of steps S106 and S116. On the other hand, in the brightest case, the bit clipping unit 127 clips the image signal from 12 to 22 bits based on the outputs of steps S109 and S113. The bit clipping unit 127 outputs the clipped image signal to the image processing unit 128, and the process proceeds to S118.

[0090] Furthermore, if the appearance frequency of the second condition bit reaches the frequency threshold and the appearance frequency of the third condition bit reaches the frequency threshold, the condition control unit 123 determines in S108 that the maximum value of the valid bit range is the 16th bit, and determines in S115 that the minimum value of the valid bit range is the 3rd bit. As a result, the condition control unit 123 determines that the valid bit range is from the 3rd bit to the 16th bit, and the bit clipping unit 127 clips the image signal from the 3rd bit to the 16th bit. In this way, the condition control unit 123 can appropriately set the number of bits in the valid bit range while changing it as needed.

[0091] In S118, the image processing unit 128 performs a conversion process on the image signal extracted by the bit extraction unit 127. The image processing unit 128 performs, for example, a gamma conversion process on the image signal. The image processing unit 128 outputs the converted image signal to the image output unit 129, and the process proceeds to S119.

[0092] In S119, the image output unit 129 performs image processing on the image signal obtained from the image processing unit 128 in accordance with the image settings obtained from the output setting obtaining unit 125. The image output unit 129 outputs the image signal after the image processing to an external display device or the like.

[0093] By performing the image processing described above, this embodiment determines the unnecessary bit range based on the frequency of occurrence of each conditional bit and reduces unnecessary information in an image with a high dynamic range, thereby improving the contrast of the image and improving the visibility of the image.

[0094] In this embodiment, the frequency threshold is set according to the image output format (output setting), so that the bit range, including the number of bits to be extracted, can be set appropriately according to the image output format.

[0095] In this embodiment, the maximum exposure time is divided into a plurality of exposure conditions, and the occurrence frequency is calculated for each of the plurality of exposure times to set the bit range. As a result, this embodiment can set a more appropriate bit range than when the bit range is set from the occurrence frequency based on a single exposure condition.

[0096] In this embodiment, a portion of the image signal that has been linearized after linearity correction is extracted based on the calculated bit range, thereby suppressing image degradation compared to when the image signal is extracted without linearity correction.

[0097] In this embodiment, an exposure condition bit indicating exposure time information at which the photon count value exceeds the count threshold is obtained for each pixel, so that the occurrence frequency can be easily calculated by counting the exposure condition bit.

[0098] In this embodiment, the upper unnecessary bit range and the lower unnecessary bit range are set separately, so that the bit range, including the number of bits, can be set more appropriately.

[0099] (Second embodiment) The second embodiment will be described in detail with reference to the drawings. In this embodiment, an image processing device will be described that can output an image with higher visibility by setting regions and calculating the frequency of appearance by assigning different weighting coefficients to each region when calculating the bit extraction conditions.

[0100] FIG. 11 is a block diagram showing the overall configuration of an imaging device according to the second embodiment.

[0101] An example of an image capturing device 1100 including an image processing device 1120 according to the second embodiment will be described in detail with reference to Fig. 11. The image capturing device 1100 of the second embodiment includes an image sensor 110 and an image processing device 1120. The image processing device 1120 includes an image signal acquiring unit 121, a condition acquiring unit 1121, a condition calculating unit 1122, a condition control unit 123, a threshold setting unit 124, an output setting acquiring unit 125, a data converting unit 126, a bit extracting unit 127, an image processing unit 128, and an image output unit 129. Compared to the configuration of the image capturing device 100 shown in Fig. 1, the image capturing device 1100 includes a condition calculating unit 1122 and a condition acquiring unit 1121 instead of the condition calculating unit 122. The other blocks of the image capturing device 1100 have the same configuration and perform the same processing.

[0102] The condition acquisition unit 1121 acquires information for dividing the image into regions and weighting coefficient information for each region when calculating the conditions. Fig. 12 is a diagram showing the central region and peripheral region of an image in the second embodiment.

[0103] Image 1200 in FIG. 12 has 1,920 horizontal pixels and 1,080 vertical pixels. Image 1200 is divided into multiple regions (two regions in this example). Specifically, image 1200 has a central region 1201 and a peripheral region 1202. Central region 1201 is an region that is 960 horizontal pixels and 540 vertical pixels and is located in the center of the image. Central region 1201 is ¼ the size of the entire image 1200. Peripheral region 1202 is located so as to surround the periphery of central region 1201. For example, if you want to prioritize capturing a subject in central region 1201, you can set the weighting coefficient for central region 1201 to be larger than the weighting coefficient for peripheral region 1202, thereby setting conditions that enable bit extraction that emphasizes the gradation of central region 1201. For example, if the weighting coefficient for the central region 1201 is 1.75 and the weighting coefficient for the peripheral region 1202 is 0.75, the sum of the occurrence frequencies under each exposure condition will match the sum of the occurrence frequencies under each exposure condition when the occurrence frequencies are calculated without weighting coefficients. In this way, by determining the weighting coefficients based on the ratio of each region so that the sum of the number of pixels included in each region multiplied by the weighting coefficient is constant, there is no need to change the frequency threshold used for comparison in the condition control unit 123, making processing easier. The size, position, and number of regions for the central region 1201 and the peripheral region 1202 are not limited to these, and the image may be divided into two equal regions vertically. Alternatively, multiple regions smaller than the central region 1201 and the peripheral region 1202 in FIG. 12 may be set.

[0104] The condition acquisition unit 1121 may set the regions by accepting region settings from the user. The condition acquisition unit 1121 may set the regions by acquiring information about the regions input by the user from an input device such as a touch panel installed on the image capturing device 1100, or may set the regions by acquiring information input by the user from an input device such as a controller for setting a frequency threshold connected to the image capturing device 1100. Furthermore, the condition acquisition unit 1121 may accept weighting coefficients from the user via the input device and set the weighting coefficients for each region.

[0105] The condition calculation unit 1122 receives area division information within the image and weighting coefficient information for each area from the condition acquisition unit 1121, and calculates the occurrence frequency by adding up the weighting coefficients of the area to which each pixel belongs for each type of exposure condition bit. For example, when the divided area and weighting coefficients are set as conditions as shown in Fig. 12, when a pixel with exposure condition bits of 000 appears once among the pixels in the surrounding area, the image processing device 120 according to the first embodiment adds 1 to the occurrence frequency of the longest condition bit, but the image processing device 1120 according to this embodiment adds 0.75 to the occurrence frequency of the longest condition bit.

[0106] According to the image processing device 1120 of this embodiment, by setting a high weighting coefficient for areas that are important to the user and a low weighting coefficient for areas that are less important, it is possible to prioritize and improve the contrast of areas that are more important to the user, thereby obtaining an image with higher visibility.

[0107] (Third embodiment) The third embodiment will be described in detail with reference to the drawings. In this embodiment, an image processing device that can output an image with higher visibility by changing the formula for gamma conversion depending on the size of the effective number of bits in image processing after bit extraction of the image signal will be described.

[0108] FIG. 13 is a block diagram showing the overall configuration of an imaging device according to the third embodiment. An example of an imaging device 1300 including an image processing device according to the third embodiment will be described in detail with reference to FIG. 13. The imaging device 1300 of the third embodiment includes an image sensor 110 and an image processing device 1320. The image processing device 1320 includes an image signal acquisition unit 121, a condition acquisition unit 1121, a condition calculation unit 1122, a condition control unit 123, a threshold setting unit 124, an output setting acquisition unit 125, a data conversion unit 126, a bit extraction unit 127, an image processing unit 1328, and an image output unit 129. Compared to the configuration of the imaging device 1100 shown in FIG. 11 , the image processing device 1320 of this embodiment includes the image processing unit 1328 instead of the image processing unit 128. The other blocks of the image processing device 1320 have the same configuration and perform similar processing.

[0109] The image processing unit 1328 changes the gamma conversion formula or parameters used for image processing based on at least one of the unnecessary bit range and the effective bit range determined by the condition control unit 123. For example, assume that gamma conversion with a gamma value of 2.2 is used. Gamma conversion with a gamma value of 2.2 is a gamma conversion in which the output luminance value rises gradually relative to the input luminance value. When gamma conversion with a gamma value of 2.2 is applied when the number of bits extracted by the bit extraction unit 127 is small, the luminance difference is maintained even when the information is compressed to suit the image output format, resulting in an image with high contrast and high visibility. However, when gamma conversion with a gamma value of 2.2 is applied when the number of bits extracted is large, the luminance difference is small when the information is compressed to suit the image output format, resulting in an image with low contrast and low visibility. Since the human eye is particularly sensitive to the luminance in dark areas, losing visibility in the dark areas of an image results in an image with low visibility.

[0110] Assume that the image processing unit 1328 uses PQ (Perceptual Quantization) gamma for image processing. Compared to gamma conversion with a gamma value of 2.2, gamma conversion using PQ gamma results in a steeper rise in the output luminance value relative to the input luminance value. When gamma conversion using PQ gamma is applied when the number of bits extracted by the bit extraction unit 127 is large, the luminance difference between both bright and dark areas is maintained even when the information is compressed to suit the image output format, resulting in an image with high visibility. However, when gamma conversion using PQ gamma is applied when the number of bits extracted is small, the luminance difference between bright and dark areas is small when the information is compressed to suit the image output format, resulting in low contrast throughout the image and low visibility. Therefore, the gamma conversion formula used for image processing is changed depending on the number of effective bits of the image signal extracted by the bit extraction unit 127.

[0111] A specific example of the processing of the image processing unit 1328 of the third embodiment will be described with reference to the flowchart in Fig. 14. The luminance information of the image sensor 110 in the flowchart in Fig. 14 is the same as that in Fig. 10. Table 1 shows the exposure time, exposure time information, and number of bits after linearity correction when n=8. Under each exposure condition, the maximum number of bits for the value of the number of input signals Nph after linearity correction is 11 bits, 14 bits, 17 bits, and 23 bits.

[0112] However, since the values ​​of n and T can be changed, the maximum number of bits after linearity correction is not limited to this, and the processing described below is not limited to this either.

[0113] In S301, the image processing unit 1328 receives bit range information of at least one of the unnecessary bit range and the effective bit range from the condition control unit 123. As described in the flowchart of FIG. 10, the bit range information is information indicating the unnecessary bit range and the effective bit range, or the maximum bit and the minimum bit of both ranges. For example, the bit range information may be the number of unnecessary lower bits and the number of unnecessary upper bits that do not contain information about the subject. The image processing unit 1328 proceeds to S302.

[0114] In S302, the image processing unit 1328 calculates the number of effective bits of the image signal based on the number of bits of the image signal sent from the bit clipping unit 127 and the bit range information sent from the condition control unit 123. Specifically, the image processing unit 1328 obtains the number of effective bits of the image signal by subtracting the number of bits obtained from the unnecessary bit information indicated by the bit range information from the number of bits of the image signal sent from the bit clipping unit 127. The image processing unit 1328 proceeds to S303.

[0115] In S303, the image processing unit 1328 compares the number of effective bits calculated in S302 with a bit number threshold for switching gamma conversion. The bit number threshold is, for example, 14 bits. If the image processing unit 1328 determines that the number of effective bits is greater than the bit number threshold, the process proceeds to S304. If the image processing unit 1328 determines that the number of effective bits is less than the bit threshold, the process proceeds to S305.

[0116] In S304, the image processing unit 1328 performs image processing including gamma conversion processing with a first gamma on the image signal. For example, the first gamma is PQ gamma. The image processing unit 1328 proceeds to S306.

[0117] In S305, the image processing unit 1328 performs image processing including gamma conversion processing with a second gamma on the image signal. For example, the second gamma is gamma conversion processing with a gamma value of 2.2. The image processing unit 1328 proceeds to S306.

[0118] In S306, the image processing unit 1328 outputs the image signal that has been subjected to the image processing to the image output unit 129.

[0119] According to the image processing device of this embodiment, it is possible to perform different image processing depending on the number of effective bits of the image signal, and to obtain an image with high visibility even if the shooting environment changes.

[0120] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0121] In the above embodiment, the exposure condition is the exposure time, but the exposure condition may be the shutter speed, the aperture value, or a combination of these values ​​with the exposure time.

[0122] In the above embodiment, the exposure condition information is the exposure time information and the exposure condition bits, but the exposure condition information is not limited to the exposure condition bits, etc., as long as it is information indicating the above exposure conditions. For example, the exposure condition information may be ID information indicating the exposure conditions, etc.

[0123] The disclosure of this specification includes the following image processing device, imaging device, image processing method, and program. (Item 1) an acquisition means for acquiring exposure condition information for each pixel of an image, the exposure condition information indicating whether a predetermined determination condition is satisfied for each of a plurality of exposure conditions; a condition calculation means for calculating the number of times that the exposure condition information appears for each of the plurality of exposure conditions as an appearance frequency; a condition control means for setting an information range for extracting an image signal based on the frequency of appearance; a cutting means for cutting out a part of an image signal of the image based on the information range; An image processing device comprising: (Item 2) further comprising an output means for converting the extracted image signal based on an output format and outputting the converted image signal; The condition control means sets the information range based on the output format. 2. The image processing device according to item 1, (Item 3) The condition calculation means calculates the appearance frequency based on weighting factors set for each of a plurality of regions of the image. 3. The image processing device according to item 1 or 2, (Item 4) a condition acquisition means for accepting and setting the settings of the plurality of areas from a user; 4. The image processing device according to item 3, further comprising: (Item 5) The condition acquisition means receives and changes weighting factors for the plurality of regions from a user. 5. The image processing device according to item 4, (Item 6) an image processing means for converting the extracted image signal based on a gamma conversion process selected based on an information range of the image signal; 6. The image processing device according to any one of items 1 to 5, further comprising: (Item 7) The condition control means sets a bit range for cutting out the image signal as the information range. 7. The image processing device according to any one of items 1 to 6, wherein: (Item 8) The condition calculation means calculates the frequency of occurrence for each of a plurality of exposure times obtained by dividing a maximum exposure time, which are the plurality of exposure conditions. 8. The image processing device according to any one of items 1 to 7, wherein: (Item 9) The condition control means sets the information range based on the occurrence frequency calculated for each of the plurality of exposure times. 9. The image processing device according to item 8, (Item 10) The cutting means cuts out a part of the linearity-corrected image signal based on the information range. 10. The image processing device according to any one of items 1 to 9, wherein: (Item 11) further comprising a threshold setting means for setting a frequency threshold for each output setting of the image signal; The condition control means sets the information range based on the appearance frequency and the frequency threshold. 11. The image processing device according to any one of items 1 to 10, wherein: (Item 12) The acquiring means acquires, for each pixel, the exposure condition information indicating one of the plurality of exposure conditions under which a count value corresponding to photons incident on a pixel of the image exceeds a count threshold value that is the predetermined judgment condition. 12. The image processing device according to any one of items 1 to 11, wherein: (Item 13) The condition control means sets an unnecessary range on the upper side of the information range and an unnecessary range on the lower side of the information range. 13. The image processing device according to any one of items 1 to 12, wherein: (Item 14) the image processing device according to item 1; an output means for outputting a pulse signal according to the number of incident photons; a counter means for counting the number of the pulse signals and outputting a count value; a determination means for outputting the exposure condition information when the count value satisfies the determination condition under any of the plurality of exposure conditions; An imaging device comprising: (Item 15) acquiring exposure condition information for each pixel of the image, the exposure condition information indicating whether a predetermined determination condition is satisfied for each of a plurality of exposure conditions; calculating the number of times the exposure condition information appears for each of the plurality of exposure conditions as an appearance frequency; setting an information range for extracting an image signal based on the frequency of appearance; extracting a portion of the image signal of the image based on the information range; An image processing method comprising: (Item 16) 14. A program for causing a computer to function as each means of the image processing device according to any one of items 1 to 13.

[0124] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0125] REFERENCE SIGNS LIST 100, 1100, 1300...imaging device, 120, 1120, 1320...image processing device, 201...pixel, 215...control unit, 301...APD, 310...waveform shaping unit, 311...counter circuit, 312...count determination circuit, 121...image signal acquisition unit, 122, 1122...condition calculation unit, 123...condition control unit, 124...threshold setting unit, 126...data conversion unit, 127...bit extraction unit, 1200...image, 1201...central region, 1202...peripheral region.

Claims

1. an acquisition means for acquiring exposure condition information for each pixel of an image, the exposure condition information indicating whether a predetermined determination condition is satisfied for each of a plurality of exposure conditions; a condition calculation means for calculating the number of times that the exposure condition information appears for each of the plurality of exposure conditions as an appearance frequency; a condition control means for setting an information range for extracting an image signal based on the frequency of appearance; a cutting means for cutting out a part of an image signal of the image based on the information range; An image processing device comprising:

2. further comprising an output means for converting the extracted image signal based on an output format and outputting the converted image signal; The condition control means sets the information range based on the output format.

2. The image processing device according to claim 1, wherein:

3. The condition calculation means calculates the appearance frequency based on weighting factors set for each of a plurality of regions of the image.

2. The image processing device according to claim 1, wherein:

4. a condition acquisition means for accepting and setting the settings of the plurality of areas from a user; The image processing device according to claim 3 , further comprising:

5. The condition acquisition means receives and changes weighting factors for the plurality of regions from a user.

5. The image processing device according to claim 4.

6. an image processing means for converting the extracted image signal based on a gamma conversion process selected based on an information range of the image signal; The image processing device according to claim 1 , further comprising:

7. The condition control means sets a bit range for cutting out the image signal as the information range.

2. The image processing device according to claim 1, wherein:

8. The condition calculation means calculates the frequency of occurrence for each of a plurality of exposure times obtained by dividing a maximum exposure time, which are the plurality of exposure conditions.

2. The image processing device according to claim 1, wherein:

9. The condition control means sets the information range based on the occurrence frequency calculated for each of the plurality of exposure times.

9. The image processing device according to claim 8,

10. The cutting means cuts out a part of the linearity-corrected image signal based on the information range.

2. The image processing device according to claim 1, wherein:

11. further comprising a threshold setting means for setting a frequency threshold for each output setting of the image signal; The condition control means sets the information range based on the appearance frequency and the frequency threshold.

2. The image processing device according to claim 1, wherein:

12. The acquiring means acquires, for each pixel, the exposure condition information indicating one of the plurality of exposure conditions under which a count value corresponding to photons incident on a pixel of the image exceeds a count threshold value that is the predetermined judgment condition.

2. The image processing device according to claim 1, wherein:

13. The condition control means sets an unnecessary range on the upper side of the information range and an unnecessary range on the lower side of the information range.

2. The image processing device according to claim 1, wherein:

14. The image processing device according to claim 1 ; an output means for outputting a pulse signal according to the number of incident photons; a counter means for counting the number of the pulse signals and outputting a count value; a determination means for outputting the exposure condition information when the count value satisfies the determination condition under any of the plurality of exposure conditions; An imaging device comprising:

15. acquiring exposure condition information for each pixel of the image, the exposure condition information indicating whether a predetermined determination condition is satisfied for each of a plurality of exposure conditions; calculating the number of times the exposure condition information appears for each of the plurality of exposure conditions as an appearance frequency; setting an information range for extracting an image signal based on the frequency of appearance; extracting a portion of the image signal of the image based on the information range; An image processing method comprising:

16. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 13.

Images