Photoelectric conversion device, image processing apparatus, movable body, processing method, and computer program

Abstract

To provide a technique that can prevent a reduction in image recognition accuracy.SOLUTION: A photoelectric conversion device has a plurality of pixels each including a sensor part that emits a pulse at a frequency according to the light reception frequency of photons, and a counter that counts the number of the pulses. The plurality of pixels are controlled by a camera control unit 605 so as to generate signals in storage operations in a plurality of different storage periods in a full-frame period on the basis of the difference in the count value of the counter between at the start and the end of the storage periods, and output the signals generated in a first storage period from the end of the first storage period until the end of a second storage period. An image processing unit 603 generates a mapping parameter, and performs tone mapping processing on image signals output from the plurality of pixels by using the mapping parameter.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to a technique for image processing in a photoelectric conversion device. [Background technology]

[0002] In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal. For example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap with each other, thereby enabling continuous shooting even in low illuminance.

[0003] Meanwhile, vehicle-mounted camera systems often use HDR (Expanded Dynamic Range). The pixel bit depth in the image signal acquired by the HDR sensor becomes deeper, and the transmission rate becomes enormous. Since the transmission bandwidth of the image signal is limited, many sensors are equipped with tone compression methods such as tone mapping. As a parameter for tone compression, a feature of viewing applications is that the compression rate on the high luminance side is high, taking into account the characteristics of human vision.

[0004] In-vehicle cameras are used for the purposes of autonomous driving and collision avoidance, so it is necessary to perform image recognition processing as quickly as possible and prevent accidents by controlling the vehicle based on the recognition results. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 7223070 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the case of conventional image gradation compression methods, the compression rate is often set high, especially on the high luminance side. When performing image recognition processing with such a compression rate setting, there is a risk of the recognition accuracy decreasing because the gradation information is reduced in pixel areas with luminance that have a high compression rate.

[0007] The present invention provides a technique capable of suppressing a decrease in image recognition accuracy. [Means for solving the problem]

[0008] A photoelectric conversion device according to one embodiment of the present invention is characterized in that it comprises a plurality of pixels, each of which has a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons and a counter that counts the number of the pulses, a control means that controls the plurality of pixels to generate a signal based on a difference in count values ​​of the counter at the start and end of a plurality of different accumulation periods within a full frame period during accumulation operation, and to output a signal generated in a first accumulation period between the end of the first accumulation period and the end of a second accumulation period, and a processing means that generates mapping parameters and performs tone mapping processing on image signals output from the plurality of pixels using the mapping parameters. Effect of the Invention

[0009] According to the present invention, it is possible to suppress a decrease in image recognition accuracy. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment of the present invention. [Diagram 2] FIG. 4 is a diagram illustrating a configuration example of a sensor substrate. [Diagram 3] FIG. 2 is a diagram illustrating an example of a configuration of a circuit board. [Figure 4] 4 is a diagram showing an equivalent circuit of a pixel and a signal processing circuit corresponding to the pixel in FIG. 2 and FIG. 3. FIG. [Diagram 5] 4 is a diagram showing a relationship between a memory circuit and a buffer according to the embodiment; [Figure 6] 1 is a diagram illustrating a relationship between the operation of an APD and an output signal. [Figure 7] 1 is a functional block diagram of a photoelectric conversion device and a moving object according to an embodiment. [Figure 8] 2 is a block diagram showing an example of a functional configuration of an image processing unit according to the embodiment; [Figure 9] 11A to 11C are diagrams for explaining a photoelectric conversion method by a camera control unit according to an embodiment. [Figure 10] FIG. 13 is a diagram showing an example of an image divided into a plurality of frames. [Figure 11] 1 is a flowchart illustrating a tone mapping process according to an embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of a histogram and mapping parameters. [Figure 13] 2 is a block diagram showing an example of a functional configuration of an image processing unit according to the embodiment; [Figure 14] 1 is a flowchart illustrating a tone mapping process according to an embodiment. [Figure 15] 11 is a diagram showing an example of an image that has been subjected to region division based on a recognition result of a recognition unit; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, parts having the same configuration or function are given the same reference numerals, and repeated explanations thereof will be omitted. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0012] First Embodiment 1 is a diagram showing a configuration example of a photoelectric conversion element according to the first embodiment. In the following, a photoelectric conversion device having a so-called stacked structure in which a photoelectric conversion element 100 is configured by stacking and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21, will be described as an example.

[0013] However, a so-called non-laminated structure in which the components included in the sensor substrate and the components included in the circuit substrate are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD).

[0015] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array that constitutes the pixel region 12 is not particularly limited.

[0016] Fig. 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0017] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0018] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. As described later, the signal processing circuit 103 is provided with a counter (counter circuit 211 in FIG. 4) and a memory (memory circuit 212 in FIG. 4), and the like, and digital values ​​are stored in the memory. The horizontal scanning circuit 111 inputs a control pulse to the signal processing circuit 103 for sequentially selecting each column in order to read out a signal from the memory of each pixel in which a digital signal is stored.

[0019] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein and connected to the vertical signal line 113.

[0020] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view.

[0021] That is, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0022] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and a configuration may be adopted in which one signal processing unit is shared by a plurality of photoelectric conversion units and performs signal processing sequentially.

[0023] FIG. 4 is a diagram showing an equivalent circuit of the pixel 101 in FIG. 2 and FIG.

[0024] The APD 201 included in the photoelectric conversion unit 102 generates a pair of electric charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. The other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0025] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0026] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0027] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0028] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0029] FIG. 4 shows an example in which the signal processing circuit 103 includes a waveform shaping section 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0030] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example in which one inverter is used as the waveform shaping unit 210 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0031] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0032] 3 to the memory circuit 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0033] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the driving voltage VH or the driving voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0034] Fig. 5 is a diagram showing the relationship between the memory circuits and buffers according to the embodiment. Fig. 5 shows a state in which the memory circuits 212 in the signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory circuit is represented as memory 1-1 to memory NM. Also, buffer 1 to buffer M in Fig. 5 indicate buffers included in the read circuit 112 in Fig. 3. Output circuit 114 in Fig. 5 corresponds to output circuit 114 in Fig. 3.

[0035] 6 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0036] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0037] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 210 and is output as a pulse signal at nodeB.

[0038] Next, a photoelectric conversion device 600 and a moving object 700 according to an embodiment will be described. Fig. 7 is a functional block diagram of the photoelectric conversion device 600 and the moving object 700 according to the embodiment. Some of the functional blocks shown in Fig. 7 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 and the moving object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0039] However, some or all of them may be realized by hardware. For the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used. Furthermore, the respective functional blocks shown in Fig. 7 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 7 also applies to Figs. 8 and 13 described later.

[0040] 1 to 6, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, etc. The photoelectric conversion element 100 is composed of an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0041] The photoelectric conversion device 600 of the embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is configured to capture an image in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0042] The image processing unit 603 is a block that performs necessary image processing on the image signal acquired by the photoelectric conversion element 100, and functions as an image processing device. In this embodiment, the image processing unit 603 performs, in addition to tone mapping processing described later, processes such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, white balance adjustment, and color conversion to generate a final image.

[0043] Moreover, the output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving body 700, and a camera control unit 605. The recognition unit 604 recognizes people, vehicles, etc. in the vicinity by performing image recognition based on the image signal, and issues a warning, etc. as necessary.

[0044] In this embodiment, the moving body 700 is described using an example of an automobile, but the moving body may be any moving body, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0045] The camera control unit 605 has a built-in CPU as a computer and a memory that stores a computer program, and controls each unit of the photoelectric conversion device 600 by the CPU executing the computer program stored in the memory.

[0046] The camera control unit 605 functions as a control means, and controls the length of the accumulation period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK via a control pulse generation unit for the photoelectric conversion element 100, for example.

[0047] The storage unit 606 includes a recording medium such as a memory card, a hard disk, etc., and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signal to the outside of the photoelectric conversion device 600 and receives various signals from the outside.

[0048] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0049] The output of ECU 701 is supplied to vehicle control unit 702 and display unit 703. Vehicle control unit 702 functions as a movement control means for driving, stopping, controlling the direction, etc. of the vehicle as a moving object based on the output of ECU 701. Display unit 703 functions as a display means, includes display elements such as a liquid crystal device or an organic EL, and is mounted on moving object 700. Vehicle control unit 702 as a movement control means controls the operation of the moving object based on the result of recognition processing using signals generated at least during the first accumulation period.

[0050] The display unit 703 displays images acquired by the photoelectric conversion element 100 and various information related to the vehicle's running state and the like to the driver of the vehicle 700 using, for example, a GUI based on the output of the ECU 701 .

[0051] In addition, the image processing unit 603, the recognition unit 604, etc. in Figure 7 do not have to be mounted on the moving body 700, and may be provided, for example, in an external terminal provided separately from the moving body 700 for remotely controlling the moving body 700 or for monitoring the movement of the moving body.

[0052] Fig. 9 is a diagram for explaining a photoelectric conversion method by the camera control unit 605 according to the embodiment. In this embodiment, photoelectric conversion is performed periodically at, for example, 30 full frames / second (the time length of one frame is 33.3 ms), and one frame having a length of 33.3 ms is divided into four. That is, as shown in Fig. 9, frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 of equal periods (8.33 ms).

[0053] In this embodiment, one frame having a length of 33.3 ms is called a full frame, and a full frame divided into four is called a frame. In addition, in the figures from FIG. 9 onwards, frames 0_3, 0_4, 1_1, 1_2, 1_3, 1_4, 2_1, and 2_2 are represented as F0_3, 0_4, 1_1, 1_2, 1_3, 1_4, 2_1, and 2_2.

[0054] The camera control unit 605 performs accumulation operations (accumulation operations of signals corresponding to the count value of the counter circuit 211) for a plurality of different accumulation periods within a full frame period. For example, frame 1_1 has an accumulation period from time T0 to time T1 of full frame 1, and frame 1_2 has an accumulation period from time T0 to time T2. Frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4. In this embodiment, for example, the accumulation period from time T0 to time T1 is called a first accumulation period, and the accumulation period from time T0 to time T4 is called a second accumulation period.

[0055] At times T1, T2, T3, and T4, which are the end points of frames 1_1, 1_2, 1_3, and 1_4, respectively, count values ​​C1_1, C1_2, C1_3, and C1_4 are read out from the counter circuit 211. Meanwhile, the counter circuit 211 is reset by the above-mentioned control pulse RES at T0 (T4), which is the end point of a full frame period (one vertical period).

[0056] The read count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in the memory circuit 212. Then, the signals for one row temporarily stored in the memory circuit 212 are sequentially output from the photoelectric conversion element 100 via the buffer of the read circuit 112, and input to the image processing unit 603. The buffer stores the count values ​​C1_1, C1_2, C1_3, and C1_4 for rows 1 to N (i.e., one frame).

[0057] In this way, according to this embodiment, the signals accumulated during the period of frame 1_1 are read out from time T1 to T2 and processed by the recognition unit 604. Therefore, image recognition can be performed quickly. Similarly, the signals accumulated during the periods of frames 1_2, 1_3, and 1_4 are sequentially read out from time T2 to T3, T3 to T4, and T4 to T1, respectively, and image recognition can be performed repeatedly.

[0058] Fig. 10 is a diagram showing an example of images of a plurality of divided frames. As shown in Fig. 10, the image of frame 1_1 is dark because the accumulation period is short, but there is little subject blur of a person who jumps out in front of the camera. On the other hand, the accumulation periods of frames 1_2, 1_3, and 1_4 are longer in that order, so subject blur is more likely to occur. Note that stopped vehicles and white lines are less likely to be blurred, and the longer the accumulation period, the more likely the contrast is to be improved.

[0059] In this manner, in this embodiment, accumulation operations are performed for the first and second accumulation periods within one full frame. Also, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period.

[0060] In this embodiment, the first and second accumulation periods overlap, and start at the same time. Furthermore, the end of the second accumulation period is a full frame boundary, and the second accumulation period is an integer multiple of the first accumulation period. That is, the end of the second accumulation period coincides with the end of the full frame.

[0061] However, the second accumulation period does not need to be an integer multiple of the first accumulation period. The second accumulation period only needs to be longer than the first accumulation period (the first accumulation period is shorter than the second accumulation period) and the second accumulation period only needs to end after the first accumulation period.

[0062] That is, an image with a short accumulation period and an image with a long accumulation period are created, the timing at which the short accumulation period ends is set earlier than the timing at which the long accumulation period ends, and as soon as the short accumulation period ends, the image is output and sent to the downstream recognition unit 604. Then, the subject is recognized based on a signal generated during at least the first accumulation period. The recognition unit 604 as a recognition means recognizes the subject based on a signal generated during at least the first accumulation period.

[0063] Therefore, while in the past image recognition was not possible until one full frame had passed, in this embodiment image recognition can be performed after as little as 1 / 4 full frame period, and for example, when a moving body is moving at high speed, obstacles and the like can be recognized quickly. Therefore, the moving body can apply the brakes quickly, or obstacles can be avoided early.

[0064] In addition, an image with a long accumulation period can improve the contrast, and can be used as a display image. That is, an image with a short accumulation period is suitable for quick subject recognition, and an image with a long accumulation period is suitable as a display image. In this way, the display device of this embodiment displays the signal generated during at least the second accumulation period as an image.

[0065] In addition, since the present embodiment uses an APD, unlike a CMOS sensor, the accumulated charge does not deteriorate due to readout, so the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not deteriorate even if it is read out many times after one accumulation.

[0066] In this embodiment, the camera control unit 605 is an example of a control means that controls a plurality of pixels as follows. That is, the control means controls the plurality of pixels to generate a signal based on the difference between the count values ​​at the start and end of a plurality of different accumulation periods in an accumulation operation within a full frame period. The control means also controls the plurality of pixels to output a signal generated in a first accumulation period between the end of the first accumulation period and the end of a second accumulation period.

[0067] 8 is a block diagram mainly showing an example of the functional configuration of the image processing unit 603. The image processing unit 603 has a histogram generation unit 6031, a tone mapping processing unit 6032, a histogram estimation unit 6033, a mapping parameter determination unit 6034, and a mapping parameter generation unit 6035.

[0068] The histogram generating unit 6031 generates a histogram of pixel values ​​for the image signal in the first accumulation period acquired by the photoelectric conversion element 100. The histogram generating unit 6031 outputs the generated histogram of pixel values ​​to the histogram estimating unit 6033.

[0069] The tone mapping processing unit 6032 performs a gradation mapping process on the image signal of the first accumulation period acquired by the photoelectric conversion element 100. At this time, the tone mapping process is performed using the mapping parameters generated by the mapping parameter generation unit 6035. The mapping parameters are information indicating a compression rate defined for each luminance (or pixel value) of each pixel. This will be described in detail later in FIG. 12. By performing the tone mapping process, it is possible to compress the gradation of the image signal. The image signal output from the tone mapping processing unit 6032 is output to the recognition unit 604.

[0070] In this embodiment, the histogram generation unit 6031 and the tone mapping unit 6032 are arranged in parallel, but the tone mapping unit 6032 may be arranged before the histogram generation unit 6031. In that case, it is preferable not to perform tone mapping processing that reduces the dynamic range.

[0071] The histogram estimation unit 6033 estimates a histogram of pixel values ​​in the second accumulation period from the histogram of pixel values ​​in the first accumulation period generated by the histogram generation unit 6031 and the length of the second accumulation period. Here, the histogram estimation unit 6033 estimates pixel values ​​in the second accumulation period by multiplying pixel values ​​in the first accumulation period by the ratio between the first accumulation period and the second accumulation period, and can infer the histogram. The histogram generated here is output to the mapping parameter determination unit 6034 and the mapping parameter generation unit 6035.

[0072] The mapping parameter determination unit 6034 compares the mapping parameters used in the tone mapping processing unit 6032 with the histogram of pixel values ​​in the second accumulation period estimated by the histogram estimation unit 6033. In this way, the mapping parameter determination unit 6034 determines whether the mapping parameters used in the tone mapping process are appropriate mapping parameters.

[0073] An example of a method for determining whether a mapping parameter is appropriate is as follows. The compression rate of the luminance of each pixel (compression rate according to the mapping parameter to be determined) is compared with the frequency of pixel values ​​of each luminance in an estimated histogram, and if a frequency equal to or exceeds a threshold occurs at a luminance with a high compression rate, the mapping parameter is determined to be inappropriate. The mapping parameter determination unit 6034 outputs the generated determination result to the mapping parameter generation unit 6035.

[0074] The mapping parameter generation unit 6035 generates mapping parameters for the second accumulation period based on the determination result output from the mapping parameter determination unit 6034 and the histogram for the second accumulation period output from the histogram estimation unit 6033. At this time, if mapping parameters are generated that reduce the compression rate of gradations with high pixel value frequency, loss of pixel values ​​of the subject component is reduced. The mapping parameters generated by the mapping parameter generation unit 6035 are output to the camera control unit 605.

[0075] The camera control unit 605 sends a control signal to each functional block. The camera control unit 605 receives mapping parameters from the mapping parameter generation unit 6035 described above, and outputs them to the tone mapping processing unit 6032 and the mapping parameter determination unit 6034. The camera control unit 605 also outputs information on the exposure times of the first accumulation period and the second accumulation period to the histogram estimation unit 6033.

[0076] In this embodiment, the image processing unit 603 is an example of a processing means that generates mapping parameters and performs tone mapping processing on the output signal using the mapping parameters.

[0077] 11 is a flowchart showing tone mapping processing according to the first embodiment. This processing is realized by the CPU of the camera control unit 605 reading out a computer program corresponding to the processing content, loading it into memory, and executing it.

[0078] In step S101, the camera control unit 605 sets initial values ​​of tone mapping parameters. At this time, the initial values ​​of the tone mapping parameters may be values ​​determined in advance or may be values ​​used in the previous frame.

[0079] In step S102, the camera control unit 605 acquires an image of frame 1_n by the photoelectric conversion element 100. Here, frame 1_n represents frames 1_1 to 1_4 in FIG.

[0080] In step S103, the histogram generating unit 6031 generates a histogram of pixel values ​​using the acquired image signal.

[0081] In step S104, the tone mapping unit 6032 performs tone mapping using the tone mapping parameters set in step S101, thereby compressing the bit depth (number of quantization bits or number of gradations) of the image.

[0082] In step S105, the camera control unit 605 checks the frame currently being processed. If the frame currently being processed is frame 1_1, 1_2, or 1_3, the process proceeds to step S106. If it is frame 1_4, the process ends and moves to the next frame 2_1. Here, the maximum value of n is an example, and a value other than n=4 may be used.

[0083] In step S106, the histogram estimation unit 6033 estimates a histogram of pixel values ​​in the image of frame 1_n+1 from the histogram generated in step S103 and information on the exposure time (accumulation period) of frame 1_n+1.

[0084] In step S107, the mapping parameter determination unit 6034 compares the histogram estimate generated in step S106 with the mapping parameters used in step S104 to determine whether they are appropriate. If they are appropriate, the process proceeds to step S108. If they are not appropriate, the process proceeds to step S109.

[0085] In step S108, since it has been determined in step S107 that the current mapping parameters are appropriate, the mapping parameters are not updated.

[0086] In step S109, since it is determined in step S107 that the current mapping parameters are inappropriate, the mapping parameter generating unit 6035 generates new mapping parameters based on the estimated values ​​of the histogram generated in step S106. That is, the mapping parameters are updated. Here, as the generated mapping parameters, parameters are set so that the gradation is compressed at a low rate in the high frequency part of the histogram in order to leave the gradation information of the image as much as possible.

[0087] Fig. 12 is a diagram showing an example of a histogram and mapping parameters, and is a diagram for explaining a method of generating mapping parameters from a histogram. Fig. 12(a) and (b) show an example of a histogram of pixel values. Fig. 12(c) shows an example of mapping parameters generated based on the histogram of Fig. 12(a). Fig. 12(d) shows an example of mapping parameters generated based on the histogram of Fig. 12(b).

[0088] In the case of a high-brightness subject, a histogram like that shown in Fig. 12(a) is generated. In Fig. 12(a), there are many pixel values ​​on the high-brightness side. In this case, mapping parameters are generated that lower the compression rate on the high-brightness side, as shown in Fig. 12(c). As a reference for understanding Figs. 12(c) and (d), in the case of parameters that have a uniform compression rate across the entire range of input luminance, the graph showing the correspondence between input luminance and output luminance will be linear (a proportional line).

[0089] On the other hand, when the subject is low-luminance, a histogram like that shown in Fig. 12(b) is generated. In Fig. 12(b), there are many pixel values ​​on the low-luminance side, so mapping parameters are generated to lower the compression rate on the low-luminance side, as shown in Fig. 12(d).

[0090] As described above, as an example of a method for determining whether a mapping parameter is appropriate, the compression rate of the luminance of each pixel is compared with the frequency of the pixel value of each luminance, and if the frequency of occurrence of a luminance with a high compression rate is equal to or higher than a threshold, the mapping parameter is determined to be inappropriate. From Fig. 12, it can be seen that for a certain mapping parameter, if the frequency of occurrence of a luminance with a high compression rate is equal to or higher than a threshold in the histogram, the mapping parameter is inappropriate.

[0091] In the case of conventional image gradation compression methods, the compression rate is often set high, especially on the high-luminance side. When performing image recognition processing with such a compression rate setting, gradation information is reduced in pixel areas with high compression rate luminance, which may result in blurred contours of subjects in high-luminance areas of the image, resulting in a decrease in the recognition accuracy of the subject. On the other hand, a low compression rate results in a deep bit depth, which increases delays in transmission and recognition processing, resulting in delayed vehicle control.

[0092] In contrast, in this embodiment, the image signal is output as soon as the first accumulation period, which is shorter than the full frame period, ends. The image processing unit 603 generates mapping parameters from a histogram of pixel values ​​in the first accumulation period, and performs tone mapping processing on the image signal in the second accumulation period using the mapping parameters. That is, the image processing unit 603 and the recognition unit 604 can start processing the image signal compressed to low bits by the tone mapping processing before the accumulation period of one full frame ends. This makes it possible to achieve low latency in image processing and image recognition. The mapping parameter generation unit 6035 also generates mapping parameters that reduce the compression rate of gradations with high pixel value frequencies. Therefore, it is possible to leave as much gradation information of the image signal, such as subject components, as possible in the tone mapping processing (suppressing loss of gradations). This makes it possible to suppress a decrease in the accuracy of the recognition processing by the recognition unit 604. That is, in this embodiment, it is possible to achieve low latency processing while suppressing a decrease in image recognition accuracy.

[0093] <Second embodiment> The second embodiment will be described below. In the second embodiment, the description of elements and functions similar to those of the first embodiment will be omitted or simplified, and differences will be mainly described. Fig. 13 is a block diagram mainly showing an example of the functional configuration of an image processing unit 603. The image processing unit 603 further has a region dividing unit 6036 in addition to the configuration of the image processing unit 603 of the first embodiment.

[0094] The area division unit 6036 divides the image corresponding to the signal output from the photoelectric conversion element 100 into areas. Here, the areas used are those divided based on the recognition results output from the recognition unit 604 described below. Fig. 15 shows an example of an image in which area division has been performed based on the recognition results of the recognition unit 604. Area 1401 in the image is an area in which a vehicle is recognized. Area 1402 is an area in which a person is recognized. Area 1403 is an area within the angle of view excluding areas 1401 and 1402.

[0095] The mapping parameter generation unit 6035 generates mapping parameters for each of the regions 1401, 1402, and 1403. The image signal output from the region division unit 6036 is output to the tone mapping processing unit 6032 and the histogram generation unit 6031.

[0096] The histogram generating unit 6031 generates a histogram of pixel values ​​in the first accumulation period using the image signal output from the region dividing unit 6036. At this time, the histogram generating unit 6031 generates a histogram of pixel values ​​for each divided region. The generated histogram for each divided region is output to the histogram estimating unit 6033. The divided regions may be referred to as divided regions hereinafter.

[0097] The tone mapping processing unit 6032 performs a gradation mapping process on the image signal output from the region dividing unit 6036. At this time, the tone mapping processing unit 6032 performs a tone mapping process for each divided region using the region output from the region dividing unit 6036 and the mapping parameters output from the mapping parameter generating unit 6035. This compresses the bit depth of the image. The image signal output from the tone mapping processing unit 6032 is output to the recognition unit 604.

[0098] The recognition unit 604 performs recognition processing on the image processed by the tone mapping processing unit 6032. The recognition result generated by the recognition processing includes information such as the position of the subject, the recognition accuracy of the subject, the type of the subject, etc. The generated recognition result is output to the camera control unit 605 and the ECU 701.

[0099] The camera control unit 605 outputs control signals to each functional block, receives mapping parameters, outputs the mapping parameters, and outputs information on exposure time, and also receives the image recognition result from the recognition unit 604. Then, the camera control unit 605 outputs the recognition result to the area division unit 6036 for area division processing by the area division unit 6036.

[0100] 14 is a flowchart showing tone mapping processing according to the second embodiment. This processing is realized by the CPU of the camera control unit 605 reading out a computer program corresponding to the processing content, loading it into memory, and executing it.

[0101] 14, the camera control unit 605 sets the initial values ​​of the mapping parameters and the initial divided regions of the image signal. At this time, the initial values ​​of the mapping parameters and the initial divided regions may be values ​​and regions determined in advance, or may be values ​​and regions used in the previous frame.

[0102] Next, in step S202, the camera control unit 605 acquires an image of frame 1_n with the imaging element.

[0103] In step S203, the area division unit 6036 performs area division on the acquired image. At this time, if a recognition result has been output in step S211 (described later), the area division is performed according to the subject position of the recognition result.

[0104] In step S204, the histogram generating unit 6031 generates a histogram of pixel values ​​using the acquired image signal. At this time, the histogram generating unit 6031 generates a histogram of pixel values ​​for each of the regions divided in step S203.

[0105] In step S205, the tone mapping processing unit 6032 performs tone mapping processing using the set mapping parameters. At this time, the tone mapping processing is performed for each area divided in step S203. By performing the tone mapping processing, the bit depth of the image can be compressed.

[0106] Step S206 is similar to step S105 shown in FIG. 11, and therefore a description thereof will be omitted.

[0107] In step S207, the histogram estimation unit 6033 estimates a histogram of pixel values ​​in the image of frame 1_n+1 from the histogram generated in step S204 and the length of the exposure time (accumulation time) of frame 1_n+1. At this time, the histogram is estimated for each divided area.

[0108] In step S208, the mapping parameter determination unit 6034 compares the estimated value of the histogram generated in step S207 with the mapping parameters used in step S205, and determines whether the parameters are appropriate for each divided region. If the parameters are appropriate for all divided regions, the process proceeds to step S209. If any divided region is inappropriate, the process proceeds to step S210.

[0109] In step S209, since it has been determined in step S208 that the current mapping parameters are appropriate for all regions, the mapping parameters are not updated.

[0110] In step S210, since it was determined in step S208 that there is an area where the current parameters are inappropriate, the mapping parameter generating unit 6035 generates (updates) new mapping parameters based on the histogram estimation values ​​generated in step S207. Here, mapping parameters are generated for each divided area.

[0111] In step S211, the recognition unit 604 performs recognition processing on the acquired image. The area division unit 6036 sets the subject area output from the recognition unit 604 as the recognition result as a divided area in frame 1_n+1 (updates the divided area). The area division unit 6036 is an example of a processing means that divides the area of ​​the image based on the recognition result by the recognition unit 604.

[0112] As described above, in this embodiment, it is possible to obtain the same effects as in the first embodiment, and to generate more appropriate mapping parameters for each divided region according to the recognition result of the recognition unit 604, and apply these to the tone mapping process. As a result, the tone mapping processing unit 6032 can perform highly accurate tone mapping process for each divided region. That is, it is possible to enhance the effect of suppressing loss of gradation in necessary image signals such as subject components. This makes it possible to improve the accuracy of the recognition process by the recognition unit 604.

[0113] <Other embodiments> For example, in the above embodiment, the shortest accumulation period (accumulation period of frame 1_1) is ¼ full frame period as shown in FIG. 9, but it may be changed to, for example, ⅕ full frame period or ⅓ full frame period.

[0114] Furthermore, even if the readout cycle is set to every 1 / 4 full frame period, the counter circuit may be reset midway through the accumulation period of frame 1_1 in Fig. 9 depending on the brightness of the subject, the image recognition accuracy, etc. In this way, the substantial accumulation period may be made shorter than the 1 / 4 full frame period.

[0115] Alternatively, the counter circuit may be reset once at time T1 in Fig. 9. This may adjust the count value read out at time T4.

[0116] In the process shown in Fig. 11, the histogram generation is performed first and then the tone mapping process (steps S103 and S104), but the order may be reversed. The same applies to the process shown in Fig. 14 (steps S204 and S205).

[0117] In the above second embodiment, the region dividing unit 6036 performed region division on all frames acquired from the photoelectric conversion element 100, but region division may be performed on each frame at a fixed period, for example, on every other frame.

[0118] 7, the image processing unit 603 is provided as a part of the functional block of the photoelectric conversion device 600. However, the image processing unit 603 and the photoelectric conversion device 600 may be separate devices and connected to each other by wire or wirelessly.

[0119] A computer program for implementing the functions of the above-described embodiment as part or all of the control in this embodiment may be provided to the photoelectric conversion device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the photoelectric conversion device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0120] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A plurality of pixels each including a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving a photon and a counter that counts the number of the pulses; a control means for controlling the plurality of pixels to generate a signal based on a difference between count values ​​of the counter at the start and end of a plurality of different accumulation periods within a full frame period, and to output a signal generated during a first accumulation period from the end of the first accumulation period to the end of a second accumulation period; and a processing unit that generates a mapping parameter and performs a tone mapping process on image signals output from the plurality of pixels using the mapping parameter. (Configuration 2) 2. The photoelectric conversion device according to configuration 1, wherein the processing means generates a histogram of pixel values ​​based on the image signal, and generates the mapping parameters based on the histogram. (Configuration 3) The photoelectric conversion device according to configuration 2, wherein the processing means estimates a histogram of pixel values ​​in the second accumulation period based on a histogram of pixel values ​​in the first accumulation period and information on the second accumulation period, and generates the mapping parameters based on the estimated histogram of pixel values ​​in the second accumulation period. (Configuration 4) 4. The photoelectric conversion device according to configuration 3, wherein the processing means determines whether the mapping parameters are appropriate by comparing the generated mapping parameters with the estimated histogram. (Configuration 5) 5. The photoelectric conversion device according to any one of configurations 1 to 4, wherein the processing means divides an area of ​​an image corresponding to the image signal, and performs the tone mapping process for each of the areas. (Configuration 6) a recognition unit that recognizes a subject based on an image signal that corresponds to a signal generated during at least the first accumulation period and that has been subjected to the tone mapping process; 6. The photoelectric conversion device according to configuration 5, wherein the processing means divides an area of ​​the image based on a recognition result by the recognition means. (Configuration 7) The photoelectric conversion device according to any one of configurations 1 to 4, further comprising a recognition means for recognizing a subject based on an image signal that corresponds to a signal generated during at least the first accumulation period and that has been subjected to the tone mapping process. (Configuration 8) 7. The photoelectric conversion device according to any one of configurations 2 to 6, wherein the processing means performs the tone mapping process after generating the histogram. (Configuration 9) 9. The photoelectric conversion device according to any one of configurations 1 to 8, wherein the first accumulation period is shorter than the second accumulation period. (Configuration 10) 10. The photoelectric conversion device according to any one of configurations 1 to 9, wherein the first accumulation period and the second accumulation period overlap each other. (Configuration 11) 11. The photoelectric conversion device according to any one of configurations 1 to 10, wherein the first accumulation period and the second accumulation period start simultaneously. (Configuration 12) 12. The photoelectric conversion device according to any one of configurations 1 to 11, wherein an end time of the second accumulation period coincides with an end time of the full frame. (Configuration 13) 13. The photoelectric conversion device according to any one of configurations 1 to 12, further comprising a display means for displaying, as an image, at least a signal generated during the second accumulation period. (Configuration 14) An image processing device that processes an image signal from a photoelectric conversion device having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, and a counter that counts the number of the pulses, the photoelectric conversion device has a control means for controlling the plurality of pixels to generate a signal based on a difference between count values ​​of the counter at the start and end of a plurality of different accumulation periods within a full frame period in an accumulation operation of the plurality of different accumulation periods, and to output a signal generated during a first accumulation period from the end of the first accumulation period to the end of a second accumulation period; 13. An image processing device comprising: a processing unit that generates a mapping parameter and performs a tone mapping process on image signals output from the plurality of pixels using the mapping parameter. (Configuration 15) The photoelectric conversion device according to any one of configurations 1 to 13, and a movement control means for controlling the movement of the moving body based on a result of a recognition process using a signal generated during at least the first accumulation period. (method) A processing method for a photoelectric conversion device having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving a photon, and a counter that counts the number of the pulses, comprising: A step of controlling the plurality of pixels so as to generate a signal based on a difference between count values ​​of the counter at the start and end of a plurality of different accumulation periods in an accumulation operation within a full frame period, and to output a signal generated during a first accumulation period from the end of the first accumulation period to the end of a second accumulation period; generating mapping parameters; and performing tone mapping on image signals output from the plurality of pixels using the mapping parameters. (program) A computer program for controlling each means of the photoelectric conversion device according to any one of configurations 1 to 13, the image processing device according to configuration 14, or the moving body according to configuration 15, by a computer.

[0121] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention. [Explanation of symbols]

[0122] 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 114: Output circuit 115: Control pulse generation unit 211: Counter circuit 212: Memory circuit 600: Photoelectric conversion device 603: Image processing unit 604: Recognition part 605: Camera control unit 703: Display section 1401~1403: Area (divided area) 6031: Histogram generation unit 6032: Tone mapping processor 6033: Histogram Estimation Unit 6034: Mapping parameter determination unit 6035: Mapping parameter generation unit 6036: Area division part

Claims

1. A plurality of pixels each including a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving a photon and a counter that counts the number of the pulses; a control means for controlling the plurality of pixels to generate a signal based on a difference between count values ​​of the counter at the start and end of a plurality of different accumulation periods within a full frame period, and to output a signal generated during a first accumulation period from the end of the first accumulation period to the end of a second accumulation period; and a processing unit that generates a mapping parameter and performs a tone mapping process on image signals output from the plurality of pixels using the mapping parameter.

2. 2. The photoelectric conversion device according to claim 1, wherein the processing means generates a histogram of pixel values ​​based on the image signal, and generates the mapping parameters based on the histogram.

3. 3. The photoelectric conversion device according to claim 2, wherein the processing means estimates a histogram of pixel values ​​in the second accumulation period based on a histogram of pixel values ​​in the first accumulation period and information on the second accumulation period, and generates the mapping parameters based on the estimated histogram of pixel values ​​in the second accumulation period.

4. 4. The photoelectric conversion device according to claim 3, wherein the processing means determines whether the generated mapping parameters are appropriate by comparing the generated mapping parameters with the estimated histogram.

5. 2. The photoelectric conversion device according to claim 1, wherein the processing means divides an image corresponding to the image signal into regions, and performs the tone mapping process for each of the regions.

6. a recognition unit that recognizes a subject based on an image signal that corresponds to at least a signal generated during the first accumulation period and that has been subjected to the tone mapping process; 6. The photoelectric conversion device according to claim 5, wherein the processing means divides the image into regions based on the recognition result by the recognition means.

7. 2. The photoelectric conversion device according to claim 1, further comprising a recognition means for recognizing a subject based on an image signal that corresponds to at least the signal generated during the first accumulation period and that has been subjected to the tone mapping process.

8. 3. The photoelectric conversion device according to claim 2, wherein the processing means performs the tone mapping process after generating the histogram.

9. 2. The photoelectric conversion device according to claim 1, wherein the first accumulation period is shorter than the second accumulation period.

10. 10. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period overlap each other.

11. 11. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period start simultaneously.

12. 2. The photoelectric conversion device according to claim 1, wherein the end of the second accumulation period coincides with the end of the full frame.

13. 2. The photoelectric conversion device according to claim 1, further comprising a display means for displaying, as an image, a signal generated during at least the second accumulation period.

14. An image processing device that processes an image signal from a photoelectric conversion device having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, and a counter that counts the number of the pulses, the photoelectric conversion device has a control means for controlling the plurality of pixels to generate a signal based on a difference between count values ​​of the counter at the start and end of a plurality of different accumulation periods within a full frame period in an accumulation operation of the plurality of different accumulation periods, and to output a signal generated during a first accumulation period from the end of the first accumulation period to the end of a second accumulation period; 13. An image processing device comprising: a processing unit that generates a mapping parameter and performs a tone mapping process on image signals output from the plurality of pixels using the mapping parameter.

15. The photoelectric conversion device according to any one of claims 1 to 13, and a movement control means for controlling the movement of the moving body based on a result of a recognition process using at least a signal generated during the first accumulation period.

16. A processing method for a photoelectric conversion device having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving a photon, and a counter that counts the number of the pulses, comprising: A step of controlling the plurality of pixels so as to generate a signal based on a difference between count values ​​of the counter at the start and end of a plurality of different accumulation periods within a full frame period in an accumulation operation, and to output a signal generated during a first accumulation period from the end of the first accumulation period to the end of a second accumulation period; generating mapping parameters; and performing tone mapping on image signals output from the plurality of pixels using the mapping parameters.

17. A computer program for controlling the photoelectric conversion device according to any one of claims 1 to 13, the image processing device according to claim 14, or each means of the moving body according to claim 15 by a computer.

Images