Imaging system, mobile body, imaging method, and computer program

Abstract

To provide an imaging system that reduces a load on recognition processing.SOLUTION: An imaging device has a plurality of pixels each having a photoelectric conversion unit 102 functioning as a sensor unit for emitting a pulse at a frequency corresponding to a frequency of receiving photons, and a signal processing circuit 103 having a counter circuit 211 for counting the number of pulses, generates a signal based on the difference between count values of a counter at a start time and an end time of an accumulation period, performs accumulation operations for a first accumulation period and a second accumulation period within one full frame, controls such that the first accumulation period is shorter than the second accumulation period, and a signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, performs image processing on a signal output from the pixel, performs image recognition on the signal that has been subjected to predetermined image processing, determines an image processing area based on a result of the image recognition, and performs image processing on a signal which is next output from the pixel based on the image processing area determined by an area determination unit.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an imaging system, a moving object, an imaging method, a computer program, and the like. [Background technology]

[0002] In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal. For example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap with each other, thereby enabling continuous shooting even in low illuminance.

[0003] When a camera outputs an image for recognition or viewing purposes, it often applies image processing such as noise reduction, scratch correction, and black level correction before outputting the image. By applying image processing, it is possible to output an image that is more suitable for recognition or viewing. In general, applying high-performance image processing has fewer drawbacks and allows for a more effective image to be output. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7223070 Summary of the Invention [Problem to be solved by the invention]

[0005] When used as an in-vehicle camera, the camera may be used for the purpose of autonomous driving, collision avoidance, etc. In such cases, it is desirable to perform recognition as quickly as possible and to control the vehicle quickly based on the recognition results.

[0006] However, when image processing is performed on the image output from the imaging element, the processing time increases and a delay may occur in outputting the recognition result. In particular, when the processing becomes complicated, the processing becomes heavy and the delay time increases, which may cause a delay in the control of the vehicle.

[0007] Therefore, an object of the present invention is to provide an imaging system capable of reducing the load on the recognition process. [Means for solving the problem]

[0008] An imaging system according to one aspect of the present invention comprises: A sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A plurality of pixels each including a counter for counting the number of the pulses; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control unit that performs an accumulation operation of a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controls so that a signal generated during the first accumulation period is output during a period from the end of the first accumulation period to the end of the second accumulation period; an image processing unit that performs image processing on the signals output from the pixels; a recognition unit that performs image recognition on the signal that has been subjected to the predetermined image processing; a region determination unit that determines an image processing region based on a result of the image recognition, The image processing unit performs image processing on a signal output from the pixel next, based on the image processing area determined by the area determination unit. Effect of the Invention

[0009] According to the present invention, it is possible to provide an imaging system capable of reducing the load on the recognition process. [Brief description of the drawings]

[0010] [Figure 1]FIG. 1 is a diagram illustrating an example of the configuration of a photoelectric conversion element according to a first embodiment of the present invention. [Diagram 2] 2 is a diagram showing a configuration example of a sensor substrate 11. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4] 2 and 3, and shows an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] FIG. 6 is a functional block diagram of an imaging device 600 and a moving object 700 according to the first embodiment. [Figure 7] 10 is a diagram for explaining an imaging method by a camera control unit 605 according to the first embodiment. FIG. [Figure 8] FIG. 13 is a diagram showing an example of an image divided into a plurality of frames. [Figure 9] FIG. 2 is a diagram illustrating the relationship between a memory circuit and a buffer in the first embodiment. [Figure 10] 6 is a flowchart showing details of an operation when determining an area based on a recognition result and applying image processing to the target area in the first embodiment. [Figure 11] FIG. 4 is a diagram showing a schematic representation of a result of recognition processing in the first embodiment. [Figure 12] FIG. 7 is a functional block diagram of an imaging device 600 and a moving object 700 according to a second embodiment. [Figure 13] 13 is a flowchart showing details of an operation example when an image processing area and an image processing intensity are changed based on a recognition result in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0012] <Embodiment 1> 1 is a diagram showing a configuration example of a photoelectric conversion element according to a first embodiment of the present invention. In the following, the photoelectric conversion element 100 will be described taking as an example an imaging device having a so-called stacked structure, in which two substrates, a sensor substrate 11 and a circuit substrate 21, are stacked and electrically connected.

[0013] However, a so-called non-laminated structure in which the components included in the sensor substrate and the components included in the circuit substrate are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD).

[0015] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array that constitutes the pixel region 12 is not particularly limited.

[0016] Fig. 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0017] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0018] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0019] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein and connected to the vertical signal line 113.

[0020] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view.

[0021] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0022] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and one signal processing circuit may be shared by a plurality of photoelectric conversion units to perform signal processing sequentially.

[0023] Fig. 4 is a diagram showing an equivalent circuit of the pixel 101 in Fig. 2 and Fig. 3 and the signal processing circuit 103 corresponding to the pixel 101. As shown in Fig. 4, each pixel 101 includes a photoelectric conversion unit 102 as a sensor unit, a counter circuit 211, a memory circuit 212, etc.

[0024] The APD 201 included in the photoelectric conversion unit 102 generates a pair of electric charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. The other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0025] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0026] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0027] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0028] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0029] FIG. 4 shows an example in which the signal processing circuit 103 includes a waveform shaping section 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0030] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example in which one inverter is used as the waveform shaping unit 210 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0031] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0032] 3 to the memory circuit 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0033] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the driving voltage VH or the driving voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0034] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0035] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0036] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 210 and is output as a pulse signal at nodeB.

[0037] Fig. 6 is a functional block diagram of the imaging device 600 and the moving object 700 according to the first embodiment. In this embodiment, an imaging system is configured by some of the functional blocks of the imaging device 600 and the moving object 700. Some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the imaging device 600 and the moving object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0038] However, some or all of them may be realized by hardware. For the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used. Furthermore, the respective functional blocks shown in Fig. 6 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 6 also applies to Fig. 12 described later.

[0039] The imaging device 600 includes a photoelectric conversion element 100, an imaging optical system 601, a region determination unit 602, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, and a communication unit 607. The photoelectric conversion element 100 includes an avalanche photodiode described with reference to FIGS. 1 to 5 for photoelectrically converting an optical image.

[0040] The imaging device of the embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is configured to capture images in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0041] The region determination unit 602 is a region determination unit that determines a region where a predetermined image processing is performed on the image signal acquired by the photoelectric conversion element 100, and determines a target region where the predetermined image processing is performed based on a recognition result output from a recognition unit 604 described later. That is, it determines an image processing region based on the result of image recognition. Region information on the region determined by the region determination unit 602 is output to a camera control unit 605.

[0042] The image processing unit 603 performs image processing on the signals output from the pixels. Moreover, the image processing unit 603 performs predetermined image processing on the image signals based on the area information output from the camera control unit 605 to generate a final image signal. The predetermined image processing includes, for example, contour enhancement, black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, data compression, and other predetermined image processing.

[0043] When the photoelectric conversion element 100 has an on-chip color filter such as RGB, it is desirable to perform processing such as white balance correction and color conversion in the image processing unit 603. The output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving object 700, and a camera control unit 605.

[0044] The recognition unit 604 performs image recognition based on the result of the image processing. That is, the recognition unit 604 recognizes people, vehicles, etc. in the vicinity by performing image recognition based on the image signal that has been image-processed by the image processing unit 603, and issues a warning, etc., as necessary.

[0045] In this embodiment, the moving body 700 is described using an example of an automobile, but the moving body may be any moving body, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0046] The camera control unit 605 has a built-in CPU as a computer and a memory that stores a computer program, and controls each unit of the imaging device 600 by the CPU executing the computer program stored in the memory.

[0047] The camera control unit 605 functions as a control means, and controls the length of the exposure period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK, for example, via a control pulse generation unit for the photoelectric conversion element 100.

[0048] The storage unit 606 includes a recording medium such as a memory card, a hard disk, etc., and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signal to the outside of the imaging device 600 and receives various signals from the outside.

[0049] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0050] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control means for driving, stopping, and controlling the direction of the vehicle as a moving object based on the output of the ECU 701. The vehicle control unit 702 as a movement control means controls the operation of the moving object based on the result of recognition processing using signals generated during at least a first accumulation period. The first accumulation period will be described later.

[0051] The display unit 703 functions as a display means, includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 700. The display unit 703 displays images acquired by the photoelectric conversion element 100 and various information related to the running state of the vehicle, etc., to the driver of the moving body 700 using, for example, a GUI based on the output of the ECU 701.

[0052] 6 may not be mounted on the moving body 700, and may be provided, for example, in an external terminal provided separately from the moving body 700 for remotely controlling the moving body 700 or for monitoring the traveling of the moving body. The imaging device including the imaging device 600 of this embodiment includes an imaging device composed of a plurality of devices connected by such a communication path.

[0053] 7 is a diagram for explaining an imaging method by the camera control unit 605 according to the first embodiment. In this embodiment, photoelectric conversion is performed periodically at, for example, 30 full frames / second. Note that in this embodiment, one frame having a length of, for example, 33.3 ms is called a full frame (or main frame), and a full frame divided into four is called a frame (or sub-frame).

[0054] That is, as shown in Fig. 7, full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 of equal periods (8.33 ms) from time T0 to time T12. In Fig. 7 and subsequent figures, frames 0_3, 0_4, 1_1, 1_2, 1_3, 1_4, 2_1, 2_2, 2_3, and 2_4 are represented as F0_3, 0_4, 1_1, 1_2, 1_3, 1_4, 2_1, 2_2, 2_3, and 2_4.

[0055] Frame 1_1 has an accumulation period from start time T0 of frame 1 to time T1, and frame 1_2 has an accumulation period from time T0 to time T2. Frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4. In this embodiment, for example, the accumulation period from time T0 to time T1 is called a first accumulation period, and the accumulation period from time T0 to time T4 is called a second accumulation period.

[0056] Then, at time T0, the counter circuit 211 is reset, and count values ​​C1_1, C1_2, C1_3, and C1_4 are obtained from the counter circuit 211 at times T1 to T4, respectively.

[0057] Moreover, the count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in the memory circuit 212. Then, the signals for one row temporarily stored in the memory circuit 212 are sequentially output from the photoelectric conversion elements via the buffer of the readout circuit 112.

[0058] In this way, according to this embodiment, the signals accumulated during the period of frame 1_1 are read out from time T1 to T2 and are promptly processed by the recognition unit 604. Therefore, image recognition can be performed promptly. Similarly, the signals accumulated during the periods of frames 1_2, 1_3, and 1_4 are sequentially read out from time T2 to T3, T3 to T4, and T4 to T1, respectively, and image recognition can be performed repeatedly.

[0059] Fig. 8 is a diagram showing an example of images of a plurality of divided frames. As shown in Fig. 8, the image of frame 1_1 is dark because the accumulation time is short, but there is little subject blur of a person who jumps out in front of the camera. On the other hand, the accumulation time becomes longer in the order of frames 1_2, 1_3, and 1_4, so subject blur is more likely to occur. Note that stopped vehicles and white lines are less likely to be blurred, and the longer the accumulation time, the more likely the contrast is to improve.

[0060] In this manner, in the control step in the camera control unit 605 of this embodiment, accumulation operations are performed for the first accumulation period and the second accumulation period within one full frame. Also, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period.

[0061] In this embodiment, the first and second accumulation periods overlap, and start at the same time. Furthermore, the end of the second accumulation period is a frame boundary, and the second accumulation period is an integer multiple of the first accumulation period. That is, the end of the second accumulation period coincides with the end of the full frame.

[0062] However, the second accumulation period does not need to be an integer multiple of the first accumulation period. The second accumulation period only needs to be longer than the first accumulation period (the first accumulation period is shorter than the second accumulation period) and the second accumulation period only needs to end after the first accumulation period.

[0063] That is, an image with a short accumulation period and an image with a long accumulation period are created, the timing when the short accumulation period ends is set to be earlier than the timing when the long accumulation period ends, and as soon as the short accumulation period ends, the image is output and sent to the downstream recognition unit.Then, the subject is recognized based on at least a signal generated during the first accumulation period.The recognition unit 604 as a recognition means recognizes the subject based on at least a signal generated during the first accumulation period.

[0064] Therefore, while in the past it was not possible to recognize an image until one full frame had elapsed, in this embodiment, image recognition can be performed after as little as ¼ full frame period, and for example, when a moving body is moving at high speed, an obstacle or the like can be recognized quickly. Therefore, it is possible to apply the brakes quickly, or to avoid the obstacle early.

[0065] Incidentally, an image with a long accumulation time can improve the contrast, and can therefore be used as an image for display. That is, an image with a short accumulation time is suitable for quick subject recognition, and an image with a long accumulation time is suitable as an image for display. In this manner, the display device of this embodiment displays at least a signal generated during the second accumulation period as an image. Incidentally, the recognition unit 604 may further recognize the subject based on a signal generated during the second accumulation period.

[0066] In addition, since the present embodiment uses an APD, unlike a CMOS sensor, the accumulated charge does not deteriorate due to readout, so the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not deteriorate no matter how many times it is read out from a single accumulation.

[0067] Fig. 9 is a diagram showing the relationship between memory circuits and buffers in embodiment 1. Fig. 9 shows a state in which memory circuits 212 in signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory circuit is represented as memory 1-1 to memory NM. Also, buffer 1 to buffer M in Fig. 9 indicate buffers included in read circuit 112 in Fig. 3. Output circuit 114 in Fig. 9 corresponds to output circuit 114 in Fig. 3.

[0068] FIG. 10 is a flowchart showing details of the operation for determining an area based on the recognition result in embodiment 1 and applying image processing to the target area. An example of the operation for changing the image processing range based on the recognition result in this embodiment will be described with reference to FIGS. 6 and 10.

[0069] The CPU serving as a computer within the camera control unit 605 executes a computer program stored in the memory, thereby sequentially carrying out the operations of the steps in the flowchart of FIG.

[0070] In step S101, the CPU sets an initial target region for image processing using the region determination unit 602. At this time, the initial target region may be a region that has been set in advance, or may be a region that has been used in the previous frame.

[0071] In step S102, the CPU obtains an image signal for frame 1_n with the imaging element, where frame 1_n represents frames (F) 1_1 to 1_4 in FIG.

[0072] In step S103, the CPU applies image processing to the target region in the set image by the image processing unit 603. That is, the image processing unit 603 performs image processing on the region set (determined) by the region determination unit 602. Here, step S103 functions as an image processing step that performs image processing on signals output from pixels, and performs predetermined image processing on the image processing region.

[0073] Here, the image processing includes, as described above, contour emphasis, black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, data compression, etc. The image signal subjected to the image processing by the image processing unit 603 is output to the recognition unit 604.

[0074] In step S104, the CPU performs recognition processing on the image after image processing by the recognition unit 604. The recognition result generated at this time includes at least one piece of information such as the position of the subject, the recognition accuracy of the subject, and the type of the subject, and is output as the recognition result. The output recognition result is output to the area determination unit 602 and the ECU 701.

[0075] Here, step S104 functions as a recognition step for performing image recognition based on the result of image processing, and performs image recognition based on a signal that has been subjected to a predetermined image processing.

[0076] In step S105, the CPU checks the current frame (value of n). If it is frame 1_1, 2, or 3, the process proceeds to step S106. If it is frame 1_4, the frame ends and the process proceeds to the next frame 2_1. Here, the maximum value of n is 4 in this embodiment, but this is just an example and a value other than n=4 may be used.

[0077] In step S106, the CPU determines whether the subject has been recognized based on the result of the recognition process. If the subject has been recognized, the process proceeds to step S107. If the subject has not been recognized, the process proceeds to step S101.

[0078] In step S107, since the subject is recognized, the CPU updates the target area for image processing so that a region including a part or all of the recognized subject is included in the target area for image processing by the region determination unit 602. Note that the target area may be determined to be a larger region on the movement direction side, taking into account the speed and movement direction of the recognized subject.

[0079] That is, in step S107, the area determination unit 602 determines an area to which image processing is to be applied, in accordance with the recognition result output from the recognition unit 604. Here, step S107 functions as an area determination step for determining an image processing area based on the result of image recognition.

[0080] 11 is a diagram showing a schematic representation of the results of the recognition process in the first embodiment, showing an example of an image in which an area has been determined based on the recognition result. Area 1101 in the image is an area in which a vehicle has been recognized. Area 1102 is an area in which a person has been recognized.

[0081] Area 1103 is an area within the angle of view excluding areas 1101 and 1102. The range to which image processing is applied is an area including all or part of the recognized subject, such as areas 1101 and 1102. Note that 1104 indicates the recognition accuracy of the subject in area 1101, which is displayed as 99% in Fig. 11. Meanwhile, 1105 indicates the recognition accuracy of the subject in area 1102, which is displayed as 30%.

[0082] As described above, the size of the region may be determined to be larger in the direction of movement of the recognized subject, taking into consideration that the recognized subject will move in the next frame. If the subject is not recognized, the region may be determined to be a preset initial region, or the entire angle of view may be determined as the region. In step S107, the region information determined (updated) by the region determination unit 602 is output to the camera control unit 605, n is incremented by 1, and the process proceeds to step S102.

[0083] In this embodiment, the image processing unit 603 performs predetermined image processing on the image processing area, and the recognition unit 604 performs image recognition based on the signal that has been subjected to the predetermined image processing. That is, by changing the range to which the image processing is applied according to the recognition result, the image processing range can be reduced, and the time required for recognition can be shortened.

[0084] <Embodiment 2> Fig. 12 is a functional block diagram of an imaging device 600 and a moving object 700 according to the second embodiment, and in the second embodiment, the image processing range and the image processing strength are changed based on the recognition result, the accumulation period, etc. In Fig. 12, the same reference numerals as in Fig. 6 represent the same functional blocks, and therefore the description thereof will be omitted.

[0085] 12, area information determined by area determination section 602 is output to image processing strength determination section 608. Image processing strength determination section 608 determines the image processing strength based on at least the result of image recognition by the recognition section.

[0086] In this embodiment, the image processing intensity determination unit 608 determines the image processing intensity when performing image processing in the image processing unit 603, based on the recognition result output from the recognition unit 604, the area information output from the area determination unit 602, and the accumulation period output from the camera control unit 605.

[0087] For example, as described above, 1104 in Fig. 11 indicates the recognition accuracy of a vehicle, which is recognized as a vehicle at "99%" here. Also, recognition accuracy 1105 indicates the recognition accuracy of a person, which is recognized as a person at "30%" here. In this way, when the recognition accuracy differs for each subject, the image processing intensity is not changed for subjects with high recognition accuracy, or the image processing intensity is reduced.

[0088] On the other hand, for a subject with low recognition accuracy, the image processing intensity is relatively increased. Also, the image processing intensity is changed according to the accumulation period output from the camera control unit 605. For example, between the first accumulation period and the second accumulation period, the first accumulation period is shorter, resulting in a darker and noisier image.

[0089] Therefore, the intensity of noise reduction, etc. is set stronger for the image signal generated in the first accumulation period than for the image signal acquired in the second accumulation period. In addition, the image processing intensity determination unit 608 outputs the region information and image processing intensity information for each piece of region information to the camera control unit 605.

[0090] The camera control unit 605 receives region information and image processing intensity information from the image processing intensity determination unit 608 described above, and outputs them to the image processing unit 603. In addition, it outputs accumulation time information to the image processing intensity determination unit 608. Then, the image processing unit 603 performs image processing on the regions determined by the region determination unit 602, and also performs image processing on each region with the intensity determined by the image processing intensity determination unit 608.

[0091] When the subject is not recognized, a predefined area may be determined as described above, or the entire angle of view may be determined as the area. The area may be changed for each image processing block (image processing process). For example, black level correction may be performed on the entire screen to avoid coloring, and the area may be limited for other image processing blocks (image processing processes).

[0092] Fig. 13 is a flowchart showing details of an example of an operation when changing the image processing area and the image processing intensity based on the recognition result in embodiment 2. Note that the operation of each step in the flowchart in Fig. 13 is performed sequentially by the CPU as a computer in the camera control unit 605 executing a computer program stored in the memory.

[0093] In step S201, the CPU sets an initial target region for image processing and an initial image processing strength. At this time, the initial target region and the initial image processing strength may be a region and image processing strength for the region that have been set in advance, or may be a region and image processing strength for the region that have been used in the previous frame.

[0094] In step S202, the CPU obtains an image signal for frame 1_n by the imaging element, where frame 1_n represents frames (F) 1_1 to 1_4 in FIG.

[0095] In step S203, the CPU applies image processing to each target area in the image set in step S201 at the set intensity. Here, the image processing includes edge enhancement, black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, data compression, etc.

[0096] In step S204, the CPU performs a recognition process on the image after the image processing. The recognition result generated at this time includes information on at least one of the subject position, subject recognition accuracy, subject type, and the like.

[0097] In step S205, the CPU checks the current frame (value of n). If it is frame 1_1, 2, or 3, the process proceeds to step S206. If it is frame 1_4, the frame ends and the process proceeds to the next frame 2_1. Here, the maximum value of n may be a value other than n=4.

[0098] In step S206, the CPU checks whether the subject has been recognized as a result of the recognition process. If the subject has been recognized, the process proceeds to step S207. If the subject has not been recognized, the process proceeds to step S201.

[0099] In step S207, since the subject has been recognized, the CPU updates the area including the subject as the target area for image processing. At this time, the target area is an area including all or a part of the recognized subject. In addition, the area may be set to a larger area depending on the direction and speed of movement, taking into consideration that the recognized subject will move in the next frame.

[0100] In step S208, the CPU changes the image processing intensity based on the recognition result and exposure time information. For example, the recognition accuracy of the recognized subject is output as the recognition result. If the recognition accuracy is low, the image processing intensity is updated to be stronger, and if the recognition accuracy is high, the image processing intensity is not changed (or is lowered). The image processing intensity may also be changed according to the exposure time.

[0101] That is, in this embodiment, the smaller the value of n in frame 1_n, the shorter the exposure time. A short exposure time tends to result in a darker, noisier image. Here, when the exposure time is short, it is desirable to apply stronger image processing to avoid reducing recognition accuracy.

[0102] In this way, in the second embodiment, the range to which image processing is applied and the image processing strength for each range are changed according to the recognition result, and processing is performed, thereby reducing the range to which recognition processing is performed and the image processing range, thereby reducing delay time. Also, image processing that is effective for performing recognition processing can be performed. Moreover, according to the second embodiment, it is possible to read out the accumulated results for a short period of time during processing, so that image processing and recognition processing can be performed quickly.

[0103] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention.

[0104] Also, for example, in the above embodiment, accumulation is performed for a 1 / 4 full frame period at the shortest, and the readout cycle is set to every 1 / 4 full frame period. However, depending on the brightness of the subject, recognition accuracy, etc., the effective accumulation period may be made shorter than the 1 / 4 full frame period by resetting the counter circuit midway through the accumulation period of frame 1_1 in FIG.

[0105] Alternatively, the counter circuit may be reset once at time T1 in Fig. 7. This may adjust the count value read out at time T4. The present invention includes the following combinations.

[0106] (Configuration 1) An imaging system comprising: a plurality of pixels, each having a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons, and a counter that counts the number of the pulses; a control unit that generates a signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, performs accumulation operations for a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controls so that the signal generated in the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; an image processing unit that performs image processing on the signal output from the pixel; a recognition unit that performs image recognition on the signal that has been subjected to the predetermined image processing; and an area determination unit that determines an image processing area based on the result of the image recognition, wherein the image processing unit performs image processing on the signal next output from the pixel based on the image processing area determined by the area determination unit.

[0107] (Configuration 2) The imaging system according to configuration 1, further comprising an image processing intensity determination unit that determines an image processing intensity based on a result of the image recognition by the recognition unit.

[0108] (Configuration 3) The imaging system according to configuration 1 or 2, wherein the first accumulation period and the second accumulation period overlap each other.

[0109] (Configuration 4) The imaging system according to any one of configurations 1 to 3, wherein the first accumulation period and the second accumulation period start simultaneously.

[0110] (Configuration 5) The imaging system according to any one of configurations 1 to 4, wherein the end of the second accumulation period coincides with the end of a frame.

[0111] (Configuration 6) The imaging system according to any one of configurations 1 to 5, further comprising a display means for displaying, as an image, a signal generated during at least the second accumulation period.

[0112] (Configuration 7) The imaging system according to any one of configurations 1 to 6, wherein the sensor section includes an avalanche photodiode.

[0113] (Configuration 8) An imaging system according to any one of configurations 1 and A moving body comprising a movement control means for controlling the movement of the moving body based on a result of a recognition process using a signal generated during at least the first accumulation period.

[0114] (Method) An imaging method in an imaging system having multiple pixels, each pixel having a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons, and a counter that counts the number of the pulses, the imaging method comprising: a control step of generating a signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, performing accumulation operations for a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling so that the signal generated in the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; an image processing step of performing image processing on the signal output from the pixel; a recognition step of performing image recognition on the signal that has been subjected to the predetermined image processing; and an area determination step of determining an image processing area based on the result of the image recognition, wherein the image processing step performs image processing on the signal next output from the pixel based on the image processing area determined in the area determination step.

[0115] (Program) A computer program for controlling each part of the imaging system according to any one of configurations 1 to 7 or the moving body according to configuration 8 by a computer.

[0116] A computer program for implementing all or part of the control in the present embodiment may be supplied to the imaging system via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the imaging system may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0117] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Memory circuit 213: Drive wire 600: Imaging device 601: Imaging optical system 602: Area determination section 603: Image processing unit 604: Recognition part 605: Camera control unit 606: Storage section 607: Communications Department 608: Image processing strength determination unit 1101: Vehicle recognition area 1102:Person recognition area 1103: Background area 1104: Vehicle recognition accuracy 1105: Character recognition accuracy

Claims

1. A sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A plurality of pixels each including a counter for counting the number of the pulses; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control unit that performs an accumulation operation of a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controls so that a signal generated during the first accumulation period is output during a period from the end of the first accumulation period to the end of the second accumulation period; an image processing unit that performs image processing on the signals output from the pixels; a recognition unit that performs image recognition on the signal that has been subjected to the predetermined image processing; a region determination unit that determines an image processing region based on a result of the image recognition, The imaging system according to claim 1, wherein the image processing unit performs image processing on a signal output from the pixel next based on the image processing area determined by the area determination unit.

2. The imaging system according to claim 1 , further comprising an image processing strength determination unit that determines an image processing strength based on a result of the image recognition by the recognition unit.

3. 2. The imaging system according to claim 1, wherein the first accumulation period and the second accumulation period overlap each other.

4. 2. The imaging system according to claim 1, wherein the first accumulation period and the second accumulation period start simultaneously.

5. 2. The imaging system according to claim 1, wherein the end of the second accumulation period coincides with the end of a frame.

6. 2. The imaging system according to claim 1, further comprising a display unit that displays, as an image, at least a signal generated during the second accumulation period.

7. 2. The imaging system according to claim 1, wherein the sensor unit includes an avalanche photodiode.

8. The imaging system according to claim 1 , A moving body comprising a movement control means for controlling the movement of the moving body based on a result of a recognition process using at least a signal generated during the first accumulation period.

9. An imaging method for an imaging system having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, and a counter that counts the number of the pulses, comprising: generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control step of performing an accumulation operation of a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling so that a signal generated during the first accumulation period is output during a period from the end of the first accumulation period to the end of the second accumulation period; an image processing step of performing image processing on the signals output from the pixels; a recognition step of performing image recognition on the signal that has been subjected to the predetermined image processing; and a region determination step of determining an image processing region based on a result of the image recognition, The imaging method according to claim 1, wherein the image processing step performs image processing on a signal output from the pixel based on the image processing area determined in the area determination step.

10. A computer program for controlling each part of the imaging system according to any one of claims 1 to 7 or the moving body according to claim 8 by a computer.

Images