Imaging system, mobile object, control method, and computer program

Abstract

To provide an imaging system that performs recognition processing according to the distance to a subject.SOLUTION: An imaging system includes a plurality of pixels, each having a sensor unit that emits pulses at a frequency corresponding to the frequency of photon reception and a counter that counts the number of pulses, control means that generates a signal on the basis of the difference between the count values of the counter at the start and the end of an accumulation period, performs accumulation operations for a first accumulation period and a second accumulation period within one full frame, in which the first accumulation period is shorter than the second accumulation period, and controls the signal generated during the first accumulation period to be output between the end of the first accumulation period and the end of the second accumulation period, distance information acquisition means that acquires distance information to the subject, and recognition priority adjustment means that determines the area and priority of a recognition target on the basis of the distance information.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an imaging system, a moving object, a control method, a computer program, and the like. [Background technology]

[0002] In recent years, imaging devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value as a photoelectrically converted digital signal from a pixel. Patent Document 1, for example, describes a configuration in which an imaging device having an APD can output multiple images whose accumulation periods overlap each other, thereby enabling continuous shooting even in low illumination. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-153346 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when considering the image sensor of an in-vehicle camera installed on a moving object, for example, normal sensor driving performs recognition processing on a frame-by-frame basis, so in the case of 30 fps, for example, recognition processing can only be performed every 33.3 ms. Therefore, even if an object appears immediately after a frame change in an in-vehicle camera, recognition processing cannot be performed until the end of the frame.

[0005] In addition, in-vehicle cameras often have an accumulation period of at least 11 ms to suppress flicker caused by traffic lights, and by extending the accumulation period, especially in low light conditions, bright images are captured. However, because the accumulation period is long, subject blur occurs when capturing fast-moving objects, reducing the recognition rate.

[0006] Furthermore, when there are multiple moving objects within a frame, for example, in a car-mounted camera, it is necessary to recognize the nearest moving object first in order to avoid collisions.

[0007] Therefore, one object of the present invention is to provide an imaging system that performs recognition processing according to the distance to the subject. [Means for solving the problem]

[0008] An imaging system according to one aspect of the present invention comprises: a plurality of pixels each including a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons, and a counter that counts the number of pulses; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control means for performing an accumulation operation of a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling the signal generated during the first accumulation period to be output between the end of the first accumulation period and the end of the second accumulation period; distance information acquisition means for acquiring distance information to a subject; and a recognition priority adjustment means for determining the area and priority of the recognition target based on the distance information. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an imaging system that performs recognition processing according to the distance to the subject. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of a photoelectric conversion element according to a first embodiment of the present invention. [Figure 2] 2 is a diagram showing an example of the configuration of a sensor substrate 11. FIG. [Figure 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4]4 is a diagram showing an equivalent circuit of the pixel 101 and the signal processing circuit 103 corresponding to the pixel 101 in FIGS. 2 and 3. FIG. [Figure 5] 2 is a diagram schematically illustrating the relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 1 is a functional block diagram of an imaging device 600 and a moving object 700 according to the first embodiment. [Figure 7] 10 is a diagram for explaining processing from photoelectric conversion to image output, distance calculation, and recognition result output by the camera control unit 605 according to the first embodiment. FIG. [Figure 8] 10(A) to 10(C) are diagrams illustrating examples of recognition priorities according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing the relationship between a memory circuit and a buffer according to the first embodiment. [Figure 10] 10 is a flowchart showing an example of recognition priority adjustment according to the first embodiment. [Figure 11] FIG. 10 is a functional block diagram of an imaging device 600 and a moving object 700 according to a second embodiment. [Figure 12] 10(A) to 10(D) are diagrams illustrating examples of recognition priorities according to the second embodiment. [Figure 13] 10 is a flowchart showing an example of recognition priority adjustment according to the second embodiment. [Figure 14] FIG. 10 is a functional block diagram of an imaging device 600 and a moving object 700 according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of an imaging system of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions will be omitted or simplified. <Embodiment 1>

[0012] 1 is a diagram showing an example of the configuration of a photoelectric conversion element according to embodiment 1 of the present invention. In the following, an imaging device having a so-called stacked structure in which a photoelectric conversion element 100 is configured by stacking and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21, will be described as an example.

[0013] However, a so-called non-laminated structure may be used in which the components included in the sensor substrate and the components included in the circuit substrate are arranged on a common semiconductor layer. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing an example of the configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across multiple rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, referred to as APD).

[0015] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array that forms the pixel region 12 is not particularly limited.

[0016] 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0017] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generating unit 115 and sequentially supplies the control pulses to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0018] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, memory, etc., and digital values ​​are stored in the memory. In order to read the signal from the memory of each pixel where the digital signal is stored, the horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103.

[0019] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built in, which are connected to each vertical signal line 113.

[0020] 2 and 3, a plurality of signal processing circuits 103 are arranged in an area overlapping the pixel area 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel area 12 in a plan view.

[0021] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0022] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 is not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Furthermore, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and a configuration may be adopted in which one signal processing unit is shared by multiple photoelectric conversion units and performs signal processing sequentially.

[0023] Fig. 4 is a diagram showing an equivalent circuit of the pixel 101 and the signal processing circuit 103 corresponding to the pixel 101 in Fig. 2 and Fig. 3. As shown in Fig. 4, each pixel 101 includes a photoelectric conversion unit 102 as a sensor unit, a counter circuit 211, a memory circuit 212, etc.

[0024] The APD 201 included in the photoelectric conversion unit 102 generates charge pairs in response to incident light through photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line that supplies a drive voltage VL (first voltage). The other of the two nodes of the APD 201 is connected, via a quench element 202, to a power supply line that supplies a drive voltage VH (second voltage) that is higher than the voltage VL.

[0025] In Figure 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. With this voltage supplied, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0026] When a reverse bias voltage is supplied, APDs can be operated in either Geiger mode, where the voltage difference between the anode and cathode is greater than the breakdown voltage, or in linear mode, where the voltage difference between the anode and cathode is close to or less than the breakdown voltage. APDs operating in Geiger mode are called SPADs.

[0027] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a drive voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0028] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD 201 and suppressing avalanche multiplication (quench operation).The quench element 202 also functions to return the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0029] FIG. 4 shows an example in which the signal processing circuit 103 includes a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0030] The waveform shaping unit 210 shapes the voltage change at the cathode of the APD 201 obtained when photons are detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. While Fig. 4 shows an example in which one inverter is used as the waveform shaping unit 210, a circuit in which multiple inverters are connected in series, or another circuit that has a waveform shaping effect, may also be used.

[0031] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a RES drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0032] 3 is supplied to the memory circuit 212 via a SEL drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0033] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0034] 5 is a diagram schematically illustrating the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is designated node A, and the output side is designated node B. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at node A drops.

[0035] As the voltage drop increases further and the potential difference applied to APD 201 decreases, avalanche multiplication in APD 201 stops, as shown at time t2, and the voltage level at node A no longer drops below a certain value. Between time t2 and time t3, a current flows from voltage VL to node A to compensate for the voltage drop, and at time t3, node A stabilizes to its original potential level. At this time, the portion of the output waveform at node A that exceeds a certain threshold is shaped by waveform shaping unit 210 and output as a pulse signal at node B.

[0036] 6 is a functional block diagram of the imaging device 600 and the moving object 700 according to embodiment 1. In this embodiment, the imaging system is configured by some of the functional blocks of the imaging device 600 and the moving object 700.

[0037] Note that some of the functional blocks shown in FIG. 6 are realized by causing a computer (not shown) included in the imaging device 600 and the moving body 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0038] However, some or all of these functions may be implemented by hardware, which may be a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP).

[0039] Furthermore, the functional blocks shown in Fig. 6 do not have to be built into the same housing, but may be configured as separate devices connected to each other via signal paths. The above explanation regarding Fig. 6 also applies to Figs. 11 and 14.

[0040] The imaging device 600 includes a photoelectric conversion element 100, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, a distance calculation unit 608, a recognition priority adjustment unit 609, and the like.

[0041] The photoelectric conversion element 100 serving as a sensor unit includes an avalanche photodiode for photoelectrically converting an optical image, as described with reference to Figures 1 to 5. In this embodiment, a stereo camera is configured with a plurality of imaging optical systems 601 and photoelectric conversion elements 100.

[0042] The imaging device of the first embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is configured to capture images in at least one direction, for example, in front, behind, or to the side of the moving body. A plurality of camera units may be provided on the moving body 700.

[0043] The image processing unit 603 performs image processing on the signals output from the pixels. Furthermore, the image processing unit 603 performs predetermined image processing on the image signals based on the region information output from the camera control unit 605 to generate a final image signal. The predetermined image processing includes, for example, image processing such as edge enhancement, black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, and data compression.

[0044] If the photoelectric conversion element 100 has an on-chip color filter such as RGB, it is desirable to perform processing such as white balance correction and color conversion in the image processing unit 603. The output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving object 700, and a camera control unit 605.

[0045] The distance calculation unit 608 functions as a distance information acquisition unit, and acquires distance information by performing distance measurement processing on signals output from pixels and calculating the distance to the subject. In this embodiment, the distance calculation unit 608 receives left and right images from multiple photoelectric conversion elements 100 functioning as a stereo camera, calculates the parallax between the left and right images, and then converts it into distance.

[0046] A known template matching process such as SAD (Sum of Absolute Difference) is used to calculate the parallax, and the distance is calculated based on the principle of triangulation.

[0047] The output of the distance calculation unit 608 is supplied to a recognition priority adjustment unit 609 and an ECU of the moving object 700. In this embodiment, distance measurement (distance calculation) is performed using signals output from pixels, but distance measurement may also be performed using image data corrected by the image processing unit 603.

[0048] The recognition priority adjustment unit 609 adjusts the recognition priority of the image in accordance with the distance information to the subject output by the distance calculation unit 608, which serves as distance information acquisition means, and the area division information and threshold information specified by the camera control unit 605. Here, the recognition priority adjustment unit 609 functions as recognition priority adjustment means that determines the area and priority of the recognition target based on the distance information. A specific method for adjusting the recognition priority will be described later.

[0049] The recognition unit 604 performs image recognition based on the results of image processing and the recognition priority. That is, based on the priority adjusted by the recognition priority adjustment unit 609, the recognition unit 604 recognizes surrounding people, vehicles, etc. in order of urgency by performing image recognition on the image signal image processed by the image processing unit 603, and issues a warning or the like as necessary. Here, the recognition unit 604 functions as recognition means that recognizes subjects in accordance with the order of priority determined by the recognition priority adjustment means.

[0050] In this embodiment, the mobile body 700 is described using an example of an automobile, but the mobile body may be any mobile body such as an airplane, train, ship, drone, AGV, or robot.

[0051] The camera control unit 605 has a built-in CPU as a computer and memory that stores a computer program, and controls each unit of the imaging device 600 by the CPU executing the computer program stored in the memory.

[0052] The camera control unit 605 functions as a control means, and controls the length of the exposure period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK, for example, via a control pulse generation unit of the photoelectric conversion element 100.

[0053] The storage unit 606 includes a recording medium such as a memory card or a hard disk, and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs generated image signals to the outside of the imaging device 600 and receives various signals from the outside.

[0054] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving object 700 by the CPU executing the computer program stored in the memory.

[0055] The output of ECU 701 is supplied to vehicle control unit 702 and display unit 703. Vehicle control unit 702 functions as a movement control means that drives, stops, controls the direction of the vehicle as a moving body, etc., using the output of ECU 701 based on the results of recognition by recognition unit 604, etc.

[0056] The display unit 703 functions as a display means, includes a display element such as a liquid crystal device or an organic EL, and is mounted on the mobile object 700. Based on the output of the ECU 701, the display unit 703 uses, for example, a GUI to display to the driver of the mobile object 700 various information such as images acquired by the photoelectric conversion element 100, the vehicle's running state, and the results recognized by the recognition unit.

[0057] 6 may not necessarily be mounted on the moving object 700. For example, they may be provided in an external terminal or the like that is provided separately from the moving object 700 and that is used to remotely control the moving object 700 or to monitor the movement of the moving object 700. The imaging system of this embodiment includes an imaging system that is configured from a plurality of devices connected by such communication paths.

[0058] 7 is a diagram for explaining the processing from photoelectric conversion to image output, distance calculation, and recognition result output by the camera control unit 605 according to the first embodiment. In this embodiment, photoelectric conversion is performed periodically at, for example, 30 full frames per second. Note that in this embodiment, one frame having a length of, for example, 33.3 ms is called a full frame (or main frame), and each division of a full frame is called a frame (or sub-frame).

[0059] That is, as shown in FIG. 7, full frame 1 having a length of 33.3 ms from time T0 to time T4 is divided into four, and full frame 1 is divided into frame 1_1, frame 1_2, frame 1_3, and frame 1_4 as shown in FIG.

[0060] In this embodiment, frame 1_1 has a period of 8.33 ms, but may have another period. Frame 1_1 has an accumulation period from start time T0 of frame 1 to time T1, and frame 1_2 has an accumulation period from time T0 to time T2. Frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4.

[0061] Here, for example, the accumulation period from start time T0 to time T1 functions as a first accumulation period. Also, the accumulation period from time T0 to time T2, or the accumulation period from time T0 to time T3, or the accumulation period from time T0 to time T4 functions as a second accumulation period, and accumulation operations for the first accumulation period and the second accumulation period are performed within one full frame.

[0062] The first and second accumulation periods overlap, and start simultaneously. The camera control unit 605 executes a control step in which the first accumulation period is shorter than the second accumulation period, and a signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period.

[0063] At time T0, the counter circuit 211 is reset, and count values ​​C1_1, C1_2, C1_3, and C1_4 are acquired from the counter circuit 211 at times T1, T2, T3, and T4, respectively.

[0064] The count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in the memory circuit 212. The signals for one row temporarily stored in the memory circuit 212 are sequentially output from the photoelectric conversion elements via the buffer of the readout circuit 112.

[0065] The signals accumulated during frame 1_1 are read out from time T1 to T2. Furthermore, image processing unit 603 and distance calculation unit 608 generate image I1_1 and calculate distance D1_1, and frame R1_1 is recognized from the near distance area in accordance with priority from time T2 to T3 by recognition unit 604. That is, the distance information acquisition means calculates distance information based on signals generated during at least the first accumulation period.

[0066] Therefore, in this embodiment, image recognition can be performed quickly, starting with the object with the highest urgency. Similarly, the signals accumulated during the period of frame 1_2 are read out from time T2 to T3, and image recognition can be repeatedly performed from time T3 to T4. Thereafter, the recognition result output for frames 1_3 and 1_4 is completed at the same timing.

[0067] Furthermore, because this embodiment uses an APD, unlike a CMOS sensor, the accumulated charge does not deteriorate when read out. Therefore, the accumulation periods can be overlapped, and since there is no readout noise, the original signal does not deteriorate no matter how many times it is read out from a single accumulation.

[0068] By performing intermediate readout multiple times, dark objects that were initially unrecognizable become recognizable in the next subframe over time. On the other hand, when a high frame rate is achieved using a CMOS sensor, etc., the accumulation times do not overlap, so dark objects do not become recognizable.

[0069] Furthermore, if the brightness level is increased by adding frames captured at a high frame rate using a CMOS sensor or the like, noise is amplified, resulting in a lower recognition rate. Even when performing HDR compositing with different exposure times, artifacts are likely to occur because the start times of the accumulation periods differ in CMOS sensors or the like. In contrast, this embodiment uses a SPAD, so the accumulation periods start simultaneously, preventing amplification of readout noise.

[0070] 8A to 8C are diagrams for explaining examples of recognition priorities according to the first embodiment, and show examples of areas and priorities of recognition targets determined by the recognition priority adjustment unit 609 based on the distance information generated by the distance calculation unit 608.

[0071] Fig. 8(A) shows an example of a captured image output by the image processing unit 603. Fig. 8(B) shows an example of distance information for each region calculated by the distance calculation unit 608, in which the camera control unit 605 divides the image into regions, and extracts the closest value (smallest value) for each divided region as a representative value.

[0072] In this embodiment, the closest value within the region is used, but this is not limited to this, and a statistical value such as the average value, median value, or mode value of the distance within the region may also be used. Fig. 8(C) shows an example of the recognition priority for each region determined by the recognition priority adjustment unit 609 based on the distance information.

[0073] In this embodiment, priority is basically given to areas that are closer in distance. Threshold information is set by the camera control unit 605, and areas that are farther away than the threshold are excluded from recognition. By reducing the number of recognition targets, the load on the recognition process can be reduced, resulting in reduced bandwidth and power consumption.

[0074] In this embodiment, the image is divided into a 4x4 grid and the recognition priority is assigned to each divided area, but this is not limiting. For example, the point with the most feature amount in the image may be set as a representative point, and the distance to this representative point may be referenced, and the recognition priority may be assigned with the distance or coordinates of this representative point as the center.

[0075] Fig. 9 is a diagram showing the relationship between memory circuits and buffers according to embodiment 1. Fig. 9 shows a state in which the memory circuits 212 in the signal processing circuit 103 of Fig. 3 are arranged in N rows and M columns, and each memory circuit is represented as memory 1-1 to memory NM.

[0076] Buffers 1 to M in Fig. 9 indicate buffers included in the read circuit 112 in Fig. 3. The output circuit 114 in Fig. 9 corresponds to the output circuit 114 in Fig. 3.

[0077] Fig. 10 is a flowchart showing an example of recognition priority adjustment according to embodiment 1. Note that the operation of each step in the flowchart of Fig. 10 is performed sequentially for each subframe by a CPU or the like serving as a computer in the camera control unit 605 executing a computer program stored in memory.

[0078] In step S1000, the CPU of the camera control unit 605 performs area division setting and threshold setting, and the process proceeds to step S1001. The above threshold is used to set distance information to exclude from recognition targets, and for example, a threshold is set to distinguish distant objects that do not pose a collision risk.

[0079] In step S1001, the CPU of the camera control unit 605 calculates a statistical value, such as an average distance, for each divided region as a distance value for each region, and proceeds to step S1002. Here, step S1001 functions as a distance information acquisition step for acquiring distance information to the subject.

[0080] In step S1002, the CPU of the camera control unit 605 determines whether the distance to the target area in the screen, for example, the upper left corner, is equal to or greater than the threshold value. If the distance to the target area is equal to or greater than the threshold value, the process proceeds to step S1003; if it is less than the threshold value, the process proceeds to step S1004.

[0081] In step S1003, the CPU of the camera control unit 605 determines that the target area is far enough away, so designates it as a recognition skip area to exclude it from the recognition targets, and the process proceeds to step S1007.

[0082] On the other hand, in step S1004, the CPU of the camera control unit 605 compares, in a predetermined order, the distance values ​​of other areas whose distances are less than the threshold and that have not yet been compared, and determines whether they are close. If they are close, the process proceeds to step S1005. If they are farther away than the other areas, the process proceeds to step S1006.

[0083] In step S1005, the CPU of the camera control unit 605 adjusts the recognition priority to increase the recognition priority, and the process proceeds to step S1007.

[0084] On the other hand, in step S1006, the CPU of the camera control unit 605 adjusts the recognition priority to lower the recognition priority, and the process proceeds to step S1007.

[0085] In step S1007, if comparison of each area with all other areas whose distance is less than the threshold has been completed in step S1004, the CPU of the camera control unit 605 determines that adjustment of the recognition priority has been completed and proceeds to step S1008.

[0086] If comparison of each area with all other areas whose distance is less than the threshold has not been completed in step S1004, it is determined that adjustment of the recognition priority has not been completed, and the process returns to step S1002 to repeat the adjustment of the recognition priority.

[0087] As a result, if the determination in step S1007 is Yes, the adjustment of the recognition priority rankings for all areas whose distance is less than the threshold is completed, as shown in Fig. 8(C). Note that instead of the processes shown in steps S1004 to S1006, all other areas whose distance is less than the threshold may be sorted by distance and the recognition priority rankings may be adjusted.

[0088] Next, in step S1008, the CPU of the camera control unit 605 performs recognition processing according to the determined priority order, and proceeds to step S1009, where it notifies the ECU of the recognition information and ends the flow of Fig. 10. Here, steps S1000 to S1008 function as recognition priority adjustment steps that determine the area and priority of the recognition target based on the distance information.

[0089] As described above, according to the first embodiment, by using distance information, it is possible to perform recognition more quickly, starting with a close-distance area with a high recognition priority. Also, by excluding distant areas from the recognition targets in advance, when there are multiple moving objects to be recognized in the close distance, it is possible to allocate resources to the recognition processing for the close-distance areas, and the recognition processing can be completed in a shorter time. <Embodiment 2>

[0090] Fig. 11 is a functional block diagram of an imaging device 600 and a moving object 700 according to the second embodiment. In the second embodiment, a reliability calculation unit 610 is added to the configuration of the first embodiment shown in Fig. 6. The reliability calculation unit 610 generates a reliability map indicating the reliability of the distance information output by the distance calculation unit 608.

[0091] That is, even if the distance information output by the distance calculation unit 608 indicates a short distance, noise may be determined to be a short distance in some cases. Also, even if the distance information indicates a long distance, sufficient texture information to measure the distance may not be available, or the contrast may be low, resulting in the distance being determined to be a long distance.

[0092] For example, there may be a case where the distance calculation unit 608 erroneously calculates the distance as 100 m even though the actual distance is 30 m due to the lack of texture in the subject. In such a case, an area that should have been given priority for recognition will end up being treated as an area outside the scope of recognition.

[0093] Therefore, in the second embodiment, by taking into account the information on the reliability of the distance, noise is suppressed so that an area that should be subjected to the recognition process is not judged as being far away and excluded from the recognition target. Note that, as a specific method for calculating the reliability, for example, the degree of variation is calculated by calculating the amount of change in contrast based on the variance value of the image of the area near the point of interest.

[0094] The greater the variation, the higher the reliability of the template matching is determined to be. In other words, if the contrast is low, the reliability is determined to be low.

[0095] Furthermore, the recognition priority adjustment unit 609 of the second embodiment adjusts the recognition priority of the image based on the distance information output by the distance calculation unit 608, the reliability map output by the reliability calculation unit 610, and the area division information and threshold information specified by the camera control unit 605. A specific adjustment method will be described later. The other functional blocks are similar to the configuration shown in Fig. 6, so their description will be omitted.

[0096] 12A to 12D are diagrams illustrating examples of recognition priorities according to embodiment 2. 12A to 12D show examples of areas and priorities of recognition targets determined by the recognition priority adjustment unit 609 based on the distance information generated by the distance calculation unit 608 and the reliability map generated by the reliability calculation unit 610.

[0097] Fig. 12(A) shows an example of a captured image output by the image processing unit 603. Fig. 12(B) shows an example of distance information for each region calculated by the distance calculation unit 608. The camera control unit 605 sets region division, and extracts the closest value for each divided region, or a statistical value such as the average, median, or mode as a representative value.

[0098] Fig. 12(C) shows a reliability map for each region calculated by the reliability calculation unit 610. Fig. 12(D) shows an example of the recognition priority for each region determined by the recognition priority adjustment unit 609 based on the distance information and the reliability map.

[0099] Also, a long distance threshold can be set by the camera control unit 605. In this embodiment, when the object is farther than the set long distance threshold, it is excluded from recognition if the reliability of the distance is high, and is included in recognition if the reliability of the distance is low, as shown in Figures 12(C) and (D).

[0100] This is because even if the distance calculation result indicates a long distance, if the reliability of the distance is low, there is a possibility that a flying object or the like is approaching at high speed. The recognition priority adjustment unit 609 determines the priority based on the distance information and the distance reliability map.

[0101] Basically, the closer the distance, the higher the priority, but there are areas where the correct distance cannot be measured due to noise, etc., so by utilizing reliability, more appropriate prioritization can be achieved.

[0102] In this embodiment, the short distance threshold can also be set by the camera control unit 605. The determination of whether to prioritize a higher reliability or a closer distance can be adjusted by the short distance threshold set by the camera control unit 605. Fig. 12(D) shows a setting example in which a higher reliability is prioritized if the reliability is equal to or less than the set short distance threshold of 30.

[0103] That is, for the areas with recognition priorities of "2" and "3" in Fig. 12(D), the distance value is 10 for the area with recognition priority of "3" and 30 for the area with recognition priority of "2", meaning that the area with recognition priority of "3" is closer. However, in the reliability map shown in Fig. 12(C), the area with recognition priority of "2" is higher, so it can be seen that the priority order is determined according to reliability.

[0104] Fig. 13 is a flowchart showing an example of recognition priority adjustment in embodiment 2. Note that the operation of each step in the flowchart in Fig. 13 is performed sequentially by a CPU or the like serving as a computer in the camera control unit 605 executing a computer program stored in memory.

[0105] In step S1300, the CPU of the camera control unit 605 performs area division setting and threshold setting, and proceeds to step S1301. The threshold is set to a long distance threshold that excludes objects from recognition. A threshold is set to determine that the distance to the object is sufficient so that there is no risk of collision.

[0106] In step S1301, the CPU of the camera control unit 605 calculates the distance for each region, and in step S1302, the reliability calculation unit 610 calculates the reliability for each region. Here, step S1302 functions as a reliability calculation step (reliability calculation means) that calculates the reliability based on signals generated during at least the first accumulation period.

[0107] Furthermore, in step S1303, the CPU of the camera control unit 605 determines whether the distance of the target area is equal to or greater than the long distance threshold. If the distance of the target area is equal to or greater than the long distance threshold, the process proceeds to step S1304. If the distance is less than the long distance threshold, the process proceeds to step S1306.

[0108] In step S1304, the CPU of the camera control unit 605 determines whether the reliability of the target region is higher than a predetermined reliability threshold. If the reliability of the target region is higher than the predetermined reliability threshold, the process proceeds to step S1305. If not, the process proceeds to step S1307.

[0109] In step S1305, the CPU of the camera control unit 605 determines that the target area is far enough away, and so designates it as a recognition skip area by the recognition priority adjustment unit 609 to exclude it from recognition targets. In this way, in step S1305, areas whose reliability is higher than a predetermined reliability threshold and whose distance is equal to or greater than the threshold are excluded from recognition targets. Then, the process proceeds to step S1311.

[0110] In step S1306, the CPU of the camera control unit 605 determines whether the reliability of the target region is higher than a predetermined reliability threshold. If the reliability of the target region is higher than the predetermined reliability threshold, the process proceeds to step S1308. If not, the process proceeds to step S1307.

[0111] In step S1307, the CPU of the camera control unit 605 does not designate the area as a recognition skip area because the reliability information is below a predetermined reliability threshold, but adjusts the recognition priority based on the reliability and distance information, and proceeds to step S1311.

[0112] In step S1308, the CPU of the camera control unit 605 compares the distance value of the current area with the distance value of other areas that have not yet been compared and whose distance is less than the threshold in a predetermined order, and determines whether the current area is close. If the current area is close, the process proceeds to step S1309. If the current area is farther away than the other areas, the process proceeds to step S1310.

[0113] In step S1309, the CPU of the camera control unit 605 adjusts the recognition priority by using the recognition priority adjustment unit 609 to increase the recognition priority. That is, in step S1309, the priority of an area where the reliability is equal to or greater than a predetermined value and the distance is less than a threshold is increased. Then, the process proceeds to step S1311.

[0114] On the other hand, in step S1310, the CPU of the camera control unit 605 adjusts the recognition priority to lower the recognition priority, and the process proceeds to step S1311.

[0115] In step S1311, if comparison of each area with all other areas whose distance is less than the threshold has been completed in step S1308, the CPU of the camera control unit 605 determines that adjustment of the recognition priority has been completed and proceeds to step S1312.

[0116] If comparison of each area with all other areas whose distance is less than the threshold has not been completed in step S1308, it is determined that adjustment of the recognition priority has not been completed, and the process returns to step S1303 to repeat the adjustment of the recognition priority.

[0117] As a result, if the determination in step S1308 is Yes, the adjustment of the recognition priority order is completed for all regions whose reliability is higher than a predetermined reliability threshold and whose distance is less than the threshold, as shown in Fig. 12(D). Note that instead of the processes shown in steps S1308 to S1310, all other regions whose reliability is higher than a predetermined reliability threshold and whose distance is less than the threshold may be sorted by distance and the recognition priority order adjusted.

[0118] Steps S1312 and S1313 are similar to steps S1008 and S1009 in Fig. 10, respectively, and therefore will not be described here. Steps S1307, S1309, S1310, etc. function as recognition priority adjustment steps (recognition priority adjustment means) that determine priorities based on the distance information calculated by the distance calculation means and the reliability calculated by the reliability calculation means.

[0119] As described above, by determining the recognition priority by combining the reliability map with the distance information, it is possible to suppress the influence of noise and to perform more accurate recognition from a short distance area with a high recognition priority.

[0120] In the second embodiment, a reliability map is used as an example of a means for determining recognition priority, but the combination with distance information is not limited to a reliability map. For example, optical flow (motion vector) information may be combined to increase the priority of approaching regions and decrease the priority of receding regions.

[0121] Furthermore, it is also possible to determine whether an area contains a moving object, and to exclude areas that are at a predetermined distance or more and are determined to contain no moving object from the recognition target. This improves the recognition speed for areas containing moving objects and reduces the bandwidth. <Embodiment 3>

[0122] Fig. 14 is a functional block diagram of an imaging device 600 and a moving object 700 according to embodiment 3. In the functional block diagram of Fig. 11 used in the description of embodiment 2, distance calculation was performed using a stereo camera configuration having two systems of imaging optical systems 601 and photoelectric conversion elements 100.

[0123] However, it is not essential to have two systems of imaging optical system 601 and photoelectric conversion element 100 for distance calculation, and distance calculation is possible with one system as shown in FIG.

[0124] That is, by using a known phase difference detection method in which the pixels of the photoelectric conversion element are divided and light beams that pass through different areas on the pupil of the photographing lens are directed to different pixels, distance calculation becomes possible even with a single imaging optical system 601 and photoelectric conversion element 100.

[0125] Combining the imaging optical system 601 and the photoelectric conversion element 100 into one system leads to miniaturization and improved ease of installation. The other configurations, operation flows, etc. are the same as those of the second embodiment, so a description thereof will be omitted.

[0126] The present invention has been described above in detail based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0127] For example, in the first to third embodiments, a stereo camera or a photoelectric conversion element of a phase difference detection type is used to measure the distance as the distance information acquisition means for acquiring the distance information to the subject. However, a sensor such as a LiDAR (Light Detection and Ranging) may also be used to generate a distance map as the distance information acquisition means for acquiring the distance information to the subject.

[0128] The present invention also includes those that realize the functions of the above-described embodiments using at least one processor or circuit such as a CPU, etc. Also, it is possible to use multiple processors to perform distributed processing.

[0129] In order to realize part or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an imaging system or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the imaging system or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0130] (Configuration 1) An imaging system characterized by having: a plurality of pixels each having a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons and a counter that counts the number of said pulses; a control means that generates a signal based on the difference between the count values ​​of said counter at the start and end of an accumulation period, performs accumulation operations for a first accumulation period and a second accumulation period within one full frame, said first accumulation period being shorter than said second accumulation period, and controls so that said signal generated in said first accumulation period is output between the end of said first accumulation period and the end of said second accumulation period; a distance information acquisition means that acquires distance information to a subject; and a recognition priority adjustment means that determines the area and priority of a recognition target based on said distance information.

[0131] (Configuration 2) The imaging system according to configuration 1, wherein the first accumulation period and the second accumulation period overlap.

[0132] (Configuration 3) The imaging system according to configuration 2, wherein the first accumulation period and the second accumulation period start simultaneously.

[0133] (Configuration 4) The imaging system according to any one of configurations 1 to 3, wherein the distance information acquisition means calculates the distance information based on the signal generated during at least the first accumulation period.

[0134] (Configuration 5) The imaging system according to configuration 4, further comprising a reliability calculation means for calculating reliability based on the signal generated during at least the first accumulation period.

[0135] (Configuration 6) The imaging system according to Configuration 5, characterized in that the recognition priority adjustment means determines a priority based on the distance information calculated by the distance information acquisition means and the reliability calculated by the reliability calculation means.

[0136] (Configuration 7) The imaging system according to Configuration 6, wherein the recognition priority adjustment means excludes areas where the reliability is higher than a predetermined reliability threshold and the distance is equal to or greater than the threshold from the recognition target.

[0137] (Configuration 8) The imaging system according to configuration 6 or 7, wherein the recognition priority adjustment means increases the priority of an area where the reliability is equal to or greater than a predetermined value and the distance is less than a threshold value.

[0138] (Configuration 9) The imaging system according to any one of configurations 1 to 8, wherein the sensor section includes an avalanche photodiode.

[0139] (Configuration 10) A moving body having a display means for displaying information relating to the results of the recognition means of the imaging system according to claim 10.

[0140] (Method) A control method for an imaging system having a plurality of pixels, each pixel having a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons, and a counter that counts the number of said pulses, the control method comprising: a control step of generating a signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, performing accumulation operations for a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling so that the signal generated in the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; a distance information acquisition step of acquiring distance information to the subject; and a recognition priority adjustment step of determining the area and priority of the recognition target based on the distance information.

[0141] (Program) A computer program for controlling each means of the imaging system according to any one of configurations 1 to 11 or the moving body according to configuration 12 by a computer. [Explanation of symbols]

[0142] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Memory circuit 213: RES drive line 214: SEL drive line 600: Imaging device 601: Imaging optical system 603: Image processing unit 604: Recognition part 605: Camera control unit 606: Storage section 607: Communications Department 700: Mobile 701:ECU 702: Vehicle control unit 703: Display section

Claims

1. a plurality of pixels each including a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons, and a counter that counts the number of pulses; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control means for performing an accumulation operation for a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling the signal generated during the first accumulation period to be output between the end of the first accumulation period and the end of the second accumulation period; distance information acquisition means for acquiring distance information to a subject; and recognition priority adjustment means for determining a region and priority of a recognition target based on the distance information.

2. 2. The imaging system according to claim 1, wherein the first accumulation period and the second accumulation period overlap.

3. 3. The imaging system according to claim 2, wherein the first accumulation period and the second accumulation period start simultaneously.

4. 2. The imaging system according to claim 1, wherein the distance information acquisition means calculates the distance information based on the signal generated during at least the first accumulation period.

5. 5. The imaging system according to claim 4, further comprising a reliability calculation unit that calculates reliability based on the signal generated during at least the first accumulation period.

6. 6. The imaging system according to claim 5, wherein the recognition priority adjustment means determines priorities based on the distance information calculated by the distance information acquisition means and the reliability calculated by the reliability calculation means.

7. 7. The imaging system according to claim 6, wherein the recognition priority adjustment means excludes areas where the reliability is higher than a predetermined reliability threshold and the distance is equal to or greater than a threshold from the recognition target.

8. 7. The imaging system according to claim 6, wherein the recognition priority adjustment means increases the priority of an area where the reliability is equal to or greater than a predetermined value and the distance is less than a threshold value.

9. 2. The imaging system according to claim 1, wherein the sensor unit includes an avalanche photodiode.

10. 2. The imaging system according to claim 1, further comprising recognition means for recognizing a subject in accordance with the order of priority determined by said recognition priority adjustment means.

11. A moving body having a movement control means for controlling the moving body based on the result of the recognition means of the imaging system according to claim 10.

12. A control method for an imaging system having a plurality of pixels, each pixel including a sensor unit that emits pulses at a frequency corresponding to a frequency of receiving photons, and a counter that counts the number of pulses, the method comprising: generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control step of performing an accumulation operation for a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling so that the signal generated in the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; a distance information acquisition step of acquiring distance information to a subject; and a recognition priority adjustment step of determining a region and priority of a recognition target based on the distance information.

13. A computer program for controlling the imaging system according to any one of claims 1 to 10 or the means of the mobile body according to claim 11 by a computer.

Images