Object detection device, object detection system and object detection method

Abstract

The object of the present invention is to detect an object more quickly and accurately by detecting the characteristics of a flicker component. An object detection device according to an embodiment of the present invention is provided with: a solid-state image capture device (200) having a plurality of pixels arranged in a matrix and detecting the occurrence of an event in a pixel according to the amount of light incident on each pixel; a flicker detection unit (12) generating flicker information based on the occurrence of an event detected by the solid-state image capture device; and an object detection unit (15) detecting an object based on the flicker information detected by the solid-state image capture device.

Description

Technical Field

[0001] The present disclosure relates to an object detection device, an object detection system and an object detection method. Background Art

[0002] In recent years, technology has been actively developed that detects objects existing around a vehicle from an image obtained by imaging the vehicle's surroundings and assists autonomous driving based on the detection result to allow the vehicle to travel autonomously or be driven by a driver.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Unexamined Patent Publication No. 2016-533140 Summary of the invention

[0006] Problems to be solved by the present invention

[0007] Here, traffic lights, electronic bulletin boards, car lights, etc. for traffic control include various flicker components corresponding to light sources, power sources, etc., and it is considered that these flicker components can be used to detect objects with higher accuracy and higher speed.

[0008] However, in a general image sensor that obtains image data at a predetermined frame rate, it is difficult to measure characteristics such as the frequency of a flicker component emitted from a traffic light, an electronic bulletin board, a car light, or the like.

[0009] Therefore, the present disclosure proposes an object detection device, an object detection system, and an object detection method that are capable of detecting characteristics of a flicker component and detecting an object at a higher speed and with higher accuracy.

[0010] Problem Solution

[0011] In order to solve the above-mentioned problems, according to an embodiment of the present disclosure, an object detection device is provided with a solid-state imaging device, a flicker detection unit and an object detection unit, the solid-state imaging device is provided with a plurality of pixels arranged in a matrix, the solid-state imaging device detects the occurrence of an event in the pixel according to the amount of light incident on each pixel, the flicker detection unit generates flicker information based on the occurrence of the event detected by the solid-state imaging device, and the object detection unit detects the object based on the flicker information detected by the solid-state imaging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A diagram for explaining an object in which a flicker component is detected within a viewing angle.

[0013] Figure 2 : is a block diagram showing a schematic configuration example of the object detection device (system) according to the first embodiment.

[0014] Figure 3 is a block diagram showing a functional configuration example of a DVS according to the first embodiment.

[0015] Figure 4 is a circuit diagram showing a schematic configuration example of a unit pixel according to the first embodiment.

[0016] Figure 5 : is a block diagram showing a schematic configuration example of the address event detection unit of the first embodiment.

[0017] Figure 6 is a circuit diagram showing another example of the current-voltage conversion unit according to the first embodiment.

[0018] Figure 7 : is a circuit diagram showing a schematic configuration example of a subtractor and a quantizer according to the first embodiment.

[0019] Figure 8 : is a block diagram showing an outline configuration example of a vehicle control system as an example of the moving body control system according to the first embodiment.

[0020] Fig. 9 : is a view showing an example of the installation position of the DVS with respect to the vehicle according to the first embodiment.

[0021] Fig.10 is a flowchart showing an example of an object detection operation performed by the object detection device according to the first embodiment.

[0022] Fig.11 It is a diagram for explaining the flicker component detected within the viewing angle.

[0023] Fig.12 : is a block diagram showing a schematic configuration example of an object detection device (or system) according to the second embodiment.

[0024] Fig.13 is a flowchart showing an example of an object detection operation performed by the object detection device according to the second embodiment.

[0025] Fig.14 is a flowchart illustrating an example of an object detection operation according to a modification example of the second embodiment.

[0026] Fig.15 is a flowchart showing an example of an object detection operation performed by the object detection device according to the third embodiment.

[0027] Fig.16 is a flowchart showing an example of an object detection operation performed by the object detection device according to the fourth embodiment.

[0028] Fig.17 is a flowchart showing an action mode determination operation according to a first example of the fifth embodiment.

[0029] Fig.18 is a flowchart showing an action mode determination operation according to a second example of the fifth embodiment.

[0030] Fig.19 is a flowchart showing an action mode determination operation according to a third example of the fifth embodiment.

[0031] Fig. 20 is a flowchart showing an operation mode determination operation according to a fourth example of the fifth embodiment. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. Note that in the following embodiments, the same parts are denoted by the same reference numerals, and description thereof will not be repeated.

[0033] Furthermore, the present disclosure is described according to the following item order.

[0034] 1. First Implementation

[0035] 1.1 Configuration Example of Object Detection Device (or System)

[0036] 1.2DVS Configuration Example

[0037] 1.3 Configuration example of unit pixel

[0038] 1.4 Configuration Example of Address Event Detection Unit

[0039] 1.4.1 Configuration example of current-voltage conversion unit

[0040] 1.4.2 Configuration Example of Subtractor and Quantizer

[0041] 1.5 Application examples of mobile objects

[0042] 1.6 Example of DVS layout

[0043] 1.7 Object Detection Operation Example

[0044] 1.8 Function and Effect

[0045] 2. Second Implementation

[0046] 2.1 Configuration Example of Object Detection Device (or System)

[0047] 2.2 Object Detection Operation Example

[0048] 2.3 Function and Effect

[0049] 2.4 Variations

[0050] 3. Third Implementation

[0051] 3.1 Object Detection Operation Example

[0052] 3.2 Function and Effect

[0053] 4. Fourth Implementation

[0054] 4.1 Object Detection Operation Example

[0055] 4.2 Function and Effect

[0056] 5. Fifth Implementation

[0057] 5.1 First Example

[0058] 5.2 Second Example

[0059] 5.3 Third Example

[0060] 5.4 Fourth Example

[0061] 1. First Implementation

[0062] First, the first embodiment is described in detail with reference to the drawings.

[0063] As mentioned above, objects such as traffic lights, electronic billboards, and vehicle lights emit flicker components that flash repeatedly at high speed. Figure 1 As shown, when there are traffic lights 51 and vehicles 52 and 53 ahead within the viewing angle G1, the lit lights of the traffic lights 51, the lit arrows of the arrow-type traffic lights, the tail lights, brake lights and flash lights of the vehicles 52 and 53 ahead emit flashing components that flash repeatedly at high speed.

[0064] It is difficult to obtain characteristics such as the frequency and duty cycle of a flicker component emitted by such an object (hereinafter referred to as flicker characteristics) through a synchronous image sensor which obtains image data at a predetermined frame rate in synchronization with a synchronization signal such as a vertical synchronization signal.

[0065] Therefore, in the present embodiment, an asynchronous image sensor (hereinafter referred to as a dynamic vision sensor (DVS)) is used to obtain characteristics of a flicker component emitted by an object, wherein a detection circuit is provided for each pixel to detect in real time that the amount of light received exceeds a threshold value as an address event.

[0066] In a general DVS, a so-called event-driven driving system is adopted, in which whether an address event occurs in each unit pixel is detected, and in the case where the occurrence of the address event is detected, a pixel signal is read from the unit pixel where the address event occurs.

[0067] Furthermore, since a read operation is performed on a unit pixel where occurrence of an address event is detected in a general DVS, the DVS has a characteristic of being able to perform read at a much higher speed compared to a synchronous image sensor that performs a read operation on all unit pixels.

[0068] By using a DVS having such characteristics, the flicker characteristics of an object can be detected with high accuracy. Therefore, an object detection system and an object detection method capable of detecting an object at a higher speed and with higher accuracy can be realized.

[0069] Note that a unit pixel in this specification is a minimum unit of a pixel including one photoelectric conversion element (also referred to as a light receiving element), and corresponds to, for example, each dot in image data read from an image sensor. In addition, an address event is an event that occurs in each address assigned to each of a plurality of unit pixels arranged in a two-dimensional dot matrix pattern, for example, an event in which a current value of a current based on a charge generated in a photoelectric conversion element (hereinafter referred to as a photocurrent) or an amount of change thereof exceeds a specific threshold value.

[0070] 1.1 Configuration Example of Object Detection Device (or System)

[0071] Figure 2 1 is a block diagram showing a schematic configuration example of an object detection device (or an object detection system, hereinafter referred to as an object detection device) according to the first embodiment. Figure 2 As shown, the object detection device 1 has, for example, an imaging lens 11 , a DVS (first solid-state imaging device) 200 , and a data processing unit 120 , and monitors occurrence of an address event in the DVS 200 based on an instruction from an external control unit 130 .

[0072] The imaging lens 11 is an example of an optical system that condenses incident light and forms an image thereof on a light receiving surface of the DVS 200. The light receiving surface may be a surface where a photoelectric conversion element is arranged in the DVS 200.

[0073] The DVS 200 detects the occurrence of an address event based on the amount of incident light and generates address information for specifying a unit pixel where the occurrence of the address event is detected as event detection data. The event detection data may include time information, such as a timestamp indicating the time when the occurrence of the address event is detected.

[0074] The generated event detection data is output to the data processing unit 120 via the signal line 209 .

[0075] The data processing unit 120 is provided with, for example, a memory 121 and a DSP (Digital Signal Processor) 122 .

[0076] The memory 121 includes, for example, a flash memory, a DRAM (Dynamic Random Access Memory), an SRAM (Static Random Access Memory), or the like, and records data input from the DVS 200 .

[0077] For example, the DSP 122 performs predetermined signal processing on the event detection data input from the DVS 200 to perform object detection processing on the object causing the address event to occur.

[0078] More specifically, the DSP 122 specifies an area (corresponding to a flicker detection area described later) for detecting flicker components (serving as a flicker detection unit) based on, for example, the number of event detection data (corresponding to the event detection number N described later) input from the DVS 200 in a unit time, a duty ratio, etc. Then, for example, the DSP 122 performs object detection processing (serving as an object detection unit) on an object causing the occurrence of an address event based on the specified flicker detection area.

[0079] At this time, for example, by executing the program stored in the memory 121, the DSP 122 can function as a machine learning unit using a deep neural network (DNN). In this case, the DSP 122 executes the program of the learning model stored in the memory 121, thereby performing a process of multiplying the dictionary coefficient stored in the memory 121 by the event detection data. Note that as a method of machine learning, various methods such as RNN (Recurrent Neural Network) and CNN (Convolutional Neural Network) can be used.

[0080] Note that the DSP 122 does not necessarily have to perform all steps of the object detection process, but may perform at least some steps. For example, in the case of detecting objects from event detection data using a CNN, the DSP 122 may perform a convolutional layer and / or a pooling layer as part of a hidden layer.

[0081] The results obtained by such object detection processing (also including the results obtained by DSP 122 performing a portion of the object detection processing) are output to, for example, an external information processing device (corresponding to the vehicle external information detection unit 12030 and / or microcomputer 12051 described later) or the like.

[0082] The control unit 130 includes, for example, a CPU (Central Processing Unit) or the like, and controls each unit in the object detection device 1 , for example, the DVS 200 , by outputting various instructions via a signal line 139 .

[0083] In the above configuration, the DVS 200 and the data processing unit 120 may include, for example, a single semiconductor chip. In addition, the semiconductor chip may be a multi-layer chip obtained by attaching a plurality of dies to each other.

[0084] 1.2DVS Configuration Example

[0085] Subsequently, a configuration example of the DVS 200 is described in detail with reference to the drawings. Figure 3 1 is a block diagram showing an example of a functional configuration of a DVS according to the first embodiment. Figure 3 As shown, the DVS 200 is equipped with a driving circuit 211 , a signal processing unit 212 , an arbitrator 213 , and a pixel array unit 300 .

[0086] In the pixel array unit 300, a plurality of unit pixels are arranged in a two-dimensional dot matrix pattern. As described in detail later, the unit pixel includes a photoelectric conversion element such as a photodiode and a pixel circuit (in this embodiment, corresponding to the address event detection unit 400 described later), which detects whether an address event occurs based on whether the current value of the photocurrent of the charge generated in the photoelectric conversion element or the amount of change thereof exceeds a predetermined threshold. Here, the pixel circuit can be shared by a plurality of photoelectric conversion elements. In this case, each unit pixel includes a photoelectric conversion element and a shared pixel circuit.

[0087] The plurality of unit pixels of the pixel array unit 300 may be grouped into a plurality of pixel blocks, each pixel block including a predetermined number of unit pixels. Hereinafter, a group of unit pixels or pixel blocks arranged in a horizontal direction is referred to as a "row", and a group of unit pixels or pixel blocks arranged in a direction perpendicular to the row is referred to as a "column".

[0088] When the occurrence of an address event is detected in the pixel circuit, each unit pixel outputs a request for reading a signal from the unit pixel to the arbiter 213 .

[0089] The arbitrator 213 arbitrates requests from one or more unit pixels and transmits a predetermined response to the unit pixel that issued the request based on the result of the arbitration. The unit pixel that receives the response outputs a detection signal indicating the occurrence of an address event to the driving circuit 211 and the signal processing unit 212.

[0090] The driving circuit 211 sequentially drives the unit pixels that output the detection signals, thereby allowing the unit pixels that detect the occurrence of the address event to output a signal according to the light reception amount to, for example, the signal processing unit 212. Note that the DVS 200 may be equipped with an analog-to-digital converter for converting a signal read from a photoelectric conversion element 333 described later into a signal of a digital value according to the amount of charge thereof (for example, for each unit pixel or a plurality of unit pixels, or for each column).

[0091] The signal processing unit 212 performs predetermined signal processing on the signal input from the unit pixel, and supplies the result of the signal processing as event detection data to the data processing unit 120 via the signal line 209. Note that, as described above, the event detection data may include address information of the unit pixel where the occurrence of the address event is detected, and time information such as a timestamp indicating the timing of the occurrence of the address event.

[0092] 1.3 Configuration example of unit pixel

[0093] Next, a configuration example of the unit pixel 310 is described. Figure 4 1 is a circuit diagram showing a schematic configuration example of a unit pixel according to the first embodiment. Figure 4 As shown, the unit pixel 310 is provided with, for example, a light receiving unit 330 and an address event detection unit 400. Note that Figure 4 The logic circuit 210 may be, for example, a Figure 3 The driving circuit 211, the signal processing unit 212 and the logic circuit of the arbiter 213 are shown in FIG.

[0094] The light receiving unit 330 is equipped with a photoelectric conversion element 333 such as a photodiode, and an output thereof is connected to the address event detection unit 400 .

[0095] The address event detection unit 400 is provided with, for example, a current voltage conversion unit 410 and a subtractor 430. However, in addition to them, the address event detection unit 400 is also provided with a buffer, a quantizer, and a transmission unit. Figure 5 etc. describe the address event detection unit 400 in detail.

[0096] In such a structure, the photoelectric conversion element 333 of the light receiving unit 330 photoelectrically converts incident light to generate charges. The charges generated in the photoelectric conversion element 333 are input to the address event detection unit 400 as photocurrent of a current value according to the amount of charges.

[0097] 1.4 Configuration Example of Address Event Detection Unit

[0098] Figure 5 1 is a block diagram showing a schematic configuration example of an address event detection unit according to the first embodiment. Figure 5 As shown, except Figure 5 In addition to the current-to-voltage conversion unit 410 , the subtractor 430 , and the quantizer 440 shown, the address event detection unit 400 is further equipped with a buffer 420 and a transmission unit 450 .

[0099] The current-voltage conversion unit 410 converts the photocurrent from the light receiving unit 330 into a voltage signal of the logarithm thereof and outputs the resulting voltage signal to the buffer 420 .

[0100] The buffer 420 corrects the voltage signal from the current-voltage conversion unit 410 and outputs the corrected voltage signal to the subtractor 430 .

[0101] The subtractor 430 reduces the voltage level of the voltage signal from the buffer 420 according to the row driving signal from the driving circuit 211 , and outputs the reduced voltage signal to the quantizer 440 .

[0102] The quantizer 440 quantizes the voltage signal from the subtractor 430 into a digital signal, and outputs the generated digital signal as a detection signal to the transmission unit 450 .

[0103] The transmission unit 450 transmits the detection signal from the quantizer 440 to the signal processing unit 212, etc. For example, when the occurrence of an address event is detected, the transmission unit 450 outputs a request to the arbitrator 213 to transmit the detection signal of the address event from the transmission unit 450 to the drive circuit 211 and the signal processing unit 212. Then, upon receiving a response to the request from the arbitrator 213, the transmission unit 450 outputs the detection signal to the drive circuit 211 and the signal processing unit 212.

[0104] 1.4.1 Configuration example of current-voltage conversion unit

[0105] Figure 5 The current-voltage conversion unit 410 in the illustrated configuration may be, for example, a so-called source follower current-voltage conversion unit equipped with an LG transistor 411, an amplifying transistor 412, and a constant current circuit 415, such as Figure 4 However, this is not limited to this; this can also be a so-called gain-boosted current-to-voltage converter, which is equipped with, for example Figure 6 Shown are two LG transistors 411 and 413 , two amplifying transistors 412 and 414 , and a constant current circuit 415 .

[0106] like Figure 4 As shown, the source of the LG transistor 411 and the gate of the amplifying transistor 412 are connected to, for example, the cathode of the photoelectric conversion element 333 of the light receiving unit 330. The drain of the LG transistor 411 is connected to, for example, the power supply terminal VDD.

[0107] Further, for example, the source of the amplification transistor 412 is grounded, and the drain thereof is connected to the power supply terminal VDD via the constant current circuit 415. The constant current circuit 415 may include, for example, a load MOS transistor such as a P-type MOS (Metal Oxide Semiconductor) transistor.

[0108] On the contrary, in the case of gain boost type, if Figure 6As shown, the source of the LG transistor 411 and the gate of the amplifying transistor 412 are connected to, for example, the cathode of the photoelectric conversion element 333 of the light receiving unit 330. In addition, the drain of the LG transistor 411 is connected to, for example, the source of the LG transistor 413 and the gate of the amplifying transistor 412. For example, the drain of the LG transistor 413 is connected to the power supply terminal VDD.

[0109] In addition, for example, the source of the amplifier transistor 414 is connected to the gate of the LG transistor 411 and the drain of the amplifier transistor 412. The drain of the amplifier transistor 414 is connected to the power supply terminal VDD via the constant current circuit 415, for example.

[0110] pass Figure 4 or Figure 6 The connection relationship shown forms a ring source follower circuit. Therefore, the photocurrent from the light receiving unit 330 is converted into a logarithmic voltage signal according to its charge amount. Note that each of the LG transistors 411 and 413 and the amplifying transistors 412 and 414 may include, for example, an NMOS transistor.

[0111] 1.4.2 Configuration Example of Subtractor and Quantizer

[0112] Figure 7 1 is a circuit diagram showing a schematic configuration example of a subtractor and a quantizer according to the first embodiment. Figure 7 As shown, the subtractor 430 is provided with capacitors 431 and 433, an inverter 432, and a switch 434. In addition, the quantizer 440 is provided with a comparator 441.

[0113] One end of the capacitor 431 is connected to the output terminal of the buffer 420, and the other end is connected to the input terminal of the inverter 432. The capacitor 433 is connected in parallel to the inverter 432. The switch 434 opens / closes the path connecting the two ends of the capacitor 433 according to the row driving signal.

[0114] The inverter 432 inverts the voltage signal input via the capacitor 431. The inverter 432 outputs the inverted signal to the non-inverting input terminal (+) of the comparator 441.

[0115] When the switch 434 is turned on, the voltage signal Vinit is input to one side of the buffer 420 of the capacitor 431. In addition, the opposite side becomes a virtual ground terminal. For convenience, the potential of the virtual ground terminal is set to zero. At this time, when the capacitance of the capacitor 431 is set to C1, the potential Qinit accumulated in the capacitor 431 is expressed by the following expression (1). On the contrary, since both ends of the capacitor 433 are short-circuited, its accumulated charge is zero.

[0116] Qinit=C1×Vinit (1)

[0117] Next, considering the case where the switch 434 is turned off and the voltage on the buffer 420 side of the capacitor 431 changes to Vafter, the charge Qafter accumulated in the capacitor 431 is expressed by the following expression (2).

[0118] Qafter=C1×Vafter (2)

[0119] In contrast, when the output voltage is set to Vout, the charge Q2 accumulated in the capacitor 433 is expressed by the following expression (3).

[0120] Q2=-C2×Vout (3)

[0121] At this time, the total amount of charge of the capacitors 431 and 433 does not change, so that the following expression (4) holds true.

[0122] Qinit=Qafter+Q2 (4)

[0123] By substituting Expressions (1) to (3) into Expression (4) and transforming, the following Expression (5) is obtained.

[0124] Vout=-(C1 / C2)×(Vafter-Vinit) (5)

[0125] Expression (5) represents the subtraction operation of the voltage signal, and the gain of the subtraction result is C1 / C2. Because it is generally desired to maximize the gain, it is preferred to design C1 to be larger and C2 to be smaller. On the contrary, if C2 is too small, the kTC noise increases and the noise characteristics may deteriorate, so that the capacitance reduction of C2 is limited to the range of allowed noise. In addition, since the address event detection unit 400 including the subtractor 430 is installed for each unit pixel, there is an area limitation in the capacitor C1 and the capacitor C2. Taking these factors into consideration, the values ​​of the capacitor C1 and the capacitor C2 are determined.

[0126] The comparator 441 compares the voltage signal from the subtractor 430 with a predetermined threshold voltage Vth applied to the inverting input terminal (−). The comparator 441 outputs a signal indicating the comparison result to the transmission unit 450 as a detection signal.

[0127] In addition, when the conversion gain of the current-to-voltage conversion unit 410 is set to CG log And when the gain of the buffer 420 is set to '1', the gain A of the entire address event detection unit 400 is expressed by the following expression (6).

[0128] [Mathematical expression 1]

[0129]

[0130] In expression (6), i photo _n represents the photocurrent of the nth unit pixel, and its unit is, for example, ampere (A). N represents the number of unit pixels 310 in the pixel block, and is '1' in the present embodiment.

[0131] 1.5 Application examples to mobile objects

[0132] The object detection device 1 described above can be applied to various products. For example, it can be installed on any type of moving body, such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobility vehicle, an airplane, a drone, a ship, and a robot.

[0133] Figure 8 : is a block diagram showing an outline configuration example of a vehicle control system as an example of the moving body control system according to the first embodiment.

[0134] like Figure 8 As shown, the vehicle control system 12000 is provided with a plurality of electronic control units connected to each other via a communication network 12001. Figure 8 In the example shown, the vehicle control system 12000 is provided with a drive system control unit 12010, a body system control unit 12020, a vehicle exterior information detection unit 12030, a vehicle interior information detection unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown as a functional configuration of the integrated control unit 12050.

[0135] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to wheels, a steering mechanism for adjusting a steering angle of the vehicle, a braking device for generating a braking force of the vehicle, and the like.

[0136] The body system control unit 12020 controls the operation of various devices installed on the vehicle body according to various programs. For example, the body system control unit 12020 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights (such as headlights, reverse lights, brake lights, flashers, or fog lights). In this case, a radio wave transmitted from a portable device instead of a key or a signal of various switches may be input to the body system control unit 12020. The body system control unit 12020 receives the input of the radio wave or the signal and controls the door lock device, the power window device, the lights, etc. of the vehicle.

[0137] The vehicle exterior information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the imaging unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle exterior information detection unit 12030 allows the imaging unit 12031 to obtain information outside the vehicle and receive the obtained data. The vehicle exterior information detection unit 12030 can perform a detection process or a distance detection process of an object such as a person, a vehicle, an obstacle, a sign, or a text on the road surface based on the received data.

[0138] The imaging unit 12031 may be an image sensor that outputs an electrical signal as an image, or may be a ranging sensor that outputs the same as ranging information. In addition, the light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0139] The imaging unit 12031 is provided with, for example, a plurality of imaging units 12101 to 12105 (see Fig. 9 ). The above-mentioned DVS 200 is used for at least one of the plurality of imaging units 12101 to 12105. The DVS 200 and the vehicle exterior information detection unit 12030 connected thereto form the object detection device 1 according to the first embodiment. In this case, the vehicle exterior information detection unit 12030 and / or the microcomputer 12051 are used as the data processing unit 120.

[0140] The vehicle interior information detection unit 12040 detects information of the interior of the vehicle. The vehicle interior information detection unit 12040 is connected to, for example, a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the vehicle interior information detection unit 12040 can calculate the driver's fatigue level or concentration level, or can determine whether the driver is dozing off based on the detection information input from the driver state detection unit 12041.

[0141] The microcomputer 12051 can perform arithmetic operations of control target values ​​of a driving force generating device, a steering mechanism, or a braking device based on the vehicle interior information obtained by the vehicle interior information detection unit 12040 and / or the vehicle exterior information obtained by the vehicle exterior information detection unit 12030, and output a control instruction to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control for realizing functions of an ADAS (Advanced Driver Assistance System), including collision avoidance or impact attenuation of a vehicle, following driving based on a vehicle-to-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, and the like.

[0142] In addition, the microcomputer 12051 can perform collaborative control for achieving automatic driving, etc., by controlling the driving force generating device, steering mechanism, braking device, etc. based on the information around the vehicle obtained by the vehicle external information detection unit 12030 or the vehicle internal information detection unit 12040 to achieve autonomous driving, etc., so as to drive autonomously independently of the driver's operation.

[0143] In addition, the microcomputer 12051 may output a control instruction to the body system control unit 12020 based on the vehicle exterior information obtained by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 may control the headlights according to the position of the preceding vehicle or the oncoming vehicle detected by the vehicle exterior information detection unit 12030 to perform cooperative control for achieving glare protection, such as switching the high beam to the low beam.

[0144] The audio image output unit 12052 transmits at least one of the audio or image output signals to an output device that can visually or auditorily notify the occupants of the vehicle or the outside of the vehicle of the information. In the example of FIG. 72 , an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are shown as output devices. The display unit 12062 may include, for example, at least one of an onboard display or a head-up display.

[0145] 1.6 Example of DVS layout

[0146] Fig. 9 1 is a view showing an example of the installation position of the DVS with respect to the vehicle according to the first embodiment. Fig. 9 , as the imaging unit 12031, a total of five imaging units including imaging units 12101, 12102, 12103, 12104, and 12105 are provided for the vehicle 12100. At least one of the imaging units 12101 to 12105 includes the DVS 200. For example, the DVS 200 is used for the imaging unit 12101 to image the area in front of the vehicle 12100.

[0147] The imaging units 12101, 12102, 12103, 12104, and 12105 are disposed in positions such as the front nose, side mirrors, rear bumper, rear door, and upper portion of the windshield in the vehicle interior of the vehicle 12100. The imaging unit 12101 disposed on the front nose and the imaging unit 12105 disposed in the upper portion of the windshield in the vehicle interior mainly obtain images of the area in front of the vehicle 12100. The imaging units 12102 and 12103 disposed on the side mirrors mainly obtain images of the sides of the vehicle 12100. The imaging unit 12104 disposed on the rear bumper or rear door mainly obtains images of the area behind the vehicle 12100. The imaging unit 12105 disposed in the upper portion of the windshield in the vehicle interior is mainly used to detect vehicles ahead, pedestrians, obstacles, traffic lights, traffic signs, lanes, and the like.

[0148] In addition, Fig. 9 , an example of the imaging ranges of the imaging units 12101 to 12104 is shown. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, and the imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or the rear door. For example, the image data imaged by the imaging units 12101 to 12104 are superimposed, thereby obtaining a top-view image of the vehicle 12100 observed from above.

[0149] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera including a plurality of imaging elements, or may be an imaging element including pixels for phase difference detection.

[0150] For example, the microcomputer 12051 (reference Figure 8 ) can specifically extract the closest three-dimensional object on the driving path of the vehicle 12100, which is traveling at a predetermined speed (for example, 0 km / h or higher) in a direction substantially the same as the direction of the vehicle 12100 as the front vehicle, based on the distance information obtained from the imaging units 12101 to 12104, by obtaining the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the time change of the distance (relative speed with the vehicle 12100). In addition, the microcomputer 12051 can also pre-set the inter-vehicle distance to be ensured with the front vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control for realizing automatic driving, etc. can be performed to autonomously travel independently of the driver's operation.

[0151] For example, the microcomputer 12051 can extract entity object data about the entity object while classifying the entity object into motorcycles, standard vehicles, large vehicles, pedestrians, and other entity objects (such as utility poles) based on the distance information obtained from the imaging units 12101 to 12104, and use it to automatically avoid obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 as obstacles visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the degree of danger of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 can perform driving assistance for avoiding collision by outputting a warning to the driver via the audio speaker 12061 and the display unit 12062 or performing forced deceleration or evasive steering via the drive system control unit 12010.

[0152] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light. For example, the microcomputer 12051 may identify a pedestrian by determining whether a pedestrian exists in an image acquired by the imaging units 12101 to 12104. Such pedestrian recognition is performed, for example, by a process of extracting feature points in an image acquired by the imaging units 12101 to 12104 as infrared cameras, and a process of performing pattern matching processing on a series of feature points indicating the outline of an object to discern whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian exists in an image acquired by the imaging units 12101 to 12104 and identifies the pedestrian, the audio image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline to emphasize the identified pedestrian for display. In addition, the audio image output unit 12052 may control the display unit 12062 to display an icon indicating a pedestrian, etc., at a desired position.

[0153] 1.7 Object Detection Operation Example

[0154] Next, an operation example of the object detection device 1 according to the first embodiment is described. Fig.10 1 is a flowchart showing an example of an object detection operation performed by the object detection device according to the first embodiment. Note that in the following description, attention is focused on the operation of the data processing unit 120 .

[0155] like Fig.10 As shown, in this operation, first, the data processing unit 120 inputs event detection data from the DVS 200 (step S101). Note that since the DVS 200 operates asynchronously as described above, when the occurrence of an address event is detected, the event detection data is input from the DVS 200 to the data processing unit 120 as needed.

[0156] In contrast, the data processing unit 120 specifies one or more regions (hereinafter referred to as event detection regions) in which the occurrence of an address event is detected based on the address information included in the input event detection data (step S102). Figure 1 In the case where the object shown is included in the field of view of the DVS 200, the data processing unit 120 specifies the area R1 of the traffic light 51 and the areas R2 and R3 of the tail lights, brake lights, etc. of the leading vehicles 52 and 53. Note that, for example, in step S102, one or more event detection areas where the occurrence of the address event is detected may be specified based on one or more event detection data input within a predetermined time (e.g., 1 millisecond (ms)).

[0157] Next, the data processing unit 120 counts the number of address events (hereinafter, referred to as the number of detected events) N whose occurrence is detected per unit time (e.g., 1 second) for each event detection area specified in step S102 (step S103). Note that the number of detected events N is a value for determining whether each event detection area includes a flicker component, and is a characteristic value corresponding to the frequency of the flicker component. As the number of detected events N, a value counted for each address (i.e., pixel) belonging to each event detection area may be used, or a maximum value, an average value, etc. of the value counted for each address in each event detection area may be used.

[0158] Next, the data processing unit 120 determines whether there is an event detection area in which the number N of detected events is equal to or greater than a preset threshold value Nth (step S104). In the case that there is no event detection area in which the number N of detected events is equal to or greater than the threshold value Nth (No in step S104), the data processing unit 120 determines that no flickering object is detected, and proceeds to step S108.

[0159] On the contrary, in the case where there is an event detection area in which the number N of detected events is equal to or greater than the threshold value Nth (yes in step S104), the data processing unit 120 specifies an area in which flicker is detected (hereinafter, referred to as flicker detection area) from the above-mentioned one or more event detection areas (step S105).

[0160] Next, the data processing unit 120 detects an object imaged in each flicker detection area based on the edge shape of the flicker detection area specified in step S105 and the number N of detection events counted for the flicker detection area (step S106 ).

[0161] Note that the object in each flicker detection area can be detected based on, for example, the edge shape of the object specified by the address information and the characteristics such as the frequency and duty cycle of the flicker component specified by the number of detected events N. For example, the object can be detected by registering the correspondence between the edge shape, flicker characteristics, etc. and the object in the memory 121 and performing pattern matching between the detected edge shape, flicker characteristics, etc. of the registered object and the edge shape, flicker characteristics, etc. Alternatively, the object can also be detected by machine learning using the edge shape, flicker characteristics, etc. as input.

[0162] Next, the data processing unit 120 outputs the result of the object detection processing performed as described above to, for example, the integrated control unit 12050 via the communication network 12001 (step S107 ), and proceeds to step S108 .

[0163] In step S108, the data processing unit 120 determines whether to end the operation, and in the case of ending (yes in step S108), the operation is ended. On the contrary, in the case of continuing (no in step S108), the data processing unit 120 returns to step S101 and performs subsequent operations.

[0164] 1.8 Actions and Effects

[0165] As described above, according to the present embodiment, an object can be detected in consideration of not only the edge shape of the object but also the characteristics of flicker generated by the object, etc. Therefore, the object can be detected at a higher speed and with higher accuracy.

[0166] 2. Second Implementation

[0167] Next, the second embodiment is described in detail with reference to the drawings. Note that in the following description, similar configurations and operations to those of the first embodiment are cited, and descriptions thereof will not be repeated.

[0168] In the first embodiment described above, a case where an object emitting a flicker component is detected using event detection data obtained from the DVS 200 is exemplified. In contrast, in the second embodiment, a case where an object emitting a flicker component is detected using image data obtained by a normal image sensor in addition to event detection data is exemplified.

[0169] For example, Fig.11 As shown in FIG. 1 , when the road surface is wetted by rain or the like, the light output from the headlight of the oncoming vehicle 55 is reflected on the road surface; in this case, not only the area R5 of the headlight but also the area R5a on the road surface reflecting the light of the headlight has a flicker component in the viewing angle G2. Therefore, in object detection based only on the flicker component, false detection may occur, such as detecting an object on the road surface where nothing originally exists. Note that in Fig.11 In the image, region R4 corresponding to the street light is also detected as a region having a flicker component.

[0170] Therefore, in the present embodiment, by combining event detection data and image data, the occurrence of erroneous detection is suppressed and more accurate object detection is enabled.

[0171] 2.1 Configuration Example of Object Detection Device (or System)

[0172] Fig.12 1 is a block diagram showing a schematic configuration example of an object detection device (or system) according to the second embodiment. Fig.12 As shown, the object detection device 2 is provided with an imaging lens 110, a DVS 200, a flicker detection unit 12, an imaging lens 13, an image sensor (a second solid-state imaging device) 14, an object detection unit 15, an action mode determination unit 16, a storage unit 17, and an interface (I / F) unit 18. In this configuration, the imaging lens 11 and the DVS 200 may be similar to the imaging lens 11 and the DVS 200 according to the first embodiment.

[0173] As with the imaging lens 11, the imaging lens 13 is an example of an optical system that converges incident light and forms an image thereof on a light receiving surface of the image sensor 14. The light receiving surface may be a surface in which the photoelectric conversion elements are arranged in the image sensor 14.

[0174] The image sensor 14 may be any of a variety of image sensors capable of acquiring image data, such as a CCD (charge coupled device) image sensor and a CMOS (complementary metal oxide semiconductor) image sensor. The image sensor 14 has, for example, a Figure 8 The imaging unit 12031 in FIG. 1 is configured and mounted on the vehicle 12100 to image at a viewing angle substantially the same as that of the DVS 200.

[0175] The flicker detection unit 12 specifies one or more flicker detection areas based on, for example, event detection data input from the DVS 200. The flicker detection unit 12 may be, for example, the data processing unit 120 in the first embodiment. In this case, the flicker detection unit 12 may not only specify the flicker detection areas but also detect an object imaged in each flicker detection area.

[0176] The object detection unit 15 detects an object imaged in each flicker detection area by using the flicker detection area detected by the flicker detection unit 12 and the image data input from the image sensor 14 .

[0177] The action mode determination unit 16 determines the action mode of the vehicle 12100 based on, for example, an object detected by the flicker detection unit 12 or an object detected by the object detection unit 15. The action mode determination unit 16 may be, for example, Figure 8 The microcomputer 12051 may be a computer different from the microcomputer 12051.

[0178] Storage unit 17 stores various programs and data necessary for operation mode determination unit 16 to determine the operation mode of vehicle 12100 .

[0179] The I / F unit 18 is an interface for transmitting and receiving data with the integrated control unit 12050 via the communication network 12001, for example, when the operation mode determination unit 16 is a computer different from the microcomputer 12051. However, when the operation mode determination unit 16 is the microcomputer 12051, the I / F unit 18 corresponds to, for example, the in-vehicle network I / F 12053.

[0180] 2.2 Object Detection Operation Example

[0181] Next, an operation example of the object detection device 2 according to the second embodiment is described. Fig.13 is a flowchart showing an example of an object detection operation performed by the object detection device according to the second embodiment. Fig.10 The operations described are similar to the operations described above, and their description will not be repeated.

[0182] like Fig.13 As shown, in this operation, first, the flicker detection unit 12 performs the same operation as in the first embodiment. Fig.10 Note that area information (eg, address information) for specifying each flicker detection area is input from the flicker detection unit 12 to the object detection unit 15.

[0183] Next, the image data is input from the image sensor 14 to the object detection unit 15, and the area information of the flicker detection area is input to the object detection unit 15 (step S201). Note that the image sensor 14 may periodically output the image data at a predetermined frame rate, or may output the image data at a predetermined frame rate by the object detection unit 15 or the control unit 130 (see Figure 1 ) to output image data at the timing indicated.

[0184] The object detection unit 15 that inputs the image data specifies an area of ​​the object including the flicker detection area (hereinafter referred to as the object detection area) in the input image data (step S202). Fig.11In the example shown, the area of ​​the vehicle 55 including the area R5 corresponding to the flicker detection area is designated as the object detection area R6. On the other hand, since the area R5a corresponding to the flicker detection area is not included in the specific object, the object detection unit 15 does not designate the area R5a as an object detection area.

[0185] Next, the object detection unit 15 detects the object imaged in each object detection area based on the edge shape of the object included in the object detection area in the image data specified in step S202 and the number N of events detected in the flicker detection area superimposed on the object detection area (step S203). Note that pattern matching and machine learning may be used for object detection, for example, as Fig.10 Step S106 in .

[0186] Next, the object detection unit 15 outputs the object detection result to, for example, the integrated control unit 12050 via the communication network 12001 (step S107). Fig.10 The subsequent steps in the process are then performed, and then, in step S108, it is determined whether the operation is completed.

[0187] 2.3 Function and Effect

[0188] As described above, according to the present embodiment, the flicker detection area for detecting flicker components is masked by the object detection area specified from the image data. This can reduce or avoid the occurrence of erroneous detections such as detection of objects on the road surface that do not exist, and can detect objects with higher accuracy.

[0189] Other configurations, operations, and effects may be similar to those of the above-described embodiment, and thus detailed descriptions thereof are omitted here.

[0190] 2.4 Variations

[0191] In the second embodiment described above, an example is given of reducing or avoiding the occurrence of false detection by masking the flicker information (e.g., flicker detection area and / or the number N of detected events) obtained from the event detection data with the object information (e.g., object detection area) obtained from the image data. However, the method of reducing / avoiding false detection by combining flicker information and object information is not limited to this method. For example, false detection can also be reduced or avoided by supporting object detection of image data using flicker information.

[0192] Fig.14 1 is a flowchart showing an example of a modified object detection operation according to the second embodiment. Fig.14As shown, in this variation, first, the image data is input to the object detection unit 15 (step S221). The object detection unit 15 to which the image data is input performs object detection processing on the input image data, thereby specifying one or more regions (object detection regions) of the object included in the image data (step S222). For example, in Fig.11 In the illustrated example, the object detection unit 15 specifies the object detection region R6 including the vehicle 55. Note that region information (eg, address information) for specifying each object detection region is input from the object detection unit 15 to the flicker detection unit 12.

[0193] Next, the flicker detection unit 12 counts the number N of events detected per unit time in each object detection area (step S223). Note that as the number N of detected events, a value counted for each address (i.e., pixel) belonging to each object detection area may be used, or a maximum value, an average value, etc. in each area of ​​the value counted for each address may be used. Note that the number N of detected events for each object detection area is input from the flicker detection unit 12 to the object detection unit 15.

[0194] Next, the object detection unit 15 detects the object imaged in each object detection area based on the edge shape of the object included in the object detection area in the image data specified in step S222 and the number N of events detected in the object detection area (step S224). Note that pattern matching and machine learning can be used for object detection, for example, Fig.10 Step S106 in .

[0195] Next, the object detection unit 15 outputs the object detection result to, for example, the integrated control unit 12050 via the communication network 12001 (step S107). Fig.10 The subsequent steps in the process are then performed, and then, in step S108, it is determined whether the operation is completed.

[0196] According to the operation as described above, since object detection for image data is supported by using flicker information, the occurrence of erroneous detection such as detecting an object on the road surface that does not originally exist can be reduced or avoided, and object detection can be performed with higher accuracy as in the second embodiment.

[0197] 3. Third Implementation

[0198] Next, the third embodiment is described in detail with reference to the drawings. Note that in the following description, similar configurations and operations to those of the above-described embodiment are cited, and descriptions thereof will not be repeated.

[0199] For example, an object emitting a flicker component such as a traffic light, an electronic billboard, or a car light is not a point but has a specific area. In contrast, most of the false detections output from the DVS 200 are very small areas of one or a few points.

[0200] Therefore, in this embodiment, a lower limit is set in the size of the area designated as the flicker detection area, and an area smaller than the lower limit is not set as the flicker detection area, thereby reducing erroneous detection of objects and reducing the amount of data to be processed in the object detection process.

[0201] A configuration example of the object detection device (or system) according to the present embodiment may be similar to, for example, a configuration example of the object detection device 1 illustrated in the first embodiment or the object detection device 2 illustrated in the second embodiment.

[0202] 3.1 Object Detection Operation Example

[0203] Fig.15 1 is a flowchart showing an example of an object detection operation performed by an object detection device according to the third embodiment. Note that in the following description, operations similar to those in the above-described embodiment are cited, and their descriptions are not repeated. In addition, in the following description, an example of an operation based on the reference in the second embodiment is given. Fig.13 The operation is described above, but the embodiment as a basis is not limited to the second embodiment, and may be other embodiments described above or later or variations thereof.

[0204] like Fig.15 As shown, in the object detection operation according to the present embodiment, for example, in the second embodiment, as in the reference Fig.13 In the same operation as described above, step S301 is added between step S104 and step S105.

[0205] In step S301, the flicker detection unit 12 determines whether there are a certain number or more of pixels (i.e., addresses) in each event detection area specified in step S102, where the number N of detected events is equal to or greater than the threshold value Nth. Note that the specific number may be, for example, a number corresponding to a size that is larger than the area size by erroneously detecting address events.

[0206] In the case where there is no event detection area in which the number N of detected events is equal to or greater than the threshold value Nth by a certain number or more of pixels (No in step S301), the operation proceeds to step S108. In contrast, in the case where there is such a case (Yes in step S301), the flicker detection unit 12 specifies the event detection area in which the number N of detected events is equal to or greater than the threshold value Nth by a certain number or more of pixels as the flicker detection area (step S105), and performs the subsequent operation.

[0207] 3.2 Actions and Effects

[0208] As described above, according to the present embodiment, since an event detection area having a specific size or larger is designated as a flicker detection area, erroneous detection of an object is reduced and higher-precision object detection becomes possible, and the amount of data to be processed in the object detection process is reduced and higher-speed object detection becomes possible.

[0209] Other configurations, operations, and effects may be similar to those of the above-described embodiment, and thus detailed descriptions thereof are omitted here.

[0210] 4. Fourth Implementation

[0211] Next, the fourth embodiment is described in detail with reference to the drawings. Note that in the following description, configurations and operations similar to those of the above-described embodiment are cited, and descriptions thereof will not be repeated.

[0212] In the above-described embodiment, the occurrence of address events is monitored without changing the resolution between the flicker detection area and other areas. In addition, in the second and third embodiments, image data of the same resolution is read from the flicker detection area and other areas.

[0213] In contrast, in the third embodiment, the resolution when monitoring the occurrence of an address event and / or the resolution of image data read from the image sensor 14 is changed between the flicker detection area and the other areas.

[0214] As a result, for example, the amount of data processing in areas other than the flicker detection area can be reduced, and high-speed object detection can be achieved.

[0215] A configuration example of the object detection device (or system) according to the present embodiment may be similar to, for example, a configuration example of the object detection device 1 illustrated in the first embodiment or the object detection device 2 illustrated in the second embodiment.

[0216] 4.1 Object Detection Operation Example

[0217] Fig.16is a flowchart showing an example of an object detection operation performed by an object detection device according to the fourth embodiment. Note that in the following description, operations similar to those in the above-described embodiment are cited and their descriptions are not repeated. In addition, in the following description, an operation based on the third embodiment is exemplified. Fig.15 The operation shown is the case, but the basic embodiment is not limited to the third embodiment, and other embodiments described above or later or their modifications may be used.

[0218] like Fig.16 As shown, in the object detection operation according to the present embodiment, for example, in the third embodiment, as in the reference Fig.15 In the same operation as described above, step S401 is added between step S107 and step S108.

[0219] In step S401, the flicker detection unit 12 sets a region other than the flicker detection region specified in step S105 as a low-resolution region. The setting of the low-resolution region can be performed on the control unit 130, for example.

[0220] On the contrary, for the low-resolution area in the DVS 200, the control unit 130 sets the resolution when monitoring the occurrence of address events to be low. For example, for the low-resolution area in the DVS 200, the control unit 130 controls the DVS 200 to stop monitoring the occurrence of address events of the unit pixels 310 in the odd-numbered rows and only monitor the occurrence of address events of the unit pixels 310 in the even-numbered rows.

[0221] Furthermore, the control unit 130 reduces the resolution of image data read from a low-resolution area in the image sensor 14. For example, for a low-resolution area in the image sensor 14, the control unit 130 reads pixel signals only from pixels in odd-numbered rows to generate image data.

[0222] 4.2 Actions and Effects

[0223] As described above, according to the present embodiment, the amount of data processing in areas other than the flicker detection area can be reduced, thereby enabling object detection to be performed at a higher speed.

[0224] Other configurations, operations, and effects may be similar to those of the above-described embodiment, and thus detailed descriptions thereof are omitted here.

[0225] 5. Fifth Implementation

[0226] Next, the fifth embodiment is described in detail with reference to the drawings. Note that in the following description, configurations and operations similar to those of the above-described embodiment are cited, and descriptions thereof will not be repeated.

[0227] In the fifth embodiment, the action mode determination operation performed by the microcomputer 12051 or the action mode determination unit 16 in the above-mentioned embodiments is described by some examples. Note that in the following description, for the sake of clarity, it is assumed that the action mode determination unit 16 performs the action mode determination operation.

[0228] 5.1 First Example

[0229] Fig.17 1 is a flowchart showing an operation of determining an action mode according to the first example. In addition, in the first example, a case where the object detection device 1 or 2 according to the above-described embodiment detects a traffic light and the vehicle 12100 is waiting at the traffic light is illustrated. In addition, at this time, it is assumed that the engine is automatically stopped by the idle stop system installed on the vehicle 12100.

[0230] like Fig.17 As shown, in the first example, the action mode determination unit 16 first waits for the input of the object detection result from the object detection device 1 or 2 (No in step S501), and when the object detection result is input (Yes in step S501), this recognizes that a traffic light is detected based on the object detection result (step S502).

[0231] Next, the action mode determination unit 16 specifies the current lighting color of the traffic light according to the object detection result (step S503). Then, in the case where the current lighting color is blue (yes in step S504), the action mode determination unit 16 starts the engine (step S505) and proceeds to step S507. At this time, if the automatic driving function of the vehicle 12100 is effective, the action mode determination unit 16 can start the forward movement of the vehicle 12100 together with the start of the engine.

[0232] In contrast, in the case where the current lighting color is a color other than blue, for example, red (No in step S504 ), the action mode determination unit 16 continues to stop (step S506 ), and proceeds to step S507 .

[0233] In step S507, the action mode determination unit 16 determines whether to end the operation. If it is ended (Yes in step S507), the operation is ended. On the contrary, if it is continued (No in step S507), the action mode determination unit 16 returns to step S501 and performs subsequent operations.

[0234] 5.2 Second Example

[0235] Fig.182 is a flowchart showing an operation mode determination operation according to the second example. In addition, in the second example, a case where an oncoming vehicle is detected by the object detection device 1 or 2 according to the above-described embodiment while the vehicle 12100 is traveling is illustrated.

[0236] like Fig.18 As shown, in the second example, first, when the object detection result is input from the object detection device 1 or 2 (Yes in step S521), the action mode determination unit 16 detects the headlights as an oncoming vehicle based on the object detection result (step S522). Subsequently, the action mode determination unit 16 detects the oncoming vehicle from the area including the headlights in the object detection result (step S523).

[0237] When the presence of an oncoming vehicle is detected in this way, the action mode determination unit 16 drives Figure 8 The audio image output unit 12052 in the control unit 12054 is used to issue a warning to the driver, for example, from the audio speaker 12061 to notify the driver of the presence of the oncoming vehicle and / or to display the presence of the oncoming vehicle on the display unit 12062 (step S524), and proceed to step S525.

[0238] In step S525, the action mode determination unit 16 determines whether to end the operation. If it is ended (yes in step S525), the operation is ended. On the contrary, if it is continued (no in step S525), the action mode determination unit 16 returns to step S521 and performs subsequent operations.

[0239] 5.3 Third Example

[0240] Fig.19 : is a flowchart showing an action mode determination operation according to the third example. In addition, in the third example, another action mode determination operation in the case where an oncoming vehicle is detected by the object detection device 1 or 2 according to the above-described embodiment while the vehicle 12100 is traveling is illustrated.

[0241] like Fig.19 As shown, in the third example, the action mode determination unit 16 first performs the same operation as in the second example. Fig.18 An oncoming vehicle is detected by performing operations similar to the operations described in steps S521 to S523 in FIG.

[0242] Next, the action mode determination unit 16 calculates the motion vector of the oncoming vehicle based on the object detection results continuously input within a predetermined period of time (step S531). Note that, for example, in step S531, the centroid or center of the object detection area corresponding to the oncoming vehicle may be calculated, and the motion vector of the oncoming vehicle may be calculated based on the amount of movement and the moving direction of the centroid or center per unit time.

[0243] Next, the action mode determination unit 16 determines whether the motion vector calculated in step S531 points to the host vehicle (vehicle 12100) (step S532). In the case where the motion vector does not point to the host vehicle 12100 (No in step S532), the action mode determination unit 16 determines that the collision risk is low or does not exist, and proceeds to step S525.

[0244] On the contrary, in the case where the motion vector of the oncoming vehicle points to the host vehicle 12100 (yes in step S532), the action mode determination unit 16 then determines whether the magnitude M of the motion vector of the oncoming vehicle is equal to or greater than the threshold value Mth set in advance (step S533). In the case where the magnitude M of the motion vector is less than the threshold value Mth (step S533), the action mode determination unit 16 determines that the collision risk is low or does not exist, and proceeds to step S525.

[0245] Furthermore, in the case where the magnitude M of the motion vector is equal to or greater than the threshold value Mth (Yes in step S533), the action mode determination unit 16 detects a collision risk (step S534). Then, the action mode determination unit 16 detects a collision risk by, for example, controlling Figure 8 The drive system control unit 12010 performs automatic braking control and drives Figure 8 The audio image output unit 12052, thereby issuing a warning to the driver from the audio speaker 12061 to inform him of the risk of collision with the oncoming vehicle, and / or displaying the risk of collision with the oncoming vehicle on the display unit 12062 (step S535), and proceeding to step S525.

[0246] In step S525, as in the second example, the action mode determination unit 16 determines whether to end the operation, and if so (yes in step S525), ends the operation, and if so (no in step S525), returns to step S521 and performs subsequent operations.

[0247] 5.4 Fourth Example

[0248] Fig. 20 1 is a flowchart showing an operation mode determination operation according to the fourth example. In the fourth example, a case where the object detection device 1 or 2 according to the above-described embodiment detects that the flasher of the vehicle ahead starts flashing while the vehicle 12100 is traveling is illustrated.

[0249] like Fig. 20As shown, in the fourth example, when the object detection result is input from the object detection device 1 or 2 (Yes in step S541), the action mode determination unit 16 detects that the flashing lights of the vehicle are the front vehicle based on the object detection result (step S542). Subsequently, the action mode determination unit 16 detects the front vehicle from the area including the flashing lights in the object detection result (step S543).

[0250] Next, the action mode determination unit 16 specifies the center of the object detection area corresponding to the detected vehicle ahead (step S544), and specifies the route change direction indicated by the vehicle ahead through the flash light based on the positional relationship between the center position corresponding to the flash light and the flash detection area (step S546).

[0251] Next, the action mode determination unit 16 drives Figure 8 The audio image output unit 12052 in the control unit 12054 may, for example, issue a warning from the audio speaker 12061 to notify the driver of a route change of the vehicle ahead and / or display the route change of the vehicle ahead on the display unit 12062 (step S546), and proceed to step S525.

[0252] In step S525, the action mode determination unit 16 determines whether to end the operation. If it is ended (yes in step S525), the operation is ended. On the contrary, if it is continued (no in step S525), the action mode determination unit 16 returns to step S521 and performs subsequent operations.

[0253] Although the embodiments of the present disclosure are described above, the technical scope of the present disclosure is not limited to the above embodiments, and various modifications can be made without departing from the gist of the present disclosure. In addition, different embodiments and changed components can be appropriately combined.

[0254] Furthermore, the effects described in each embodiment of the present specification are merely illustrative and not restrictive; there may also be another effect.

[0255] Note that the present technology can have the following configurations. (1)

[0257] An object detection device comprising:

[0258] a first solid-state imaging device provided with a plurality of pixels arranged in a matrix, the first solid-state imaging device detecting occurrence of an event in a pixel according to an amount of light incident on each pixel;

[0259] a flicker detection unit that generates flicker information based on the occurrence of an event detected by the first solid-state imaging device; and

[0260] The object detection unit detects an object based on the flicker information detected by the first solid-state imaging device. (2)

[0262] The object detection device according to (1) above, wherein:

[0263] The flicker detection unit specifies an event detection area in which occurrence of an event is detected among a plurality of pixels, and generates flicker information based on the number of occurrence events detected per predetermined time for the event detection area. (3)

[0265] The object detection device according to (2) above, wherein

[0266] The flicker detection unit generates flicker information based on a maximum value or an average value of the number of occurrence events detected per predetermined time in each pixel belonging to the event detection area. (4)

[0268] The object detection device according to (2) or (3) above, wherein:

[0269] The flicker information includes the edge shape of the event detection area and the number of occurrence events detected per predetermined time. (5)

[0271] The object detection device according to any one of (2) to (4) above, wherein:

[0272] The flicker detection unit sets a region in the event detection region where pixels whose number of occurrence events detected per predetermined time is a first threshold or more are arranged as the flicker detection region;

[0273] The object detection unit detects an object imaged in the flicker detection area based on the flicker information. (6)

[0275] The object detection device according to (5) above, wherein:

[0276] The flicker detection unit sets, in the event detection area, a region in which the number of pixels whose number of occurrence events detected per predetermined time is a first threshold value or more is a second threshold value or more as the flicker detection area. (7)

[0278] The object detection device according to (2) above further comprises:

[0279] A second solid-state imaging device obtains image data, wherein:

[0280] The object detection unit detects an object based on the flicker information and the image data. (8)

[0282] The object detection device according to (7) above, wherein:

[0283] a flicker detection unit specifies an event detection area in which occurrence of an event is detected among a plurality of pixels, and generates flicker information based on the number of occurrence events detected per predetermined time for the event detection area,

[0284] The object detection unit specifies an object detection area including an area on the image data corresponding to the event detection area and corresponding to the object, and detects the object based on the object detection area and the flicker information. (9)

[0286] The object detection device according to (7) above, wherein:

[0287] The object detection unit specifies an object detection area corresponding to the object from the image data;

[0288] The flicker detection unit generates flicker information based on the number of occurrence events detected per predetermined time for an area corresponding to the object detection area among the plurality of pixels, and

[0289] The object detection unit detects an object based on the object detection area and the flicker information. (10)

[0291] The object detection device according to (2) above further comprises:

[0292] A control unit controls reading of the first solid-state imaging device, wherein:

[0293] The control unit reduces a resolution in monitoring occurrence of the event for a region in which pixels other than pixels included in the event detection region among the plurality of pixels are arranged. (11)

[0295] The object detection device according to (7) above further comprises:

[0296] A control unit controls reading of the second solid-state imaging device, wherein:

[0297] The control unit reduces the resolution of image data read from the area corresponding to the event detection area in the second solid-state imaging device. (12)

[0299] An object detection system having:

[0300] a first solid-state imaging device provided with a plurality of pixels arranged in a matrix, the first solid-state imaging device detecting occurrence of an event in a pixel according to an amount of light incident on each pixel;

[0301] a flicker detection unit that generates flicker information based on the occurrence of an event detected by the first solid-state imaging device; and

[0302] The object detection unit detects an object based on the flicker information detected by the first solid-state imaging device. (13)

[0304] An object detection method having:

[0305] generating flicker information based on the occurrence of an event detected by a solid-state imaging device provided with a plurality of pixels arranged in a matrix, the solid-state imaging device detecting the occurrence of an event in a pixel according to the amount of light incident on each pixel; and

[0306] Detect objects based on flicker information.

[0307] Explanation of symbols

[0308] 1.2 Object detection device (system)

[0309] 11.13 Imaging lens

[0310] 12 Flicker Detection Unit

[0311] 14 Image Sensor

[0312] 15 Object Detection Unit

[0313] 16 Action mode determination unit

[0314] 17 Storage Unit

[0315] 18 I / F Unit

[0316] 120 Data processing unit

[0317] 121 Memory

[0318] 122 DSP

[0319] 130 Control unit

[0320] 200 DVS

[0321] 210 Logic Circuits

[0322] 211 drive circuit

[0323] 212 Signal Processing Unit

[0324] 213 Arbitrator

[0325] 300 pixel array unit

[0326] 310 pixels per unit

[0327] 330 Light receiving unit

[0328] 333 Photoelectric conversion element

[0329] 400 Address Event Detection Unit

[0330] 410 Current-voltage conversion unit

[0331] 411, 413 LG transistor

[0332] 412, 414 Amplifier transistor

[0333] 415 Constant Current Circuit

[0334] 420 Buffer

[0335] 430 Subtractor

[0336] 431, 433 capacitor

[0337] 432 Inverter

[0338] 434 Switch

[0339] 440 Quantizer

[0340] 441 Comparator

[0341] 450 Transmission Unit

[0342] 12001 Communication Network

[0343] 12010 Drive system control unit

[0344] 12030 Vehicle external information detection unit

[0345] 12031, 12101 to 12105 Imaging unit

[0346] 12050 Integrated control unit

[0347] 12051 Microcomputer

[0348] 12052 Audio and video output unit

[0349] 12053 In-vehicle network I / F

[0350] 12061 Audio Speaker

[0351] 12062 Display unit

[0352] 12100 vehicles.

Claims

1. An object detection device, comprising: a first solid-state imaging device provided with a plurality of pixels arranged in a matrix, the first solid-state imaging device detecting occurrence of an event in each of the plurality of pixels based on an amount of light incident on each of the plurality of pixels; a flicker detection unit that generates flicker information based on the occurrence of an event detected by the first solid-state imaging device; as well as The object detection unit detects an object based on the generated flicker information.

2. The object detection device according to claim 1, wherein: The flicker detection unit specifies an event detection area in which occurrence of an event is detected among a plurality of pixels, and generates the flicker information based on the number of occurrence events detected per predetermined time for the event detection area.

3. The object detection device according to claim 2, wherein: The flicker detection unit generates the flicker information based on a maximum value or an average value of the number of occurrence events detected per predetermined time in each pixel belonging to the event detection area.

4. The object detection device according to claim 2, wherein: The flicker information includes an edge shape of the event detection area and the number of occurrence events detected per predetermined time.

5. The object detection device according to claim 2, wherein: The flicker detection unit sets, as the flicker detection area, an area in the event detection area where pixels whose number of occurrence events detected per predetermined time is a first threshold or more are arranged; The object detection unit detects an object imaged in the flicker detection area based on the flicker information.

6. The object detection device according to claim 5, wherein: The flicker detection unit sets, in the event detection area, a region in which the number of pixels whose number of occurrence events detected per predetermined time is a first threshold value or more is a second threshold value or more as the flicker detection area.

7. The object detection device according to claim 2, further comprising: A second solid-state imaging device obtains image data, wherein: The object detection unit detects an object based on the flicker information and the image data.

8. The object detection device according to claim 7, wherein: the flicker detection unit specifies an event detection area in which occurrence of an event is detected among a plurality of pixels, and generates flicker information based on the number of occurrence events detected per predetermined time for the event detection area, The object detection unit specifies an object detection area including an area on the image data corresponding to the event detection area and corresponding to an object, and detects the object based on the object detection area and the flicker information.

9. The object detection device according to claim 7, wherein: The object detection unit specifies an object detection area corresponding to an object from the image data; The flicker detection unit generates the flicker information based on the number of occurrence events detected per predetermined time for an area corresponding to the object detection area among a plurality of pixels, and The object detection unit detects an object based on the object detection area and the flicker information.

10. The object detection device according to claim 2, further comprising: A control unit controls reading of the first solid-state imaging device, wherein: The control unit reduces a resolution in monitoring occurrence of the event for a region in which pixels other than pixels included in the event detection region among the plurality of pixels are arranged.

11. The object detection device according to claim 7, further comprising: A control unit controls reading of the second solid-state imaging device, wherein: The control unit reduces a resolution of image data read from a region corresponding to the event detection region in the second solid-state imaging device.

12. An object detection system comprising: a first solid-state imaging device provided with a plurality of pixels arranged in a matrix, the first solid-state imaging device detecting occurrence of an event in each of the plurality of pixels based on an amount of light incident on each of the plurality of pixels; a flicker detection unit that generates flicker information based on the occurrence of an event detected by the first solid-state imaging device; as well as The object detection unit detects an object based on the generated flicker information.

13. A method for object detection, comprising: detecting occurrence of an event in each of the plurality of pixels according to an amount of light incident on each of the plurality of pixels, wherein the solid-state imaging device is provided with the plurality of pixels arranged in a matrix; generating flicker information based on the occurrence of the event; and An object is detected based on the generated scintillation information.

Images