Imaging System

Abstract

The imaging system includes: an event detection sensor that detects an event; and a controller that controls event detection at the event detection sensor. A color filter is set to each pixel of the event detection sensor. In addition, the controller controls event detection in a specific wavelength band based on the color filter. Due to this configuration, information of various wavelength ranges can be detected as an event.

Description

Technical Field

[0001] The present disclosure relates to an imaging system. Background Art

[0002] One of the imaging devices of the event-driven scheme is an asynchronous imaging device called DVS (Dynamic Vision Sensor). The asynchronous imaging device is capable of detecting a brightness change of a pixel that photoelectrically converts incident light exceeding a predetermined threshold as an event. Therefore, this type of asynchronous imaging device may also be referred to as an event detection sensor. Existing event detection sensors are mounted on vehicles and used as event-based vision sensors to monitor road surfaces (for example, see PTL 1).

[0003] Reference List

[0004] Patent Literature

[0005] PTL1: Japanese Unexamined Patent Application Publication No. 2013-79937 Summary of the invention

[0006] Problems to be solved by the present invention

[0007] By the way, when the vehicle is traveling, a plurality of information of various wavelength bands enters the driver's field of vision, including lighting (flashing) of brake lights or tail lights of a vehicle traveling in front of the own vehicle, flashing of turn indicators, color changes of traffic lights, electronic signs, etc. With regard to an event detection sensor configured for use on a moving body (e.g., a vehicle), it would be very convenient if the event detection sensor could detect information of various wavelength bands around the own vehicle as an event.

[0008] An object of the present disclosure is to provide an imaging system capable of detecting information of various wavelength bands as events.

[0009] Solutions to the problem

[0010] The imaging system of the present disclosure for achieving the above-mentioned object includes: an event detection sensor for detecting an event; and a controller for controlling the event detection at the event detection sensor.

[0011] The event detection sensor is provided with a color filter for each pixel, and the controller controls event detection of a specific wavelength band based on the color filter. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram showing an example of a system configuration of an imaging system according to a first embodiment of the present disclosure;

[0013] Figure 2A is a block diagram showing a configuration example of a motion identifier in an imaging system according to the first embodiment, Figure 2Bis a block diagram showing a configuration example of an object identifier in an imaging system according to the first embodiment;

[0014] Figure 3 is a block diagram showing a configuration example of an event detection sensor in an imaging system according to the first embodiment;

[0015] Figure 4 is a block diagram showing a configuration example of a pixel array section in an event detection sensor;

[0016] Figure 5 is a circuit diagram showing an example of a circuit configuration of a pixel in an event detection sensor;

[0017] Figure 6 is a block diagram showing an example of a circuit configuration of an event detector in a pixel of an event detection sensor;

[0018] Figure 7 is a circuit diagram showing a configuration example of a current-voltage conversion section in an event detector;

[0019] Figure 8 is a circuit diagram showing a configuration example of a subtractor and a quantization section in an event detector;

[0020] Fig. 9 is an exploded perspective view showing an overview of a stacked chip structure of an event detection sensor;

[0021] Fig. 10A and Fig. 10B is a color filter arrangement diagram showing various arrangement examples of color filters provided to pixels of an event detection sensor;

[0022] Fig.11 is a circuit diagram showing an example of a circuit configuration of a pixel shared by four pixels;

[0023] Fig.12 is a flowchart showing the flow of processing according to the top-level concept of Example 1;

[0024] Fig.13 is a flowchart showing a process flow for detecting tail lights according to Example 2;

[0025] Fig.14 is a flowchart showing a process flow for identifying a vehicle and sensing a tail light according to Example 3;

[0026] Fig.15 is a flowchart showing a process flow for identifying a vehicle and sensing a turn indicator according to Example 4;

[0027] Fig.16 is a flowchart showing a process flow for identifying a vehicle and sensing a turn indicator according to Example 5;

[0028] Fig.17 is a flowchart showing a process flow for recognizing a traffic light and sensing a red light according to Example 6;

[0029] Fig.18 is a flowchart showing a process flow for identifying a traffic light and sensing a change from a red light to a green light according to Example 7;

[0030] Fig.19 is a block diagram showing an example of a system configuration of an imaging system according to a second embodiment of the present disclosure;

[0031] Fig. 20 is a block diagram showing an overview of the configuration of a CMOS image sensor as an example of an image sensor in an imaging system according to a second embodiment;

[0032] Fig.21 is a circuit diagram showing an example of a circuit configuration of a pixel in an image sensor;

[0033] Fig. 22 is a plan view showing an overview of a flat chip structure of an image sensor;

[0034] Fig.23 is a plan view showing an overview of a stacked chip structure of an image sensor;

[0035] Fig.24 is a flowchart showing the flow of processing according to the top-level concept of Example 8;

[0036] Fig.25 is a flowchart showing a process flow for identifying a vehicle and sensing a tail light according to Example 9;

[0037] Fig.26 is a flowchart showing a process flow for identifying a vehicle and sensing a turn indicator according to Example 10;

[0038] Fig. 27 is a flowchart showing a process flow for identifying a traffic light and sensing a red light according to Example 11;

[0039] Fig.28 is a flowchart illustrating a process flow for identifying a traffic light and sensing a change from a red light to a green light according to Example 12;

[0040] Fig.29 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile body control system to which the technology according to the present disclosure can be applied;

[0041] Fig.30: is a diagram showing an example of an installation position of an imaging device in a mobile subject control system. DETAILED DESCRIPTION

[0042] Below, a mode for implementing the technology according to the present disclosure (hereinafter referred to as an "embodiment") will be described in detail with reference to the accompanying drawings. The technology according to the present disclosure is not limited to the embodiments. In the following description, the same elements or elements with the same functions are represented by the same reference numerals, and redundant descriptions are omitted. It should be noted that the description will be given in the following order.

[0043] 1. General Description of the Imaging System of the Present Disclosure

[0044] 2. First embodiment of the present disclosure

[0045] 2-1. Configuration Example of Imaging System According to First Embodiment

[0046] 2-2. Configuration example of event detection sensor

[0047] 2-2-1. Configuration example of pixel array section

[0048] 2-2-2. Example of pixel configuration

[0049] 2-2-3. Configuration example of event detector

[0050] 2-2-3-1. Configuration example of current-voltage conversion unit

[0051] 2-2-3-2. Configuration example of subtractor and quantization unit

[0052] 2-2-4. Example of chip configuration

[0053] 2-2-5. Necessity of color filters

[0054] 2-2-6. Configuration example of pixel circuit shared by four pixels

[0055] 2-3. Example 1 (Example of Processing of Superordinate Concept of First Embodiment)

[0056] 2-4. Example 2 (Example of sensing tail lights)

[0057] 2-5. Example 3 (Example of sensing taillights after recognizing a vehicle)

[0058] 2-6. Example 4 (Modification of Example 3: Example of detecting relative speed)

[0059] 2-7. Example 5 (Example of sensing a turn indicator after recognizing a vehicle)

[0060] 2-8. Example 6 (Example of sensing a red light after recognizing a traffic light)

[0061] 2-9. Example 7 (Modified example of Example 6: Example of sensing a change from a red light to a green light)

[0062] 3. Second embodiment of the present disclosure

[0063] 3-1. Configuration Example of Imaging System According to Second Embodiment

[0064] 3-2. Image sensor configuration example

[0065] 3-2-1. Configuration example of CMOS image sensor

[0066] 3-2-2. Example of pixel configuration

[0067] 3-2-3. Example of chip configuration

[0068] 3-2-3-1. Flat chip structure (so-called flat structure)

[0069] 3-2-3-2. Stacked chip structure (so-called stacked structure)

[0070] 3-3. Example 8 (Example of Processing of Superordinate Concept of Second Embodiment)

[0071] 3-4. Example 9 (Example of sensing taillights after recognizing a vehicle)

[0072] 3-5. Example 10 (Example of sensing a turn indicator after recognizing a vehicle)

[0073] 3-6. Example 11 (Example of sensing a red light after recognizing a traffic light)

[0074] 3-7. Example 12 (Modified example of Example 11: Example of sensing a change from a red light to a green light)

[0075] 4. Modify the example

[0076] 5. Application example of technology according to the present disclosure (example of mobile subject)

[0077] 6. Possible Configurations of the Present Disclosure

[0078] <Overall Description of Imaging System of the Present Disclosure>

[0079] In the imaging system of the present disclosure, the event detection sensor may have a configuration including an asynchronous imaging device that detects a brightness change of a pixel that photoelectrically converts incident light exceeding a predetermined threshold as an event. The imaging system of the present disclosure is preferably configured to be used on a moving subject.

[0080] In the imaging system of the present disclosure including the above-mentioned preferred configuration, the controller can be configured to perform signal processing based on the object of interest after identifying the object of interest based on event detection of the event detection sensor. In addition, the controller can be configured to send a predetermined instruction signal to the control system of the mobile subject when the taillight of the vehicle traveling in front of the vehicle is identified as the object of interest.

[0081] In addition, in the imaging system of the present disclosure including the above-mentioned preferred configuration, the controller can be configured to identify the taillights of the vehicle traveling in front of the vehicle as the object of interest after identifying the vehicle traveling in front of the vehicle. In addition, the controller can be configured to send a predetermined instruction signal to the control system of the mobile subject when the relative speed between the vehicle and the vehicle traveling in front of the vehicle is equal to or higher than a predetermined threshold after identifying the taillights as the object of interest.

[0082] In addition, in the imaging system of the present disclosure including the above-mentioned preferred configuration, the controller can be configured to send a predetermined command signal to the control system of the mobile body after identifying a vehicle traveling in front of the vehicle and when the turn indicator of the vehicle traveling in front of the vehicle is identified as an object of interest.

[0083] In addition, in the imaging system of the present disclosure including the above-mentioned preferred configuration, the controller can be configured to send a predetermined command signal to the control system of the mobile subject when a traffic light is identified as an object of interest and a red light is sensed, or when it is sensed that a traffic light in a red light state has changed from red to green.

[0084] In addition, the imaging system of the present disclosure including the above-mentioned preferred configuration may have a configuration including an image sensor that performs imaging at a predetermined frame rate. In addition, the controller may be configured to perform signal processing based on the object of interest when the object of interest is identified based on event detection of the event detection sensor after performing object recognition based on image data of the image sensor.

[0085] In addition, in the imaging system of the present invention, the controller can be configured to send a predetermined command signal to the control system of the mobile body after identifying a vehicle traveling in front of the vehicle, when the tail lights of the vehicle traveling in front of the vehicle are identified as the object of interest, or after identifying a vehicle traveling in front of the vehicle, when the turn indicator of the vehicle traveling in front of the vehicle is identified as the object of interest.

[0086] Furthermore, in the imaging system of the present disclosure including the above-mentioned preferred configuration, the controller may be configured to send a predetermined instruction signal to the control system of the mobile subject when the traffic light is identified as the object of interest and a red light is sensed. Alternatively, the controller may be configured to send a predetermined instruction signal to the control system of the mobile subject when the traffic light in the red light state is sensed to have changed from red to green.

[0087] <<First Embodiment of the Present Disclosure>>

[0088] <Configuration Example of Imaging System According to First Embodiment>

[0089] Figure 1 is a block diagram showing an example of a system configuration of an imaging system according to the first embodiment of the present disclosure.

[0090] like Figure 1 As shown, the imaging system 1A according to the first embodiment of the present disclosure has a configuration including an event detection sensor 10, a motion identifier 30, an object identifier 40, a controller 50, an operation mode definition section 60, an image recording section 70, and an interface 80. The imaging system 1A according to the first embodiment can be used on a mobile subject such as a vehicle.

[0091] For example, in the case of use on a vehicle, the imaging system 1A will be set for use at a predetermined position on the vehicle, for example, at least one of the positions on the vehicle including, for example, the front nose, the side mirrors, the rear bumper, the rear door, and the upper portion of the windshield inside the vehicle. An application example of the technology according to the present disclosure (i.e., the imaging system 1A according to the first embodiment) will be described in detail later.

[0092] As the event detection sensor 10, an asynchronous imaging device called DVS can be used, which detects a brightness change of a pixel that photoelectrically converts incident light exceeding a predetermined detection threshold as an event. The asynchronous imaging device is an imaging device that detects an event asynchronously with a vertical synchronization signal, as opposed to a synchronous imaging device that performs imaging synchronously with the vertical synchronization signal. The details of the event detection sensor 10 including the asynchronous imaging device will be described later. It should be noted that although the DVS is described here as an asynchronous imaging device, the DVS may be adapted to detect an event synchronously with the vertical synchronization signal.

[0093] The motion recognizer 30 recognizes (detects) the motion of an object based on an event signal (event data) output from the event detection sensor 10 and indicating the occurrence of an event. Figure 2A An example of a specific configuration of the motion identifier 30 is shown in FIG. The motion identifier 30 includes, for example, an event frame generator 31 and a motion detector 32.

[0094] The event frame generator 31 generates an event frame by framing a plurality of events occurring within a specific time period based on the event signal output from the event detection sensor 10. The motion detector 32 performs motion detection in the event frame generated by the framing of the event frame generator 31. It should be noted that motion detection can be performed by causing the motion identifier 30 to directly receive the event signal output in an asynchronous manner without involving the framing of the event frame generator 31.

[0095] Based on the result of the motion detection supplied from the motion identifier 30 , the object identifier 40 performs a recognition process on the object detected as the event. Figure 2B An example of a specific configuration of the object recognizer 40 is shown in . The object recognizer 40 includes, for example, a ROI extractor 41 and a recognition processing section 42 .

[0096] The ROI extractor 41 performs extraction of a specific region for performing object recognition, that is, performs extraction of ROI (region of interest: ROI). The recognition processing section 42 performs recognition processing on the object based on the data of the region extracted by the ROI extractor 41. In order to recognize the object at the recognition processing section 42, a pattern recognition technology by machine learning such as a neural network, that is, a technology for performing image recognition by comparing between feature points of an image provided as training data and feature points of a captured subject image, for example, can be adopted.

[0097] The controller 50 includes, for example, a processor (CPU), and controls the event detection sensor 10, specifically, controls the resolution of the event detection sensor 10 based on information provided from the operation mode definition section 60. Various information (for example, vehicle speed) is transmitted from a vehicle control system 12000 (see FIG. 1 ) described later via an interface 80. Fig.29 ) is provided to the controller 50, and the vehicle control system 12000 is an example of a mobile subject control system to which the technology according to the present disclosure can be applied. The details of the control of the resolution of the event detection sensor 10 will be described later.

[0098] Under the control of the controller 50, the operation mode definition unit 60 senses the driving state of a vehicle as an example of a mobile subject, such as a traffic congestion state, a driving state on a highway, etc., by using the result of motion recognition provided by the motion identifier 30 and the result of object recognition provided by the object identifier 40.

[0099] The information output from the operation mode definition unit 60 is provided to the controller 50 as information for controlling the resolution of the event detection sensor 10, and is stored in the image recording unit 70 as needed. In addition, the information output from the operation mode definition unit 60 is provided to the vehicle control system 12000 (see FIG. 1 ) via the interface 80. Fig.29 ).

[0100] In the imaging system 1A according to the first embodiment of the present disclosure having the above-described configuration, the imaging device of the present disclosure is configured by including at least the event detection sensor 10 and the controller 50. In the imaging device of the present disclosure, the controller 50 performs control according to the driving state of a vehicle as an example of a mobile subject to switch the resolution of the event detection sensor 10. In addition, an imaging system configuration in which the vehicle control system 12000 is provided with the functions of the operation mode definition section 60 and the image recording section 70 may also be adopted.

[0101] <Configuration example of event detection sensor>

[0102] The details of the event detection sensor 10 will be described below. Figure 3 is a block diagram showing an example of the configuration of the event detection sensor 10 in the imaging system 1 of the present disclosure having the above-described configuration.

[0103] like Figure 3 As shown, the event detection sensor 10 includes a pixel array section 12, which includes a plurality of pixels 11 arranged two-dimensionally in a matrix form (array form). The plurality of pixels 11 each generate an analog signal of a voltage corresponding to a photocurrent as a pixel signal, and the photocurrent is an electrical signal generated by photoelectric conversion. In addition, the plurality of pixels 11 each detects the presence or absence of an event based on whether a change exceeding a predetermined threshold value occurs in the photocurrent corresponding to the brightness of the incident light. In other words, the plurality of pixels 11 each detects a brightness change exceeding a predetermined threshold value as an event.

[0104] The event detection sensor 10 includes, in addition to the pixel array section 12 , a drive section 13 , an arbitrator section (arbitration section) 14 , a column processing section 15 , and a signal processing section 16 as peripheral circuit parts of the pixel array section 12 .

[0105] When an event is detected, each of the plurality of pixels 11 outputs a request for output of event data indicating the occurrence of the event to the arbitrator section 14. Then, in the case of receiving a response indicating agreement to output the event data from the arbitrator section 14, each of the plurality of pixels 11 outputs the event data to the drive section 13 and the signal processing section 16. In addition, the pixel 11 that has detected the event outputs an analog pixel signal generated by photoelectric conversion to the column processing section 15.

[0106] The driving section 13 drives each pixel 11 in the pixel array section 12 . For example, the driving section 13 drives the pixel 11 that has detected an event and outputs event data, and causes an analog pixel signal of the pixel 11 to be output to the column processing section 15 .

[0107] The arbitrator section 14 arbitrates requests for outputting event data supplied from the corresponding plurality of pixels 11 , and transmits a response based on the arbitration result (approval / disapproval of outputting event data) and a reset signal for resetting event detection to each pixel 11 .

[0108] The column processing section 15 includes, for example, an analog-to-digital conversion section including components of an analog-to-digital converter provided for each pixel column of the pixel array section 12. Examples of the analog-to-digital converter include a single-slope analog-to-digital converter.

[0109] In the column processing section 15, processing is performed on each pixel column of the pixel array section 12 to convert analog pixel signals output from the pixels 11 in the column into digital signals. The column processing section 15 may also perform CDS (Correlated Double Sampling) processing on the digitized pixel signals.

[0110] The signal processing section 16 performs predetermined signal processing on the digitized pixel signal supplied from the column processing section 15 and the event data output from the pixel array section 12 , and outputs the signal-processed event data and pixel signal.

[0111] As described above, the change in the photocurrent generated at the pixel 11 can be regarded as a change in the amount of light (a change in brightness) of the light entering the pixel 11. Therefore, the event can also be regarded as a change in the amount of light (a change in brightness) at the pixel 11 exceeding a predetermined threshold. The event data indicating the occurrence of the event includes at least position information, for example, coordinates, which indicate the position of the pixel 11 where the change in the amount of light as an event has occurred. In addition to the position information, the event data may include the polarity of the change in the amount of light.

[0112] Regarding the sequence of event data output from the pixel 11 at the time when the event occurs, it can be considered that the event data implicitly includes time information indicating the relative time when the event occurs, as long as the interval between the event data remains in the same state as when the event occurs. However, if the interval between the event data no longer remains in the same state as when the event occurs due to reasons such as recording the event data in a memory, the time information implicit in the event data is lost. In order to solve this problem, before the interval between the event data no longer remains in the same state as when the event occurs, the signal processing unit 16 adds time information indicating the relative time when the event occurs, such as a timestamp, to the event data.

[0113] [Configuration Example of Pixel Array Section]

[0114] Figure 4 is a block diagram showing a configuration example of the pixel array section 12 in the event detection sensor 10 .

[0115] In the pixel array section 12 in which a plurality of pixels 11 are two-dimensionally arranged in a matrix form, the plurality of pixels 11 each include a light receiving section 61 , a pixel signal generator 62 , and an event detector 63 .

[0116] In the pixel 11 having the above configuration, the light receiving section 61 performs photoelectric conversion on the incident light to generate a photocurrent. Then, according to the control of the driving section 13, the light receiving section 61 supplies a voltage signal corresponding to the photocurrent generated by performing photoelectric conversion on the incident light to the pixel signal generator 62 or the event detector 63 (see Figure 3 ).

[0117] The pixel signal generator 62 generates an analog pixel signal SIG from a voltage signal corresponding to the photocurrent supplied from the light receiving section 61. The pixel signal generator 62 then supplies the generated analog pixel signal SIG to the column processing section 15 via a vertical signal line VSL wired for each pixel column of the pixel array section 12 (see Figure 3 ).

[0118] The event detector 63 detects the occurrence of an event based on whether the amount of change in the photocurrent provided from each light receiving unit 61 exceeds a predetermined threshold. These events include, for example, an on event indicating that the amount of change in the photocurrent exceeds an upper threshold and an off event indicating that the amount of change is lower than a lower threshold. In addition, the event data indicating the occurrence of an event includes one bit indicating the detection result of the on event and one bit indicating the detection result of the off event. It should be noted that the event detector 63 can also be configured to detect only the on event.

[0119] When an event occurs, the event detector 63 reports to the arbitrator section 14 (see Figure 3 ) outputs a request for output of event data indicating the occurrence of an event. Then, upon receiving a response to the request from the arbitrator section 14, the event detector 63 outputs the event data to the drive section 13 and the signal processing section 16.

[0120] [Circuit Configuration Example of Pixel]

[0121] Figure 5 is a circuit diagram showing an example of a circuit configuration of the pixel 11 of the pixel array section 12 in the event detection sensor 10 .

[0122] As described above, the plurality of pixels 11 each have a configuration including the light receiving section 61 , the pixel signal generator 62 , and the event detector 63 .

[0123] In the pixel 11 having the above-described configuration, the light receiving section 61 has a configuration including a light receiving element (photoelectric conversion element) 611, a transfer transistor 612, and a transfer transistor 613. For example, N-type MOS (metal oxide semiconductor) transistors can be used as the transfer transistor 612 and the transfer transistor 613. The transfer transistor 612 and the transfer transistor 613 are coupled in series with each other.

[0124] The light receiving element 611 is coupled between a common connection node N1 between the transfer transistor 612 and the transfer transistor 613 and the ground, and photoelectrically converts incident light to generate charges having a charge amount corresponding to a light amount of the incident light.

[0125] The transmission signal TRG is from Figure 3 The driving section 13 shown is supplied to the gate electrode of the transfer transistor 612. The transfer transistor 612 is turned on in response to the transfer signal TRG, thereby supplying the pixel signal generator 62 with an electric signal generated by the photoelectric conversion performed at the light receiving element 611.

[0126] The control signal OFG is supplied from the driving section 13 to the gate electrode of the transfer transistor 613. The transfer transistor 613 is turned on in response to the control signal OFG, thereby supplying the event detector 63 with an electric signal generated by the photoelectric conversion performed at the light receiving element 611. The electric signal to be supplied to the event detector 63 is a photocurrent including electric charges.

[0127] The pixel signal generator 62 has a configuration including a reset transistor 621, an amplifying transistor 622, a selecting transistor 623, and a floating diffusion layer 624. As the reset transistor 621, the amplifying transistor 622, and the selecting transistor 623, for example, N-type MOS transistors can be used.

[0128] The charge generated by the photoelectric conversion at the light receiving element 611 of the light receiving section 61 is provided to the pixel signal generator 62 by the transfer transistor 612. The charge provided from the light receiving section 61 is accumulated in the floating diffusion layer 624. The floating diffusion layer 624 converts the accumulated charge to generate a voltage signal having a voltage value corresponding to the amount of charge. That is, the floating diffusion layer 624 serves as a charge voltage conversion section that converts the charge into a voltage.

[0129] The reset transistor 621 is coupled to the power supply voltage V DD The power supply line of the reset transistor 621 and the floating diffusion layer 624 are connected to each other. A reset signal RST is supplied from the driving section 13 to the gate electrode of the reset transistor 621. The reset transistor 621 is turned on in response to the reset signal RST, thereby initializing (resetting) the floating diffusion layer 624.

[0130] The amplifier transistor 622 is at the power supply voltage V DDThe select transistor 623 is coupled in series between the power supply line and the vertical signal line VSL. The amplifier transistor 622 amplifies the voltage signal generated by the charge-voltage conversion at the floating diffusion layer 624.

[0131] The selection signal SEL is supplied from the driving section 13 to the gate electrode of the selection transistor 623. The selection transistor 623 is turned on in response to the selection signal SEL, thereby outputting the voltage signal amplified by the amplifier transistor 622 as an analog pixel signal SIG to the column processing section 15 (see FIG. 14 ) via the vertical signal line VSL. Figure 3 ).

[0132] In the event detection sensor 10 including the pixel array section 12 in which the pixels 11 having the above-described configuration are two-dimensionally arranged, the drive section 13 is composed of Figure 1 The controller 50 shown instructs to start event detection. When instructed to start event detection, the drive unit 13 supplies a control signal OFG to the transfer transistor 613 of the light receiving unit 61, thereby driving the transfer transistor 613 so that a photocurrent corresponding to the charge generated at the light receiving element 611 is supplied to the event detector 63.

[0133] Then, once an event is detected at a certain pixel 11, the driving section 13 turns off the transfer transistor 613 of the pixel 11 so as to stop supplying the photocurrent to the event detector 63. Next, the driving section 13 supplies the transfer signal TRG to the transfer transistor 612, thereby driving the transfer transistor 612 to cause the charge generated by the photoelectric conversion at the light receiving element 611 to be transferred to the floating diffusion layer 624.

[0134] In this manner, the event detection sensor 10 including the pixel array section 12 in which the pixels 11 having the above-described configuration are two-dimensionally arranged outputs only the pixel signals of the pixels 11 that have detected an event to the column processing section 15. This makes it possible to reduce the power consumption of the event detection sensor 10 and the amount of image processing compared to the case where the pixel signals of all pixels are output regardless of the presence or absence of an event.

[0135] It should be noted that the configuration of the pixel 11 illustrated here is an example, and therefore, such a configuration example is non-limiting. For example, in the case where it is not necessary to output a pixel signal, a pixel configuration without the pixel signal generator 62 may be adopted. In the case of adopting such a pixel configuration, it is sufficient to omit the transfer transistor 612 from the light receiving portion 61. In addition, Figure 3 The column processing section 15 may also be configured not to have an analog-to-digital conversion function. By adopting a pixel configuration that does not output a pixel signal, an increase in the scale of the event detection sensor 10 can be suppressed.

[0136] [Configuration example of event detector]

[0137] Figure 6 is a block diagram showing a configuration example of the event detector 63 in the pixel 11 of the event detection sensor 10 .

[0138] like Figure 6 As shown, the event detector 63 according to the present example has a configuration including a current-voltage conversion section 631 , a buffer 632 , a subtractor 633 , a quantization section 634 , and a transmission section 635 .

[0139] The current-voltage conversion section 631 converts the photocurrent provided from the light receiving section 61 of the pixel 11 into a voltage signal of the logarithm of the photocurrent (in some cases, it may be referred to as "photovoltage" hereinafter), and provides the photovoltage to the buffer 632. The buffer 632 buffers the photovoltage provided from the current-voltage conversion section 631, and provides the buffered photovoltage to the subtractor 633.

[0140] The subtractor 633 calculates the difference between the photoelectric voltage at the current time and the photoelectric voltage at a time period very close to the current time, and supplies a differential signal corresponding to the difference to the quantization section 634. The quantization section 634 quantizes the differential signal supplied from the subtractor 633 into a digital signal, and supplies the digital value of the differential signal to the transmission section 635.

[0141] When the digital value of the differential signal is supplied from the quantization section 634, the transmission section 635 supplies a request for transmission of the event data to the arbitration section 14. Then, upon receiving a response to the request, i.e., a response approving the output of the event data, from the arbitrator section 14, the transmission section 635 supplies the event data to the drive section 13 and the signal processing section 16 according to the digital value of the differential signal supplied from the quantization section 634.

[0142] Next, a description will be given of a configuration example of the current-voltage conversion section 631 , the subtractor 633 , and the quantization section 634 in the event detector 63 .

[0143] (Configuration Example of Current-Voltage Converter)

[0144] Figure 7 is a circuit diagram showing a configuration example of the current-voltage conversion section 631 in the event detector 63 .

[0145] like Figure 7 As shown, the current-voltage conversion section 631 according to this example has a circuit configuration including a transistor 6311, a transistor 6312, and a transistor 6313. N-type MOS transistors can be used as the transistor 6311 and the transistor 6313. P-type MOS transistors can be used as the transistor 6312.

[0146] The transistor 6311 is coupled to the power supply voltage V DDThe transistor 6312 and the transistor 6313 are connected in series to the power supply voltage V DD between the power line and ground. Figure 6 The gate electrode of transistor 6311 and the input terminal of buffer 632 are shown to be coupled to the common connection node N2 between transistor 6312 and transistor 6313.

[0147] Predetermined bias voltage V bias is applied to the gate electrode of the transistor 6312. The transistor 6312 thereby supplies a constant current to the transistor 6313. The photocurrent is input from the light receiving section 61 to the gate electrode of the transistor 6313 via the signal input line 6314.

[0148] The drain electrode of transistor 6311 is coupled to the power supply voltage V DD The gate electrode of the transistor 6313 is coupled to the source electrode of the transistor 6311. The photocurrent from the light receiving section 61 is then converted into a photovoltage corresponding to the logarithm of the photocurrent by the transistor 6311 and the transistor 6313 having a source follower configuration.

[0149] (Configuration example of subtractor and quantization section)

[0150] Figure 8 is a circuit diagram showing a configuration example of the subtractor 633 and the quantization section 634 in the event detector 63 .

[0151] The subtractor 633 according to the present example has a configuration including a capacitance element 6331 , an operational amplifier 6332 , a capacitance element 6333 , and a switch element 6334 .

[0152] One end of the capacitor 6331 is coupled to Figure 6 The output terminal of the buffer 632 is shown, and the other end of the capacitor element 6331 is coupled to the input terminal of the operational amplifier 6332. Thus, the photovoltage provided from the buffer 632 is provided to the input terminal of the operational amplifier 6332 through the capacitor element 6331.

[0153] The capacitor element 6333 is coupled in parallel to the operational amplifier 6332. The switch element 6334 is coupled between both ends of the capacitor element 6333. As a control signal for opening and closing the switch element 6334, a reset signal is generated from Figure 3 The arbitrator section 14 is shown to be provided to a switch element 6334. The switch element 6334 opens or closes a path coupling both ends of the capacitor element 6333 in response to a reset signal.

[0154] In the subtractor 633 of the above configuration, when the switch element 6334 enters the on (closed) state, the photovoltage to be input to the terminal on the buffer 632 side of the capacitor 6331 will be expressed as V init When the photovoltage V init When the voltage is input to the buffer 632 side terminal of the capacitor 6331, the terminal on the opposite side becomes a virtual ground terminal. For convenience, it is assumed that the potential of the virtual ground terminal is zero. At this time, if the capacitance value of the capacitor 6331 is represented by C1, the charge Q accumulated in the capacitor 6331 is init It is expressed by the following equation (1).

[0155] Q init =C1×V init ...(1)

[0156] In addition, when the switch element 6334 is in the on state, both ends of the capacitor element 6333 are short-circuited, so the charge accumulated in the capacitor element 6333 is zero. Thereafter, the switch element 6334 enters the cut-off (disconnected) state. When the switch element 6334 is in the off state, the photovoltage at the buffer 632 side terminal of the capacitor element 6331 will be expressed as V after When the switch element 6334 enters the off state, the charge Q to be accumulated in the capacitor element 6331 after It is expressed by the following equation (2).

[0157] Q after =C1×V after ...(2)

[0158] If the capacitance value of the capacitor element 6333 is represented as C2, and the output voltage of the operational amplifier 6332 is represented as V out , then the charge Q2 to be accumulated in the capacitor element 6333 is expressed by the following equation (3).

[0159] Q2=-C2×V out ...(3)

[0160] Since the total charge amount of the charge amount of the capacitor 6331 and the charge amount of the capacitor 6333 does not change before and after the switch element 6334 is turned off, the following equation (4) holds.

[0161] Q init =Q after +Q2 ...(4)

[0162] Substituting equations (1) to (3) into equation (4), the following equation (5) is obtained.

[0163] V out=-(C1 / C2)×(V after -V init ) ...(5)

[0164] According to equation (5), the photovoltage V init and photovoltage V after Subtraction between, that is, calculation and photovoltage V init and photovoltage V after The difference between (V init -V after ) corresponds to the differential signal V out In addition, according to equation (5), the subtraction gain of the subtractor 633 is C1 / C2. It is generally desirable to maximize the subtraction gain of the subtractor 633. Therefore, preferably, the capacitance value C1 of the capacitor element 6331 is designed to be large, and the capacitance value C2 of the capacitor element 6333 is designed to be small.

[0165] On the other hand, if the capacitance C2 of capacitor 6333 is too small, kTC noise will increase, and the noise characteristic will be degraded. Therefore, the capacitance reduction of the capacitance C2 of capacitor 6333 is limited to the range that can tolerate noise. In addition, because the event detector 63 including subtractor 633 will be installed for each pixel 11, capacitor 6331 and capacitor 6333 are subject to area restrictions. Consider these factors to determine the capacitance C1 of capacitor 6331 and the capacitance C2 of capacitor 6333.

[0166] exist Figure 8 , the quantization section 634 has a configuration including a comparator 6341. The comparator 6341 receives the differential signal from the subtractor 633 (ie, the output signal of the operational amplifier 6332) as a non-inverting (+) input, and receives a predetermined threshold voltage V th , as the inverting (-) input. Then, the comparator 6341 compares the differential signal V from the subtractor 633 out and a predetermined threshold voltage V th The high level or low level representing the comparison result is used as the differential signal V out The quantized value is output to Figure 6 The transmission portion 635 is shown.

[0167] The differential signal V from the quantization unit 634 is out The quantized value of V can be recognized when a light amount change (brightness change) has occurred as an event, that is, when the differential signal V out Greater than (or less than) a predetermined threshold voltage V th In the case of Figure 3The signal processing unit 16 outputs, for example, high-level event data indicating the occurrence of an event. That is, the threshold voltage V th It is a threshold value for detecting an event based on a change in the amount of light (brightness change) of the pixel 11 .

[0168] The signal processing unit 16 outputs the event data provided from the transmission unit 635, and includes in the event data the position information of the pixel 11 that has detected the event indicated by the event data, the time information indicating the time when the event occurred, and the polarity information of the light amount change as the event as required.

[0169] For example, a data format called AER (Address Event Representation) can be used as a data format of event data including position information of the pixel 11 that has detected an event, time information indicating the time when the event occurred, and polarity information of a change in light amount as an event.

[0170] It should be noted that a filter that transmits predetermined light, for example, a color filter, is provided for the pixel 11, so that the pixel 11 can receive the desired light as incident light. For example, in the case where the pixel 11 receives visible light as incident light, the event data indicates that a change in the pixel value in the image on which a visually recognizable subject appears occurs. In addition, for example, in the case where the pixel 11 receives infrared rays, millimeter waves, etc. for distance measurement as incident light, the event data indicates that a change in the distance to the subject occurs. In addition, for example, in the case where the pixel 11 receives infrared rays for measuring temperature as incident light, the event data indicates that a temperature change occurs in the subject. In the present embodiment, the pixel 11 receives visible light as incident light.

[0171] [Configuration example of chip structure]

[0172] As a chip (semiconductor integrated circuit) structure of the above-mentioned event detection sensor 10, for example, a stacked chip structure can be adopted. Fig. 9 is an exploded perspective view showing an overview of the stacked chip structure of the event detection sensor 10 .

[0173] like Fig. 9 As shown, the stacked chip structure or the so-called stacked structure is a structure in which at least two chips including a light receiving chip 101 as a first chip and a detection chip 102 as a second chip are stacked on each other. Figure 5 In the circuit configuration of the pixel 11 shown, each light receiving element 611 is provided on the light receiving chip 101, and all elements except the light receiving element 611 and elements in other circuit parts of the pixel 11 are provided on the detection chip 102. The light receiving chip 101 and the detection chip 102 are electrically coupled to each other through a connection portion such as a through hole (VIA), a Cu-Cu bond, or a bump.

[0174] It should be noted that although a description of a configuration example in which the light receiving element 611 is disposed on the light receiving chip 101 and elements other than the light receiving element 611 and elements in other circuit parts of the pixel 11 are disposed on the detection chip 102 has been given as an example here, this configuration example is non-restrictive.

[0175] For example, in Figure 5 In the circuit configuration of the pixel 11 shown, a configuration may be adopted in which each element of the light receiving section 61 is provided on the light receiving chip 101, and elements of sections other than the light receiving section 61 and elements in other circuit sections of the pixel 11 are provided on the detection chip 102. In addition, a configuration may be adopted in which each element of the light receiving section 61 and the reset transistor 621 and the floating diffusion layer 624 of the pixel signal generator 62 are provided on the light receiving chip 101, and other elements are provided on the detection chip 102. In addition, a configuration may be adopted in which some elements included in the event detector 63 are provided on the light receiving chip 101 together with each element of the light receiving section 61.

[0176] [About the necessity of color filters]

[0177] Incidentally, when the vehicle is traveling, information of various wavelength bands enters the driver's field of vision, including the lighting (flashing) of the brake lights or tail lights of the vehicle traveling in front of the vehicle, the flashing of the turn indicator, the color change of the traffic light, the electronic sign, etc. Basically, the driver visually detects these various information and determines the content thereof. However, it would be very convenient if the event detection sensor 10 could perform detection and determination in the same manner as the driver.

[0178] Therefore, in the imaging system 1A according to the present embodiment, a color filter as an example of a wavelength selection element is provided for each pixel 11 in the event detection sensor 10 so that an event of each color can be detected by performing threshold detection at each pixel 11. Then, Figure 1 The motion identifier 30 shown performs motion detection for each color on an object detected as an event. This makes it possible to utilize the event signal of each color in the corresponding wavelength band when detecting (sensing) the lighting (flashing) of a vehicle's brake lights or tail lights, the flashing of a turn indicator, the color change of a traffic light, an electronic sign, etc.

[0179] Here, a case where a pixel group of two pixels in the horizontal direction (row direction) × two pixels in the vertical direction (column direction), that is, a so-called 2×2 pixel, is used as a unit for arranging color filters is described as an example. However, 2×2 pixels is an example, and therefore, the pixel group as the unit for arranging color filters is not limited to 2×2 pixels.

[0180] When 2×2 pixels are used as a unit, examples of the arrangement of color filters include an RCCC filter combining R (red) pixels and C (clear) pixels, an RCCB filter combining B (blue) pixels with R pixels and C pixels, and an RGB Bayer arrangement filter combining R pixels, G (green) pixels, and B pixels, such as Fig. 10A It should be noted that a C (transparent) pixel is a pixel with no color filter or a transparent color filter, similar to a W (white) pixel.

[0181] Among the three types of color filter arrangements described above, the RCCC filter that combines R (red) pixels and C (transparent) pixels is particularly capable of achieving high sensitivity that allows imaging of distant obstacles or people, etc., even at low brightness equivalent to a moonlit night. In addition, the RCCC filter is capable of improving the detection accuracy of light in the red wavelength band that is important, such as tail lights, red traffic lights, etc., for example, in vehicle-mounted sensing.

[0182] Apart from Fig. 10A In addition to the three examples shown, examples of the arrangement of the color filters include a RCGB color filter combining R pixels, C (transparent) pixels, G pixels, and B pixels; a RIR (infrared) GB color filter combining R pixels, IR (infrared) pixels, G pixels, and B pixels; and a GMgCyYe color filter combining G pixels, Mg (magenta) pixels, Cy (cyan) pixels, and Ye (yellow) pixels. However, the arrangement of the color filters is not limited to Fig. 10A and Fig. 10B The arrangement illustrated in FIG.

[0183] [Circuit Configuration Example of Pixel Shared by Four Pixels]

[0184] Here, as an example, a description will be given of the circuit configuration of the pixel 11 with reference to a case where color filters are arranged in an RGB Bayer arrangement. Basically, as Figure 5 As shown, the pixel signal generator 62 and the event detector 63 are provided in the light receiving section 61 for each pixel 11. However, a circuit configuration may be adopted in which the pixel signal generator 62 and the event detector 63 are shared between, for example, four light receiving sections 61 in units of four pixels of 2×2 pixels, similar to the configuration example of the circuit described below.

[0185] Fig.11 is a circuit diagram showing a circuit configuration example of a four-pixel-sharing pixel. Fig.11 FIG. 2 shows a circuit configuration example of a pixel block PB in which four pixels of 2×2 pixels are used as a unit. Fig.11As shown, the pixel block PB has a configuration including, for example, a light receiving section 61 , a pixel signal generator 62 , an event detector 63 , and a logic circuit 64 .

[0186] It should be noted that Fig.11 In the example, the pixel signal generator 62 and the event detector 63 correspond to Figure 5 The logic circuit 64 includes the pixel signal generator 62 and the event detector 63 in FIG. 1 and is similar thereto in basic circuit configuration. Therefore, its detailed description will be omitted. Figure 3 The driving unit 13, the signal processing unit 16 and the arbitrator unit 14 are included in the embodiment.

[0187] In the case where the color filters are arranged in an RGB Bayer arrangement, the light receiving section 61 for four-pixel sharing includes a light receiving element 611R having a red (R) color filter, a light receiving element 611Gr having a green (Gr) color filter, a light receiving element 611Gb having a green (GB) color filter, and a light receiving element 611B having a blue (B) color filter. The light receiving section 61 also includes four transfer transistors 612R, 612Gr, 612Gb, and 612B provided for the four light receiving elements 611R, 611Gr, 611Gb, and 611B, respectively, and a transfer transistor 613 provided in common for the four light receiving elements 611R, 611Gr, 611Gb, and 611B.

[0188] Transmission signal TRG R ,TRG Gr ,TRG Gb and TRG B The control signal OFG is supplied from the driving section 13 of the logic circuit 64 to the respective gate electrodes of the four transfer transistors 612R, 612Gr, 612Gb and 612B. The control signal OFG is supplied from the driving section 13 to the gate electrode of the transfer transistor 613. The signal output from the light receiving elements 611R, 611Gr, 611Gb and 611B via the transfer transistors 612R, 612Gr, 612Gb and 612B is output at the node N. 61 Node N 61 It is coupled to the output terminal of the pixel signal generator 62 and the input terminal of the event detector 63 via the transfer transistor 613 .

[0189] In the circuit configuration example of a pixel shared by four pixels having the above configuration, for example, in the case of detecting a brightness change (detecting an event) of a light receiving element 611R provided with a red (R) filter, it is sufficient to turn on only the transfer transistor 612R. In addition, for example, by turning on all the transfer transistors 612R, 612Gr, 612Gb, and 612B in sequence after detecting a red (R) event, the brightness value of the detected event can be detected with higher accuracy. Therefore, the event detection sensor 10 can perform not only event detection but also image acquisition.

[0190] According to the configuration example of the above circuit, for example, in units of four pixels of 2×2 pixels, the pixel signal generator 62 and the event detector 63 are shared among four light receiving sections 61. Therefore, the circuit scale of the pixel 11 can be reduced compared to the case where pixel sharing is not adopted.

[0191] Hereinafter, a description is given of a specific example in which, in the imaging system 1A according to the first embodiment, an event is detected in a specific wavelength band from information of the pixel 11 provided with a color filter, and predetermined signal processing is performed based on the detection result. It should be noted that the processing in each example described below is basically Figure 1 The imaging system 1A shown is executed under the control of the controller 50 .

[0192] <Example 1>

[0193] Example 1 is an example of processing of the top-level concept of the first embodiment. Fig.12 The processing flow of the top-level concept according to Example 1 is shown in the flowchart of .

[0194] During the travel of the vehicle equipped with the imaging system 1A according to the first embodiment, the controller 50 performs event detection in a specific wavelength band based on the signal of the pixel 11 provided with the filter of the specific wavelength band (step S11), and then Figure 1 At the motion identifier 30 shown, motion detection is performed on the object detected as an event (step S12).

[0195] Next, the controller 50 Figure 2B The ROI extractor 41 shown in FIG. 1 performs extraction of a specific region ROI to perform object recognition at the position where the motion is detected (step S13), and then Figure 2B The identification processing section 42 shown performs object identification in the extracted ROI portion (step S14 ).

[0196] Next, the controller 50 determines whether the object is identified as an object of interest through object recognition (step S15). In the case where the object is not identified as an object of interest (No in S15), the controller 50 returns to step S11. In the case where the object is identified as an object of interest (Yes in S15), the controller 50 performs predetermined signal processing based on the object of interest (step S16), and ends a series of processing according to the top concept of Example 1.

[0197] As described above, in Example 1, event detection is performed in a specific wavelength band, and motion detection is performed on an object detected as an event. When an object is identified as an object of concern, predetermined signal processing based on the object is performed. Therefore, information in various wavelength bands can be detected as events. Specifically, when the vehicle is traveling, information in various wavelength bands enters the driver's field of vision, and the information in various wavelength bands includes the lighting (flashing) of the brake lights or tail lights of the vehicle traveling in front of the vehicle, the flashing of the turn indicator, the color change of the traffic light, the electronic sign, etc., the imaging system A can detect and determine it in the same way as the driver.

[0198] Here, for example, "predetermined signal processing" refers to Fig.29 The drive system control unit 12010 of the vehicle control system 12000 shown processes command signals for various automatic controls, etc. When receiving the command signal from the event detection sensor 10, the vehicle control system 12000 automatically performs various controls, for example, applying brakes by automatic control or attracting the driver's attention by displaying an alarm.

[0199] <Example 2>

[0200] Example 2 is an example of sensing the taillights of a vehicle traveling in front of the vehicle. Fig.13 The processing flow of taillight sensing according to Example 2 is shown in the flowchart of . In Example 2, for example, Fig. 10A The RCCC filter shown is used as a color filter provided for each pixel 11 in the event detection sensor 10. In the case of the RCCC filter, red (R) is used as a specific wavelength band. The same applies to the examples to be described later.

[0201] During the travel of the vehicle equipped with the imaging system 1A according to the first embodiment, the controller 50 performs event detection in the red wavelength band (step S21), and then performs motion detection of the object detected as the event at the motion identifier 30 (step S22). Next, the controller 50 performs extraction of the ROI at the ROI extractor 41 for the position where the motion is detected (step S23), and then performs object recognition in the extracted portion of the ROI at the recognition processing section 42 (step S24).

[0202] Next, the controller 50 determines whether the object is recognized as the taillight of the vehicle traveling in front of the vehicle through object recognition (step S25). If the object is not recognized as the taillight (No in S25), the controller 50 returns to step S21. If the object is recognized as the taillight (Yes in S25), the controller 50 returns to step S21. Figure 1 The interface 80 shown in FIG. 1 sends command signals for brake control, alarm display, etc. to the Fig.29 The driving system control unit 12010 of the vehicle control system 12000 shown in FIG. 12 (step S26 ). Then, a series of processes for tail light sensing according to Example 2 ends.

[0203] When receiving a command signal for brake control, warning display, etc. from the imaging system 1A according to the first embodiment, the drive system control unit 12010 of the vehicle control system 12000 performs control, for example, applying brakes by automatic control or attracting the driver's attention by displaying an alarm.

[0204] As described above, in Example 2, event detection is performed in the red wavelength band, and motion detection is performed on the object detected as an event. In the case where the object is identified as the taillight of a vehicle traveling in front of the vehicle, it is determined that the distance between the vehicle and the other vehicle traveling in front of the vehicle is close enough to be able to identify the taillights of the other vehicle. Then, command signals for brake control, alarm display, etc. are sent to the drive system control unit 12010 of the vehicle control system 12000. The vehicle control system 12000 is therefore able to perform control, for example, by applying brakes through automatic control or by displaying an alarm to attract the driver's attention, thereby being able to contribute to safe driving.

[0205] It should be noted that in Example 2, when the recognition process is performed on the tail lamp, the object is recognized as the tail lamp through object recognition; however, the blinking of the tail lamp may be detected and used as information for object recognition (object sensing).

[0206] <Example 3>

[0207] Example 3 is an example of sensing taillights after recognizing a vehicle traveling in front of the host vehicle. Fig.14 The processing flow of vehicle recognition and taillight sensing according to Example 3 is shown in the flowchart of .

[0208] During the travel of the vehicle equipped with the imaging system 1A according to the first embodiment, the controller 50 performs event detection in all wavelength bands (step S31), and then performs motion detection of the object detected as the event at the motion identifier 30 (step S32). Next, the controller 50 performs extraction of the ROI at the position where the motion is detected at the ROI extractor 41 (step S33), and then performs object recognition in the portion of the recognition processing section 42 that is in the extracted ROI (step S34).

[0209] Next, the controller 50 determines whether the object is identified as a vehicle traveling ahead through object recognition (step S35). If the object is not identified as a vehicle (No in S35), the controller 50 returns to step S31. If the object is identified as a vehicle (Yes in S35), the controller 50 extracts the red event signal area in the area where the object is identified as a vehicle (step S36).

[0210] Next, the controller 50 determines whether the object is recognized as a tail light through object recognition at the recognition processing unit 42 (step S37). In the case where the object is not recognized as a tail light (No in S37), the controller 50 returns to step S36. In the case where the object is recognized as a tail light (Yes in S37), the controller 50 sends a signal to the interface 80. Fig.29 The driving system control unit 12010 of the illustrated vehicle control system 12000 sends a command signal for brake control, warning display, etc. (step S38). Then, a series of processes for vehicle recognition and taillight sensing according to Example 3 ends.

[0211] When receiving a command signal for brake control, warning display, etc. from the imaging system 1A according to the first embodiment, the drive system control unit 12010 of the vehicle control system 12000 performs control, for example, applying brakes by automatic control or attracting the driver's attention by displaying an alarm.

[0212] As described above, in Example 3, event detection is performed in all wavelength bands, and motion detection is performed on the object detected as an event. After the vehicle traveling in front of the own vehicle is identified, the recognition processing related to the taillight is performed. Compared with the case of Example 2 in which the taillight of the vehicle traveling in front of the own vehicle is directly identified, this makes it possible to improve the recognition accuracy of the taillight.

[0213] It should be noted that in Example 3, when the recognition process is performed on the tail lamp, the object is recognized as the tail lamp through object recognition; however, the blinking of the tail lamp may be detected and used as information for object recognition (object sensing).

[0214] <Example 4>

[0215] Example 4 is a modified example of Example 3, and is an example of detecting the relative speed of the own vehicle and another vehicle (a vehicle traveling ahead of the own vehicle) when recognizing the taillights. Fig.15 The processing flow of vehicle identification and taillight sensing according to Example 4 is shown in the flowchart of .

[0216] Example 4 includes Fig.14 The processing in the flowchart of Example 3 shown, wherein step S41 and step S42 are interposed between step S37 and step S38. When the object is identified as a tail light in step S37 (Yes in S37), the controller 50 detects / calculates the relative speed of the host vehicle and the other vehicle (the vehicle traveling in front of the host vehicle) (step S41), and then determines whether the relative speed is equal to or higher than a predetermined threshold value (relative speed) (step S42).

[0217] If the relative speed is less than the predetermined threshold value (No in S42), the controller 50 returns to step S41. If the relative speed is equal to or higher than the predetermined threshold value (Yes in S42), the controller 50 sends the signal to the interface 80. Fig.29 The driving system control unit 12010 of the illustrated vehicle control system 12000 sends a command signal for brake control, warning display, etc. (step S38). Then, a series of processes for vehicle recognition and taillight sensing according to Example 4 ends.

[0218] As described above, in Example 4, when the taillights of the vehicle (other vehicle) traveling ahead of the vehicle are recognized, the relative speed between the vehicle and the other vehicle is detected / calculated. When the relative speed is equal to or higher than a predetermined threshold, it is determined that the vehicle is likely to quickly approach the vehicle traveling ahead. This is more conducive to safe driving than in Example 3 where the relative speed is not detected.

[0219] <Example 5>

[0220] Example 5 is an example of sensing a turn indicator (flash light) after recognizing a vehicle traveling in front of the vehicle. Fig.16 The processing flow for vehicle identification and turn indicator sensing according to Example 5 is shown in the flowchart of .

[0221] During the travel of the vehicle equipped with the imaging system 1A according to the first embodiment, the controller 50 performs event detection in all wavelength bands (step S51), and then performs motion detection of an object detected as an event at the motion identifier 30 (step S52). Next, the controller 50 performs extraction of an ROI at a position where motion is detected at the ROI extractor 41 (step S53), and then performs object recognition in a portion of the extracted ROI at the recognition processing section 42 (step S54).

[0222] Next, the controller 50 determines whether the object is identified as a vehicle traveling ahead through object recognition (step S55). If the object is not identified as a vehicle (No in S55), the controller 50 returns to step S51. If the object is identified as a vehicle (Yes in S55), the controller 50 extracts a yellow or red event signal area in the area where the object is identified as a vehicle (step S56).

[0223] Next, the controller 50 determines whether the object is recognized as a turn indicator by object recognition at the recognition processing unit 42 (step S57). In the case where the object is not recognized as a turn indicator (No in S57), the controller 50 returns to step S56. In the case where the object is recognized as a turn indicator (Yes in S57), the controller 50 sends a command signal for executing control of the vehicle based on the indication content of the turn indicator to the control unit 42 via the interface 80. Fig.29 The driving system control unit 12010 of the vehicle control system 12000 shown (step S58). Then, a series of processes for vehicle recognition and turn indicator sensing according to Example 5 ends.

[0224] As described above, in Example 5, event detection is performed in all wavelength bands, and motion detection is performed on objects detected as events. After identifying a vehicle traveling in front of the vehicle, recognition processing is performed on the turn indicator in the area where the vehicle has been identified. Then, a command signal for executing control of the vehicle based on the indication content of the turn indicator (right turn, left turn or danger) is sent to the drive system control unit 12010 of the vehicle control system 12000.

[0225] Here, for example, when the vehicle in front provides a direction indication of a right turn, it can be inferred that the vehicle traveling in front of the vehicle will no longer appear in front due to the right turn. Therefore, the drive system control unit 12010 that controls the vehicle control system 12000 increases the vehicle speed, which can be understood as "controlling the vehicle based on the indication content of the turn indicator." Alternatively, when the direction indicator light of the vehicle in front indicates danger (flashing display), the vehicle traveling in front is likely to stop. Therefore, for example, it can be understood that the drive system control unit 12010 that controls the vehicle control system 12000 automatically brakes.

[0226] It should be noted that in Example 5, when the recognition process is performed on the turn indicator, the object is recognized as the turn indicator through object recognition; however, the blinking of the turn indicator may be detected and used as information for object recognition (object sensing).

[0227] <Example 6>

[0228] Example 6 is an example of identifying a traffic light and sensing a red light. Fig.17The processing flow for traffic light recognition and red light sensing according to Example 6 is shown in the flowchart of .

[0229] During the travel of the vehicle equipped with the imaging system 1A according to the first embodiment, the controller 50 performs event detection in all wavelength bands (step S61), and then performs motion detection of an object detected as an event at the motion identifier 30 (step S62). Next, the controller 50 performs extraction of an ROI at a position where motion is detected at the ROI extractor 41 (step S63), and then performs object recognition in a portion of the extracted ROI at the recognition processing section 42 (step S64).

[0230] Next, the controller 50 determines whether the object is recognized as a traffic light (step S65). In the case where the object is not recognized as a traffic light (No in S65), the controller 50 returns to step S61. In the case where the object is recognized as a traffic light (Yes in S65), the controller 50 extracts the red event signal area in the area where the object is recognized as a traffic light (step S66).

[0231] Next, the controller 50 determines whether the indication content of the traffic light is recognized as a red light through object recognition at the recognition processing unit 42 (step S67). In the case where the indication content is not recognized as a red light (No in S67), the controller 50 returns to step S66. In the case where the indication content is recognized as a red light (Yes in S67), the controller 50 sends a signal to the interface 80 via the interface 80. Fig.29 The driving system control unit 12010 of the illustrated vehicle control system 12000 sends a command signal for brake control, warning display, etc. (step S68). Then, a series of processes for traffic light recognition and red light sensing according to Example 6 ends.

[0232] As described above, in Example 6, event detection is performed in all wavelength bands, and motion detection is performed on an object detected as an event. In the case where the object is a traffic light and its indication content is a red light, a command signal for brake control, alarm display, etc. is sent to the drive system control unit 12010 of the vehicle control system 12000. The vehicle control system 12000 can therefore perform control, for example, to apply brakes by automatic control or to attract the driver's attention by displaying an alarm, thereby being able to contribute to safe driving.

[0233] It should be noted that in Example 6, when performing identification processing on a traffic light, the object is identified as a traffic light through object recognition; however, flickering information (e.g., frequency, duty cycle, etc.) can be detected and used as information for object identification (object sensing).

[0234] <Example 7>

[0235] Example 7 is a modified example of Example 6, and is an example of sensing a traffic light changing from red to green. Fig.18 The processing flow for traffic signal recognition and sensing the change from a red light to a green light according to Example 7 is shown in the flowchart of .

[0236] Example 7 includes Fig.17 The processing in the flowchart of Example 6 shown, in which steps S71 to S73 are introduced as processing after step S78. After recognizing as a red light and sending a command signal for brake control, alarm display, etc. (step S68), the controller 50 performs event detection in the green light wavelength band in the area where the object is recognized as a traffic light (step S71), and then determines whether the traffic light is recognized as a green light (step S72).

[0237] In the case where the traffic light is not recognized as a green light (No in S72), the controller 50 returns to step S61. In the case where the traffic light is recognized as a green light (Yes in S72), the controller 50 sends a command signal for instructing to start the engine, start, etc. to the drive system control unit 12010 of the vehicle control system 12000 via the interface 80 (step S73). Then, a series of processes for traffic signal recognition and sensing a change from a red light to a green light according to Example 7 are ended.

[0238] As described above, in Example 7, when the traffic light in the red light state has changed from red to green, a command signal for instructing to start the engine, start, etc. is sent to the drive system control unit 12010 of the vehicle control system 12000. The vehicle control system 12000 is thereby able to perform control to automatically start the engine and start the vehicle, thereby making it possible to prevent the occurrence of problems such as delayed start due to overlooking the traffic light.

[0239] It should be noted that when a traffic light is identified as changing from red to green, the threshold for blue or green may be set to a low value in areas where the object is identified as a traffic light and set to a high value in other areas, thereby allowing only green lights to be detected.

[0240] <<Second Embodiment of the Present Disclosure>>

[0241] <Configuration Example of Imaging System According to Second Embodiment>

[0242] Fig.19 is a block diagram showing an example of a system configuration of an imaging system according to a second embodiment of the present disclosure.

[0243] like Fig.19As shown, the imaging system 1B according to the second embodiment of the present disclosure has a configuration including an event detection sensor 10 , an image sensor 20 , a motion identifier 30 , an object identifier 40 , a controller 50 , an operation mode definition section 60 , and an image recording section 70 .

[0244] The functions of the event detection sensor 10, the motion identifier 30, the object identifier 40, the controller 50, the operation mode definition section 60, and the image recording section 70 are as described in the imaging system 1A according to the first embodiment. It should be noted that the event detection sensor 10 includes, for example, Fig. 10A or Fig. 10B The color filter shown is as in the imaging system 1A according to the first embodiment. Furthermore, similarly to the imaging system 1A according to the first embodiment, the imaging system 1B according to the second embodiment can be used on a mobile subject, for example, a vehicle.

[0245] <Configuration example of image sensor>

[0246] The basic configuration of the image sensor 20 in the imaging system 1B according to the second embodiment will be described. Here, as the image sensor 20, a CMOS image sensor which is a type of image sensor of the XY addressing scheme will be described as an example. The CMOS image sensor is an image sensor manufactured by applying a CMOS process or partially using a CMOS process. However, the image sensor 20 is not limited to a CMOS image sensor.

[0247] [Configuration example of CMOS image sensor]

[0248] Fig. 20 1 is a block diagram showing an overview of the configuration of a CMOS image sensor as an example of the image sensor 20 in the imaging system 1B according to the second embodiment.

[0249] The image sensor 20 according to the present example has a configuration including a pixel array section 22 in which pixels 21 including a light receiving section (photoelectric conversion section) are arranged two-dimensionally in a row direction and a column direction (in the form of a matrix) and a peripheral circuit section for the pixel array section 22. Here, the row direction refers to the arrangement direction of the pixels 21 in a pixel row, and the column direction refers to the arrangement direction of the pixels 21 in a pixel column. The pixels 21 perform photoelectric conversion to generate and accumulate photocharges corresponding to the amount of received light.

[0250] The image sensor 20 according to the present example is, for example, an RGB sensor in which the pixels 21 of the pixel array section 22 include filters of respective colors of R (red), G (green), and B (blue). However, the image sensor 20 is not limited to an RGB sensor.

[0251] The peripheral circuit portion of the pixel array section 22 includes, for example, a row selector 23, a constant current source section 24, an analog-to-digital conversion section 25, a horizontal transfer scanner 26, a signal processing section 27, a timing controller 28, and the like.

[0252] In the pixel array section 22, the pixel drive lines 311 to 31 m (In some cases, it may be collectively referred to as "pixel drive line 31" below) is wired along the row direction of each pixel row in the matrix form pixel arrangement. In addition, for each pixel column, vertical signal lines 321 to 32 n The pixel drive lines 31 (hereinafter collectively referred to as "vertical signal lines 32" in some cases) are wired along the column direction. When a signal is read out from the pixel 21, the pixel drive line 31 transmits a drive signal for performing driving. Figure 1 Although shown as a single wiring in FIG. 1 , the number of the pixel driving lines 31 is not limited to one. Respective one ends of the pixel driving lines 31 are coupled to the output ends of the row selectors 23 corresponding to the corresponding rows.

[0253] Each circuit portion of the peripheral circuit portion of the pixel array section 22, ie, the row selector 23, the constant current source section 24, the analog-to-digital conversion section 25, the horizontal transfer scanner 26, the signal processing section 27, and the timing controller 28 will be described below.

[0254] The row selector 23 includes a shift register, an address decoder, etc., and controls scanning of a pixel row and an address of a pixel row when selecting each pixel 21 of the pixel array section 22. Although illustration of a specific configuration of the row selector 23 is omitted, the row selector 23 generally has a configuration of two scanning systems including a readout scanning system and a full scanning system.

[0255] The readout scanning system selectively scans the pixels 21 of the pixel array section 22 row by row in sequence to read out pixel signals from the pixels 21. The pixel signals read out from the pixels 21 are analog signals. The sweep scanning system performs scanning on the readout row for which the readout scanning system performs the readout scanning earlier than the readout scanning by a time corresponding to the shutter speed.

[0256] The sweep scanning of the sweep scanning system sweeps out unnecessary charges from the light receiving portion (photoelectric conversion portion) of the pixel 21 in the readout row, thereby resetting the light receiving portion. Therefore, the sweeping (resetting) of unnecessary charges by the sweep scanning system enables the so-called electronic shutter operation to be performed. The electronic shutter operation here refers to an operation for sweeping out the photocharge of the light receiving portion and starting a new exposure (starting the accumulation of photocharge).

[0257] The constant current source section 24 includes a plurality of current sources I (see Fig.18 ), including, for example, corresponding vertical signal lines 321 to 322 coupled to each pixel columnn MOS transistors, and through the vertical signal lines 321 to 32 n Each of supplies a bias current to each pixel 21 of a pixel row selectively scanned by the row selector 23.

[0258] The analog-to-digital conversion section 25 includes a plurality of analog-to-digital converter components provided corresponding to the pixel columns of the pixel array section 22 (for example, provided for each pixel column). The analog-to-digital conversion section 25 is a column-parallel analog-to-digital conversion section that converts the analog-to-digital signals of the pixels of the pixel array section 22 to the corresponding vertical signal lines 321 to 32 n The analog pixel signal output on a per-pixel-column basis is converted into a digital signal.

[0259] As the analog-to-digital converter in the column-parallel analog-to-digital conversion section 25, for example, a single-slope analog-to-digital converter as an example of an analog-to-digital converter of a reference signal comparison type is available. However, the analog-to-digital converter is not limited to the single-slope analog-to-digital converter, and a successive approximation analog-to-digital converter, a ΔΣ modulation analog-to-digital converter, etc. are available.

[0260] The example of the analog-to-digital converter in the column-parallel analog-to-digital conversion section 25 is also applicable to the column processing section 15 included in the aforementioned event detection sensor 10 (see Figure 3 ) in the analog-to-digital converter in the analog-to-digital conversion unit.

[0261] The horizontal transfer scanner 26 includes a shift register, an address decoder, and the like, and controls scanning of pixel columns and addresses of pixel columns when reading out signals from each pixel 21 of the pixel array section 22. Under the control of the horizontal transfer scanner 26, pixel signals converted into digital signals by the analog-to-digital conversion section 25 are read out to a horizontal transfer line (horizontal output line) 29 on a per-pixel-column basis.

[0262] The signal processing section 27 performs predetermined signal processing on the digital pixel signal provided through the horizontal transmission line 29, and generates two-dimensional image data. For example, the signal processing section 27 performs correction of line defects and point defects or signal clamping, and performs digital signal processing including parallel-to-serial conversion, compression, encoding, addition, averaging, and intermittent sampling operations. The signal processing section 27 outputs the generated image data as an output signal of the image sensor 20 to a downstream device.

[0263] The timing controller 28 generates various timing signals, clock signals, control signals, etc. based on the vertical synchronization signal VD and the horizontal synchronization signal HD supplied from the outside and the master clock MCK (not shown), etc. Based on these generated signals, the timing controller 28 then performs drive control on the row selector 23, the constant current source section 24, the analog-to-digital conversion section 25, the horizontal transfer scanner 26, the signal processing section 27, etc.

[0264] Imaging is performed at the image sensor 20 in synchronization with a synchronization signal such as a vertical synchronization signal VD under the control of the timing controller 28. That is, the image sensor 20 is a synchronous imaging device that performs imaging at a predetermined frame rate.

[0265] [Circuit Configuration Example of Pixel]

[0266] Fig.21 is a circuit diagram showing an example of a circuit configuration of the pixel 21 of the pixel array section 22 in the image sensor 20 .

[0267] The pixel 21 includes, for example, a photodiode 211 as a light receiving section (photoelectric conversion section). The pixel 21 has a pixel configuration including a transfer transistor 212, a reset transistor 213, an amplification transistor 214, and a selection transistor 215 in addition to the photodiode 211.

[0268] It should be noted that although, for example, N-type MOS transistors are used here as four transistors including a transfer transistor 212, a reset transistor 213, an amplification transistor 214, and a selection transistor 215, the combination of conductivity types of the four transistors 212 to 215 illustrated here is merely an example, and thus this combination is non-limiting.

[0269] In the pixel 21, a plurality of pixel drive lines used as the above-mentioned pixel drive line 31 are wired to be shared by the respective pixels 21 in the same pixel row. These plurality of pixel drive lines are coupled to the output terminals of the row selectors 23 corresponding to the respective pixel rows on a per-pixel row basis. The row selector 23 appropriately outputs the transfer signal TRG, the reset signal RST, and the selection signal SEL to the plurality of pixel drive lines.

[0270] The photodiode 211 has an anode electrode coupled to a low potential side power supply (e.g., ground). The photodiode 211 photoelectrically converts the received light into a photocharge (in this example, a photoelectron) having a charge amount corresponding to the light amount, and accumulates the photocharge. The photodiode 211 has a cathode electrode electrically coupled to the gate electrode of the amplifier transistor 214 via the transfer transistor 212. Here, the region to which the gate electrode of the amplifier transistor 214 is electrically coupled is a floating diffusion (floating diffusion region or impurity diffusion region) FD. The floating diffusion FD is a charge-voltage conversion section that converts charge into voltage.

[0271] At a high level (for example, V DD A transfer signal TRG of an active state (level) is supplied from the row selector 23 to the gate electrode of the transfer transistor 212. The transfer transistor 212 is turned on in response to the transfer signal TRG, thereby transmitting the photocharge generated by the photoelectric conversion at the photodiode 211 and accumulated in the photodiode 211 to the floating diffusion FD.

[0272] The reset transistor 213 is coupled to the power supply voltage V DD A reset signal RST in a high-level active state is supplied from the row selector 23 to the gate electrode of the reset transistor 213. The reset transistor 213 is turned on in response to the reset signal RST, and the charge of the floating diffusion FD is swept out to the voltage V DD node to reset the floating diffusion FD.

[0273] The amplifying transistor 214 has a gate electrode coupled to the floating diffusion FD and a gate electrode coupled to the power supply voltage V DD The amplifier transistor 214 is used as an input portion of a source follower that reads out a signal obtained by photoelectric conversion at the photodiode 211. The amplifier transistor 214 has a source coupled to the vertical signal line 32 via the selection transistor 215. In addition, the amplifier transistor 214 and the current source I coupled to one end of the vertical signal line 32 constitute a source follower that converts the voltage of the floating diffusion FD into the potential of the vertical signal line 32.

[0274] The selection transistor 215 has a drain coupled to the source of the amplification transistor 214 and a source coupled to the vertical signal line 32. A selection signal SEL in a high-level active state is supplied from the row selector 23 to a gate electrode of the selection transistor 215. The selection transistor 215 is turned on in response to the selection signal SEL, thereby bringing the pixel 21 into a selection state to allow the signal output from the amplification transistor 214 to be transmitted to the vertical signal line 32.

[0275] It should be noted that although a 4-Tr configuration including a transfer transistor 212, a reset transistor 213, an amplifying transistor 214, and a selecting transistor 215 (i.e., including four transistors (Trs)) is described here as an example of a pixel circuit of the pixel 21, this is not restrictive. For example, the selecting transistor 215 may be omitted to adopt a 3-Tr configuration in which the amplifying transistor 214 has the function of the selecting transistor 215, or a 5-Tr configuration including more transistors may be adopted as needed.

[0276] [Configuration example of chip structure]

[0277] Examples of the chip (semiconductor integrated circuit) structure of the image sensor 20 having the above-described configuration include a flat chip structure and a stacked chip structure. The image sensor 20 of either the flat chip structure and the stacked chip structure may adopt a front-side illumination pixel structure that captures illumination light from the front side, or a back-side illumination pixel structure that captures illumination light from the back side opposite to the front side, wherein the substrate surface on the side on which the wiring layer is provided serves as the front side (front) of the pixel 21. The flat chip structure and the stacked chip structure will be described below.

[0278] (Flat chip structure)

[0279] Fig. 22 2 is a plan view showing an overview of a flat chip structure of the image sensor 20 .

[0280] like Fig. 22 As shown, the flat chip structure (so-called flat structure) is a structure in which the peripheral circuit part of the pixel array section 22 is formed on the semiconductor substrate 201, which is the same as the semiconductor substrate of the pixel array section 22 including the pixels 21 arranged in a matrix form. Specifically, the row selector 23, the constant current source section 24, the analog-to-digital conversion section 25, the horizontal transfer scanner 26, the signal processing section 27, the timing controller 28, etc. are formed on the same semiconductor substrate 201 as the pixel array section 22.

[0281] (Stacked chip structure)

[0282] Fig.23 is an exploded perspective view showing an overview of the stacked chip structure of the image sensor 20 .

[0283] like Fig.23 As shown, the stacked chip structure (so-called stacked structure) is a structure in which at least two semiconductor substrates including a first semiconductor substrate 202 and a second semiconductor substrate 203 are stacked on each other. In the stacked structure, the pixel array section 22 is formed on the first semiconductor substrate 202 serving as the first layer. In addition, a circuit portion including a row selector 23, a constant current source section 24, an analog-to-digital conversion section 25, a horizontal transfer scanner 26, a signal processing section 27, a timing controller 28, etc. is formed on the second semiconductor substrate 203 serving as the second layer. In addition, the first semiconductor substrate 202 as the first layer and the second semiconductor substrate 203 as the second layer are electrically coupled to each other through connection sections 33A and 33B (for example, through holes (VIA) or Cu-Cu bonding).

[0284] Using the image sensor 20 having a stacked structure makes it possible to apply a process suitable for manufacturing the pixels 21 to the semiconductor substrate 202 as the first layer, and to apply a process suitable for manufacturing the circuit portion to the semiconductor substrate 203 as the second layer. This enables process optimization when manufacturing the image sensor 20. In particular, application of a leading process becomes possible when manufacturing the circuit portion.

[0285] It should be noted that although a two-layer stacked structure in which the first semiconductor substrate 202 and the second semiconductor substrate 203 are stacked on each other is described here as an example, the stacked structure is not limited to a two-layer structure, and a structure including three or more layers may be adopted. In addition, in the case of a stacked structure having three or more layers, circuit parts including a row selector 23, a constant current source section 24, an analog-to-digital conversion section 25, a horizontal transfer scanner 26, a signal processing section 27, etc. may be formed in such a manner that these circuit parts are distributed in the second and subsequent semiconductor substrates.

[0286] In the imaging system 1B according to the second embodiment having the above-described configuration, the event detection sensor 10 and the image sensor 20 respectively perform an event detection operation and an imaging operation under the control of the controller 50. The event signal (event data) output from the event detection sensor 10 and the image data output from the image sensor 20 are provided to the motion identifier 30.

[0287] The motion identifier 30 identifies (detects) the motion of the object based on the event signal output from the event detection sensor 10. More specifically, the motion identifier 30 generates an event frame by framing the event signal output from the event detection sensor 10, and performs motion detection in the event frame. In the case of performing object recognition of the event using the event signal output from the event detection sensor 10, the object identifier 40 will perform object recognition based on the result of the motion detection provided from the motion identifier 30.

[0288] The image sensor 20 includes a synchronous imaging device, and performs imaging at a predetermined frame rate (e.g., a fixed frame rate). Therefore, it is not necessary to generate an event frame as in the case of the event detection sensor 10. Therefore, the image data output on a per-frame basis from the image sensor 20 is directly provided to the object recognizer 40. The object recognizer 40 then performs object recognition based on the image data in the frame.

[0289] Incidentally, since the event detection sensor 10 including the asynchronous imaging device has a pixel configuration with the event detector 63, the pixel size is inevitably larger than the pixel size of the image sensor 20 including the synchronous imaging device. Therefore, the resolution of the event detection sensor 10 is lower than that of the image sensor 20 that performs imaging at a fixed frame rate. On the contrary, the image sensor 20 including the synchronous imaging device is superior in resolution to the asynchronous imaging device.

[0290] Hereinafter, a description is given of a specific example in which, in the imaging system 1B according to the second embodiment, the event detection sensor 10 provided with a color filter and the image sensor 20 are used in combination to detect an event in a specific wavelength band, and predetermined signal processing is performed based on the detection result. It should be noted that the processing in each example described below is basically the same as that in FIG. Figure 1 The imaging system 1A shown is executed under the control of the controller 50 .

[0291] <Example 8>

[0292] Example 8 is an example of processing of the top-level concept of the second embodiment. Fig.24 The processing flow of the top-level concept according to Example 8 is shown in the flowchart of .

[0293] During the travel of the vehicle equipped with the imaging system 1B according to the second embodiment, the controller 50 first Fig.19 The object recognizer 40 shown performs object recognition using the image data of the image sensor 20, with the front as the focus area (step S101). As described above, the image sensor 20 is superior in resolution to the event detection sensor 10. Therefore, compared with the case where object recognition is performed based on the output of the event detection sensor 10, performing object recognition based on the output of the image sensor 20 enables the accuracy of recognition to be improved.

[0294] However, since the image sensor 20 performs imaging at a predetermined frame rate, the event detection sensor 10 is superior to the image sensor 20 in terms of power consumption. Therefore, the processing in step S102 and thereafter is performed using the event detection sensor 10. The same applies to the examples to be described later.

[0295] Next, the controller 50 performs ROI extraction in the area where object recognition has been performed as a focus area based on the output of the event detection sensor 10 (step S102), then detects an event in the area of ​​the extracted ROI (step S103), and then performs motion detection on the object detected as an event (step S104).

[0296] Next, the controller 50 performs object recognition in the event detection area of ​​the specific wavelength band (step S105), and then determines whether the object is recognized as an object of interest (step S106). In the case where the object is not recognized as an object of interest (no in S106), the controller 50 returns to step S101. In the case where the object is recognized as an object of interest (yes in S106), the controller 50 performs predetermined signal processing based on the object of interest (step S107), and ends a series of processing according to the top concept of Example 8.

[0297] As described above, according to Example 8, object recognition can be performed in the area of ​​interest by using image data of the image sensor 20 having a resolution superior to that of the event detection sensor 10. In addition, event detection is performed in a specific wavelength band, motion detection is performed on an object detected as an event, and in the case where an object is identified as an object of interest, predetermined signal processing based on the object is performed. Therefore, information in various wavelength bands can be detected and determined in the same manner as the driver, including lighting (flashing) of brake lights or tail lights of a vehicle traveling in front of the vehicle, flashing of turn indicators, color changes of traffic lights, electronic signs, etc.

[0298] Here, for example, "predetermined signal processing" means Fig.29 The drive system control unit 12010 of the vehicle control system 12000 shown processes command signals for various automatic controls, etc. When receiving the command signal from the event detection sensor 10, the vehicle control system 12000 automatically performs various controls, for example, applying brakes by automatic control or attracting the driver's attention by displaying an alarm.

[0299] <Example 9>

[0300] Example 9 is an example of sensing the taillights of the vehicle traveling ahead of the own vehicle after recognizing the vehicle traveling ahead of the own vehicle. Fig.25 The process flow of vehicle identification and taillight sensing according to Example 9 is shown in the flowchart of . In Example 9, for example, Fig. 10A The RCCC filter shown is used as a color filter provided for each pixel 11 in the event detection sensor 10. In the case of the RCCC filter, red (R) is used as a specific wavelength band. The same applies to the examples to be described later.

[0301] During the travel of the vehicle equipped with the imaging system 1B according to the second embodiment, the controller 50 firstly generates an image by using the image data of the image sensor 20. Fig.19 The object identifier 40 shown identifies the object as a front vehicle (step S111). Subsequently, the controller 50 performs extraction of ROI in the area where the object is identified as a vehicle (step S112), and then performs event detection in the red wavelength band in the area of ​​the extracted ROI (step S113).

[0302] Next, the controller 50 performs motion detection on the object detected as an event (step S114), then performs object recognition in the area where the motion is detected (step S115), and then determines whether the object is recognized as an object of interest, that is, recognized as a taillight (step S116).

[0303] Then, if the object is not recognized as a tail light (No in S116), the controller 50 returns to step S111. If the object is recognized as a tail light (Yes in S116), the controller 50 sends a signal to the user via the interface 80. Fig.29 The driving system control unit 12010 of the illustrated vehicle control system 12000 sends a command signal for brake control, warning display, etc. (step S117). Then, a series of processes for vehicle recognition and taillight sensing according to Example 9 ends.

[0304] When receiving a command signal for brake control, warning display, etc. from the imaging system 1B according to the second embodiment, the drive system control unit 12010 of the vehicle control system 12000 performs control, for example, applying brakes by automatic control or attracting the driver's attention by displaying an alarm.

[0305] As described above, in Example 9, object recognition is performed using image data of the image sensor 20 having a resolution superior to that of the event detection sensor 10, and thereafter, event detection in the red wavelength band and motion detection of the object detected as an event are performed. Then, in the case where the object is recognized as the taillight of a vehicle traveling in front of the own vehicle, it is determined that the distance between the own vehicle and the other vehicle traveling in front of the own vehicle is close enough to recognize the taillight of the other vehicle, and a command signal for brake control, alarm display, etc. is sent to the drive system control unit 12010 of the vehicle control system 12000. The vehicle control system 12000 is thus able to perform control, for example, to apply brakes by automatic control or to draw the driver's attention by displaying an alarm, thereby being able to contribute to safe driving.

[0306] It should be noted that in Example 9, when the recognition process is performed on the tail lamp, the object is recognized as the tail lamp through object recognition; however, the blinking of the tail lamp may be detected and used as information in object recognition (object sensing).

[0307] <Example 10>

[0308] Example 10 is an example of sensing a turn indicator (flashing light) after recognizing a vehicle traveling in front of the vehicle. Fig.26 The processing flow for vehicle identification and turn indicator sensing according to Example 10 is shown in the flowchart of .

[0309] During the travel of the vehicle equipped with the imaging system 1B according to the second embodiment, the controller 50 first recognizes the front object as a vehicle by using the image data of the image sensor 20 (step S121). Then, the controller 50 performs extraction of ROI in the area where the object is recognized as the vehicle (step S122), and then detects an event in the red wavelength band in the area of ​​the extracted ROI (step S123).

[0310] Next, the controller 50 performs motion detection on the object detected as the event (step S124), then performs object recognition in the area where the motion is detected (step S125), and then determines whether the object is recognized as an object of interest, that is, a turn indicator (flash light) (step S126). In the case where the object is not recognized as a turn indicator (No in S126), the controller 50 returns to step S121.

[0311] In addition, when the object is recognized as a turn indicator (Yes in S126), the controller 50 sends a Fig.29 The driving system control unit 12010 of the illustrated vehicle control system 12000 sends a command signal for executing control of the host vehicle based on the indication content of the turn indicator (step S127). Then, a series of processes for vehicle recognition and turn indicator sensing according to Example 10 ends.

[0312] As described above, in Example 10, object recognition is performed using image data of the image sensor 20 having a resolution superior to that of the event detection sensor 10, and thereafter, event detection in the red wavelength band and motion detection of the object detected as an event are performed. Then, after the vehicle traveling in front of the host vehicle is identified, recognition processing of the turn indicator is performed in the area where the vehicle has been identified, and a command signal for performing control of the host vehicle based on the indication content of the turn indicator (right turn, left turn, or danger) is sent to the drive system control unit 12010 of the vehicle control system 12000.

[0313] Here, for example, when the vehicle in front provides a direction indication of a right turn, it can be inferred that the vehicle traveling in front of the vehicle will no longer appear in front due to the right turn. Therefore, controlling the drive system control unit 12010 of the vehicle control system 12000 to increase the vehicle speed can be understood as "controlling the vehicle based on the indication content of the turn indicator". Alternatively, when the direction indicator of the vehicle in front indicates danger (flashing display), the vehicle traveling in front is likely to stop. Therefore, for example, it can be understood that the drive system control unit 12010 that controls the vehicle control system 12000 automatically applies the brakes.

[0314] It should be noted that in Example 10, when the recognition process of the turn indicator is performed, the object is recognized as the turn indicator through object recognition; however, the flashing of the turn indicator may be detected and used as information in object recognition (object sensing).

[0315] <Example 11>

[0316] Example 11 is an example of identifying a traffic light and sensing a red light. Fig. 27The processing flow for traffic light recognition and red light sensing according to Example 11 is shown in the flowchart of .

[0317] During travel of the vehicle equipped with the imaging system 1B according to the second embodiment, the controller 50 first performs object recognition by using image data of the image sensor 20 (step S131 ), and then detects a traffic light as an object to be recognized (step S132 ).

[0318] Next, the controller 50 performs extraction of ROI in the area where the traffic light is detected (step S133), then detects an event in the red wavelength band in the area of ​​the extracted ROI (step S134), and then performs motion detection on the object detected as the event (step S135).

[0319] Next, the controller 50 performs object recognition in the area where the motion is detected (step S136), and then determines whether the object is recognized as an object of interest, that is, a red light of a traffic light (step S137). If the object is not recognized as a red light (no in S137), the controller 50 returns to step S131.

[0320] In addition, when the object is recognized as a red light (Yes in S137), the controller 50 sends a Fig.29 The driving system control unit 12010 of the illustrated vehicle control system 12000 sends a command signal for brake control, warning display, etc. (step S138). Then, a series of processes for traffic light recognition and red light sensing according to Example 11 ends.

[0321] As described above, in Example 11, object recognition is performed using image data of the image sensor 20 having a better resolution than the event detection sensor 10, and thereafter, in the case where the object is a traffic light and its indication content is a red light, a command signal for brake control, alarm display, etc. is sent to the drive system control unit 12010 of the vehicle control system 12000. The vehicle control system 12000 is thus able to perform control, for example, to apply brakes by automatic control or to draw the driver's attention by displaying an alarm, thereby being able to contribute to safe driving.

[0322] It should be noted that in Example 11, when the traffic light is identified, the object is identified as a traffic light through object recognition; however, flickering information (e.g., frequency, duty cycle, etc.) can be detected and used as information in object recognition (object sensing).

[0323] <Example 12>

[0324] Example 12 is a modified example of Example 11, and is an example of sensing a traffic light changing from red to green. Fig.28 The processing flow for traffic signal recognition and sensing a change from a red light to a green light according to Example 12 is shown in the flowchart of .

[0325] Example 12 includes Fig. 27 The processing in the flowchart of Example 11 shown, in which steps S141 to S143 are introduced as processing after step S138. After the object is identified as a red light and a command signal for brake control, alarm display, etc. is sent (step S138), event detection in the green light wavelength band is performed in the area where the object is identified as a traffic light (step S141), and then, it is determined whether the traffic light is identified as a green light (step S142).

[0326] In the case where the traffic light is not recognized as a green light (No in S142), the controller 50 returns to step S131, and in the case where the traffic light is recognized as a green light (Yes in S142), the controller 50 sends a command signal for instructing to start the engine, start, etc. to the drive system control unit 12010 of the vehicle control system 12000 via the interface 80 (step S143). Then, a series of processes for traffic signal recognition and sensing a change from a red light to a green light according to Example 12 ends.

[0327] As described above, in Example 12, in the case where the vehicle is in a stopped state at a red light and the traffic light changes from red to green, a command signal for instructing to start the engine, start, etc. is sent to the drive system control unit 12010 of the vehicle control system 12000. The vehicle control system 12000 is thereby able to perform control to automatically start the engine and start the vehicle, thereby making it possible to prevent the occurrence of problems such as delayed start due to overlooking the traffic light.

[0328] It should be noted that when a traffic light is identified as changing from red to green, the threshold for blue or green may be set to a low value in areas where the object is identified as a traffic light and set to a high value in other areas, thereby allowing only green lights to be detected.

[0329] <<Modification Example>>

[0330] The technology according to the present disclosure has been described above based on the preferred embodiments; however, the technology according to the present disclosure is not limited to the embodiments. The configuration and structure of the imaging system described in the foregoing embodiments are illustrative and are modifiable.

[0331] <<Application Examples of Technology According to the Present Disclosure>>

[0332] The technology according to the present disclosure is applicable to various products. More specific application examples will be described below. For example, the technology according to the present disclosure can be implemented as an imaging device or imaging system installed on any type of mobile subject, such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), etc.

[0333] <Moving Subject>

[0334] The technology according to the present disclosure (the present technology) is applicable to various products. For example, the technology according to the present disclosure can be implemented as an imaging device installed on any type of mobile subject, such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), etc.

[0335] Fig.29 is a block diagram showing an example of a schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied.

[0336] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig.29 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown as the functional configuration of the integrated control unit 12050.

[0337] The drive system control unit 12010 controls the operation of the devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 is used as a control device to control: a drive force generating device for generating the drive force of the vehicle, such as an internal combustion engine, a drive motor, etc., a drive force transmitting mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating the braking force of the vehicle, etc.

[0338] The body system control unit 12020 controls the operation of various types of equipment configured for the body according to various programs. For example, the body system control unit 12020 is used as a control device to control the following items: a keyless entry system, a smart key system, a power window device, or various lights such as headlights, reverse lights, brake lights, turn signals, fog lights, etc. In this case, the body system control unit 12020 can receive as input radio waves transmitted from a mobile device that replaces the key or signals from various switches. The body system control unit 12020 receives these input radio waves or signals to control the door lock device, power window device, lights, etc. of the vehicle.

[0339] The vehicle exterior information detection unit 12030 detects information of the exterior of the vehicle including the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected with an imaging unit 12031. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image of the exterior of the vehicle and receives the captured image. Based on the received image, the vehicle exterior information detection unit 12030 may perform a process of detecting an object (such as a person, vehicle, obstacle, sign, symbol, etc. on the road), or perform a process of detecting the distance of the object.

[0340] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or can output the electrical signal as information about the measured distance. In addition, the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared rays.

[0341] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. For example, the in-vehicle information detection unit 12040 may be connected to a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that photographs the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 may calculate the driver's fatigue level or the driver's concentration level, or may distinguish whether the driver is dozing off.

[0342] The microcomputer 12051 can calculate a control target value for a driving force generation device, a steering mechanism, or a braking device based on information about the exterior or interior of the vehicle obtained by the exterior information detection unit 12030 or the interior information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control of functions intended to implement an advanced driver assistance system (ADAS), including collision avoidance or impact buffering for the vehicle, following driving based on a vehicle-to-vehicle distance, vehicle speed maintenance driving, an alarm for a vehicle collision, an alarm for a vehicle leaving a lane, and the like.

[0343] In addition, the microcomputer 12051 can perform collaborative control aimed at autonomous driving by controlling the driving force generating device, the steering mechanism, and the braking device based on the information about the outside or inside of the vehicle obtained by the outside information detection unit 12030 or the inside information detection unit 12040, which enables the vehicle to drive autonomously without relying on the driver's operation.

[0344] Furthermore, the microcomputer 12051 is capable of outputting a control command to the body system control unit 12020 based on the information about the exterior of the vehicle obtained by the exterior information detection unit 12030. For example, the microcomputer 12051 may control the headlights to change from high beam to low beam based on the position of a preceding vehicle or an oncoming vehicle detected by the exterior information detection unit 12030, thereby performing cooperative control aimed at preventing glare by controlling the headlights.

[0345] The sound / image output unit 12052 transmits an output signal of at least one of sound and image to an output device, which can notify information to the passengers of the vehicle or the outside of the vehicle in a visual or auditory manner. Fig.29 In the example of FIG. 1 , an audio speaker 12061, a display portion 12062, and an instrument panel 12063 are shown as output devices. The display portion 12062 may include, for example, at least one of an in-vehicle display and a head-up display.

[0346] Fig.30 12031 is a diagram showing an example of the installation position of the imaging unit 12031.

[0347] exist Fig.30 In the figure, vehicle 12100 includes imaging units 12101, 12102, 12103, 12104 and 12105 as imaging unit 12031.

[0348] The imaging units 12101, 12102, 12103, 12104 and 12105 may be disposed at the positions of the front nose, side mirrors, rear bumper, rear door and the upper portion of the windshield inside the vehicle 12100. The imaging unit 12101 disposed at the front nose and the imaging unit 12105 disposed at the upper portion of the windshield inside the vehicle mainly obtain images in front of the vehicle 12100. The imaging units 12102 and 12103 disposed at the side mirrors mainly obtain images on the sides of the vehicle 12100. The imaging unit 12104 disposed at the rear bumper or rear door mainly obtains images on the rear of the vehicle 12100. The front images obtained by the imaging units 12101 to 12105 are mainly used to detect front vehicles, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0349] By the way, Fig.30 An example of the shooting range of the imaging units 12101 to 12104 is shown. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided at the front nose. Imaging ranges 12112 and 12113 respectively indicate the imaging ranges of the imaging units 12102 and 12103 provided at the side mirrors. Imaging range 12114 indicates the imaging range of the imaging unit 12104 provided at the rear bumper or the rear door. For example, a bird's-eye view image of the vehicle 12100 observed from above can be obtained by superimposing the image data imaged by the imaging units 12101 to 12104.

[0350] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0351] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the time change of the distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and thereby extract the nearest three-dimensional object as the leading vehicle, which specifically exists on the driving path of the vehicle 12100 and travels at a predetermined speed (e.g., equal to or greater than 0 km / h) in substantially the same direction as the vehicle 12100. In addition, the microcomputer 12051 can pre-set the following distance to be maintained from the leading vehicle, and perform automatic braking control (including following parking control), automatic acceleration control (including following starting control), etc. Therefore, it is possible to perform cooperative control such as autonomous driving aimed at making the vehicle autonomously drive without relying on the operation of the driver.

[0352] For example, the microcomputer 12051 can classify the three-dimensional object data about the three-dimensional object into three-dimensional object data of two-wheeled vehicles, standard-sized vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104, extract the classified three-dimensional object data, and use the proposed three-dimensional object data for automatic avoidance of obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 that can be visually recognized by the driver of the vehicle 12100, and identifies obstacles that are difficult for the driver of the vehicle 12100 to visually recognize. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle. In the case where the collision risk is equal to or higher than a set value and therefore there is a possibility of collision, the microcomputer 12051 outputs an alarm to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or avoidance steering via the drive system control unit 12010. Thus, the microcomputer 12051 can assist driving to avoid collisions.

[0353] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 is capable of identifying pedestrians by determining whether there are pedestrians in the imaging images of the imaging units 12101 to 12104. This pedestrian recognition is performed, for example, by the following programs: a program for extracting feature points in the imaging images of the imaging units 12101 to 12104 as infrared cameras, and a program for determining whether it is a pedestrian by performing pattern matching processing on a series of feature points representing the outline of an object. When the microcomputer 12051 determines that there are pedestrians in the imaging images of the imaging units 12101 to 12104 and thus identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 to superimpose and display a square outline for emphasis on the identified pedestrian. The sound / image output unit 12052 can also control the display unit 12062 to display an icon representing a pedestrian at a desired position.

[0354] A description of an example of a vehicle control system 12000 to which the technology according to the present disclosure is applicable has been given above. The technology according to the present disclosure is applicable to, for example, the imaging unit 12031 in the above-mentioned configuration, etc. That is, the imaging system 1A of the first embodiment or the imaging system 1B according to the second embodiment of the present disclosure can be used as the imaging unit 12031 in the vehicle control system 12000 having the above-mentioned configuration, etc. Applying the technology according to the present disclosure to the imaging unit 12031, etc. makes it possible to perform event detection for each color in the corresponding wavelength band, and thus makes it possible to detect (sensing) the lighting (flashing) of the brake lights or tail lights of the vehicle, the flashing of the turn indicator, the color change of the traffic light, the electronic sign, etc. Therefore, the detection result can be fed to the autonomous driving, etc., thereby contributing to the safe driving of the vehicle.

[0355] <Possible Configuration of the Present Disclosure>

[0356] It should be noted that the present disclosure may have the following configurations.

[0357] <<Imaging System>>

[0358] [A-1] An imaging system comprising:

[0359] an event detection sensor that detects an event; and

[0360] a controller that controls event detection at the event detection sensor, wherein

[0361] The event detection sensor is provided with a color filter on a per-pixel basis, and

[0362] The controller controls event detection in a specific wavelength band based on the color filters.

[0363] [A-2] The imaging system according to [A-1], wherein the event detection sensor includes an asynchronous imaging device that detects a brightness change of a pixel that photoelectrically converts incident light exceeding a predetermined threshold as an event.

[0364] [A-3] The imaging system according to [A-2], configured to be used on a moving subject.

[0365] [A-4] The imaging system according to [A-3], wherein, when an object of interest is identified based on event detection by the event detection sensor, the controller performs signal processing based on the object of interest.

[0366] [A-5] The imaging system according to [A-4], wherein when the taillights of a vehicle traveling ahead of the own vehicle are identified as the object of interest, the controller sends a predetermined instruction signal to the control system of the mobile body.

[0367] [A-6] The imaging system according to [A-5], wherein, after recognizing the vehicle traveling ahead of the vehicle, the controller recognizes the taillights of the vehicle traveling ahead of the vehicle as the object of interest.

[0368] [A-7] An imaging system according to [A-6], wherein, when a tail light is identified as an object of interest, the controller sends a predetermined command signal to the control system of the mobile body when the relative speed between the vehicle and the vehicle traveling ahead is equal to or higher than a predetermined threshold.

[0369] [A-8] An imaging system according to [A-4], wherein, after identifying a vehicle traveling in front of the vehicle, when the turn indicator of the vehicle traveling in front of the vehicle is identified as an object of interest, the controller sends a predetermined command signal to the control system of the mobile body.

[0370] [A-9] The imaging system of [A-4], wherein when a traffic light is recognized as an object of interest and a red light is sensed, the controller sends a predetermined instruction signal to a control system of the mobile body.

[0371] [A-10] The imaging system according to [A-9], wherein, when sensing that a traffic light in a red light state has changed from red to green, the controller sends a predetermined instruction signal to the control system of the mobile body.

[0372] [A-11] The imaging system according to any one of [A-1] to [A-3], including an image sensor that performs imaging at a predetermined frame rate.

[0373] [A-12] An imaging system according to [A-11], wherein, after performing object recognition based on image data of the image sensor, the controller performs signal processing based on the object of interest when the object of interest is recognized based on event detection of the event detection sensor.

[0374] [A-13] An imaging system according to [A-12], wherein, after identifying a vehicle traveling in front of the vehicle, when the taillights of the vehicle traveling in front of the vehicle are identified as an object of interest, the controller sends a predetermined command signal to the control system of the mobile body.

[0375] [A-14] An imaging system according to [A-12], wherein, after identifying a vehicle traveling ahead of the vehicle, when the turn indicator of the vehicle traveling ahead of the vehicle is identified as an object of interest, the controller sends a predetermined command signal to the control system of the mobile body.

[0376] [A-15] The imaging system of [A-12], wherein when a traffic light is identified as an object of interest and a red light is sensed, the controller sends a predetermined instruction signal to a control system of the mobile body.

[0377] [A-16] The imaging system according to [A-15], wherein, when sensing that a traffic light in a red light state has changed from red to green, the controller sends a predetermined instruction signal to the control system of the mobile body.

[0378] Reference numerals list

[0379] 1A Imaging System According to First Embodiment

[0380] 1B Imaging System According to Second Embodiment

[0381] 10 Event Detection Sensors

[0382] 11 pixels

[0383] 12 Pixel array section

[0384] 13. Drive unit

[0385] 14 Arbitrator Department (Arbitration Department)

[0386] 15 Column Processing Unit

[0387] 16 Signal Processing Unit

[0388] 20 Image Sensor

[0389] 21 pixels

[0390] 22 Pixel array section

[0391] 23 Row Selector

[0392] 24 Constant current source

[0393] 25 Analog-to-digital conversion unit

[0394] 26 Horizontal transfer scanner

[0395] 27 Signal Processing Unit

[0396] 28 Timing Controller

[0397] 30 Motion Recognizer

[0398] 40 Object Recognizer

[0399] 50 Controller

[0400] 60 Operation mode definition section

[0401] 70 Image recording unit

[0402] 80 interface

Claims

1. An imaging system, comprising: Event detection sensors detect events; as well as A controller controls event detection at the event detection sensor, wherein: The event detection sensor is provided with a color filter on a per-pixel basis so that the event detection sensor can detect an event in each of a plurality of specific wavelength bands, and the controller controls the event detection in the plurality of specific wavelength bands based on the color filters, The imaging system is configured for use on a moving subject, When an object of interest is identified based on the event detection of the event detection sensor, the controller performs signal processing based on the object of interest, and the signal processing refers to sending a corresponding command signal to the control system of the mobile body to perform automatic control.

2. The imaging system according to claim 1, wherein: The event detection sensor includes an asynchronous imaging device that detects, as the event, a change in brightness of a pixel that photoelectrically converts incident light exceeding a predetermined threshold.

3. The imaging system according to claim 1, wherein: When recognizing the taillight of a vehicle traveling ahead of the own vehicle as the object of interest, the controller sends a predetermined instruction signal to the control system of the mobile body.

4. The imaging system according to claim 3, wherein: After recognizing the vehicle traveling ahead of the host vehicle, the controller recognizes the taillight of the vehicle traveling ahead of the host vehicle as the object of interest.

5. The imaging system according to claim 4, wherein: When the taillight is recognized as the object of interest, the controller sends the predetermined instruction signal to the control system of the mobile body in the case where the relative speed between the own vehicle and the vehicle traveling ahead is equal to or higher than a predetermined threshold.

6. The imaging system according to claim 1, wherein: After recognizing a vehicle traveling ahead of the own vehicle, the controller transmits a predetermined instruction signal to the control system of the mobile body when recognizing a turn indicator of the vehicle traveling ahead of the own vehicle as the object of interest.

7. The imaging system of claim 1, wherein: When a traffic light is recognized as the object of interest and a red light is sensed, the controller sends a predetermined instruction signal to the control system of the mobile body.

8. The imaging system according to claim 7, wherein: When sensing that the traffic light in a red light state has changed from red to green, the controller transmits the predetermined instruction signal to the control system of the mobile body.

9. The imaging system according to claim 1, comprising an image sensor that performs imaging at a predetermined frame rate.

10. The imaging system of claim 9, wherein: After performing object recognition based on image data of the image sensor, when an object of interest is recognized based on the event detection of the event detection sensor, the controller performs signal processing based on the object of interest.

11. The imaging system according to claim 10, wherein: After recognizing a vehicle traveling ahead of the own vehicle, the controller transmits a predetermined instruction signal to a control system of the mobile body when recognizing a tail light of the vehicle traveling ahead of the own vehicle as the object of interest.

12. The imaging system of claim 10, wherein: After recognizing a vehicle traveling ahead of the own vehicle, the controller transmits a predetermined instruction signal to a control system of the mobile body when recognizing a turn indicator of the vehicle traveling ahead of the own vehicle as the object of interest.

13. The imaging system of claim 10, wherein: When a traffic light is identified as the object of interest and a red light is sensed, the controller sends a predetermined instruction signal to a control system of the mobile body.

14. The imaging system of claim 13, wherein: When sensing that the traffic light in a red light state has changed from red to green, the controller transmits the predetermined instruction signal to the control system of the mobile body.

Images