Imaging device, imaging method, and imaging system

Abstract

[Problem] The present disclosure provides an asynchronous imaging device, an imaging method, and an imaging system, which make it possible to convert, into a pixel value, luminance information of a region where no events are taking place. [Solution] An imaging device according to the present disclosure comprises: a photoelectric conversion unit having a plurality of photoelectric conversion elements which each photoelectrically convert light for irradiating different locations of a two-dimensional imaging surface and generate an electric signal; a light source unit that irradiates the imaging surface with irradiation light; and a first detection unit that detects a detection signal when the amount of change in the electric signal generated by each of the photoelectric conversion elements exceeds a prescribed value.

Description

Imaging device, imaging method, and imaging system

[0001] The present disclosure relates to an imaging device, an imaging method, and an imaging system.

[0002] As one type of event-driven imaging device, an asynchronous imaging device called a DVS (Dynamic Vision Sensor) has been proposed (see, for example, Patent Document 1). In an asynchronous imaging device, only when some event (e.g., movement) occurs in a scene, data of a portion of the image where the brightness level has changed due to the event is acquired. Therefore, an asynchronous imaging device can acquire image data faster than a general synchronous imaging device, which unnecessarily acquires all image data at a fixed frame rate.

[0003] Special Publication No. 2017-535999

[0004] However, if no event occurs, the image information is not updated, which may make it difficult to determine whether the subject is simply not moving or whether the subject is not within the angle of view.

[0005] Therefore, the present disclosure provides an asynchronous imaging device, imaging method, and imaging system that are capable of converting luminance information of an area where no event has occurred into pixel values.

[0006] In order to solve the above problems, according to the present disclosure, there is provided an imaging device comprising: a photoelectric conversion unit having a plurality of photoelectric conversion elements that each photoelectrically converts light irradiated at different positions on a two-dimensional imaging surface to generate an electrical signal; a light source unit that irradiates the imaging surface with the irradiating light; and a first detection unit that detects a detection signal when a change in the electrical signal generated by the plurality of photoelectric conversion elements exceeds a predetermined value.

[0007] The imaging device may further include an imaging optical system that directs light onto the imaging surface.

[0008] The light source unit may monotonically increase or decrease the amount of illumination light irradiated onto the imaging surface while light is incident on the imaging surface via the imaging optical system.

[0009] The image generating unit may further comprise an image generating unit that generates image data in which values ​​relating to the number of times of detection signals detected by the first detecting unit are arranged in two-dimensional coordinates.

[0010] The image forming apparatus may further include a light source driving unit that controls the light source unit.

[0011] The light source driving section may monotonically increase the amount of irradiation light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount.

[0012] The light source driving section may monotonically decrease the amount of irradiation light irradiated onto the imaging surface from the second irradiation amount to the first irradiation amount.

[0013] The first detector may detect the detection signal when a change in the electrical signal exceeds a predetermined ratio.

[0014] The image generating section may generate, as the image data, a pixel value corresponding to a value obtained by raising the number of times the detection signal is detected by the first detecting section to a power based on the ratio.

[0015] The light source unit may include a plurality of light sources, and the light source driver may monotonically increase or decrease the light intensity of the plurality of light sources over different time ranges.

[0016] The light source unit may have a first light source of a first wavelength and a second light source of a second wavelength different from the first wavelength, the light source driving unit may monotonically increase the amount of irradiation light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount using the irradiation light of the first light source, and the image generating unit may generate the image data as first image data.

[0017] After generating the first image data, the light source driving unit may monotonically increase the amount of light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount using light irradiated from the second light source, and the image generating unit may generate the image data as second image data.

[0018] The light source unit may further have a third light source having a third wavelength different from the first wavelength and the second wavelength, and after generating the second image data, the light source driving unit monotonically increases the irradiation light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount using the irradiation light of the third light source, and the image generating unit may generate the image data as third image data.

[0019] The image generating section may generate color image data based on the first image data, the second image data, and the third image data.

[0020] The first to third light sources may be any one of a red LED, a blue LED, and a green LED.

[0021] The first detection unit may detect the detection signal when the absolute value of the amount of change in the direction in which the signal level of the electrical signal increases exceeds a first threshold, and may also detect the detection signal when the absolute value of the amount of change in the direction in which the signal level of the electrical signal decreases exceeds a second threshold.

[0022] The image capturing apparatus may further include a control unit that switches between a first mode in which the light source of the light source unit is turned off or the amount of light is maintained constant and an image is generated based on the detection signal, and a second mode in which the light source of the light source unit is turned on and, with light incident on the imaging surface via the imaging optical system, the amount of irradiation light irradiated onto the imaging surface is monotonically increased or decreased and an image is generated based on the number of detection signals.

[0023] The second mode may further include a third mode in which no image is generated based on the detection signal.

[0024] In order to solve the above-mentioned problems, according to the present disclosure, there is provided an imaging method, which includes monotonically increasing or monotonically decreasing irradiated light irradiated onto an imaging surface while light is incident on the imaging surface via an imaging optical system, photoelectrically converting the irradiated light using a plurality of photoelectric conversion elements to generate electrical signals, and detecting the number of times a detection signal is generated when the absolute value of the amount of change in the electrical signals exceeds a predetermined threshold.

[0025] In order to solve the above problems, according to the present disclosure, there is provided an imaging system comprising: an imaging device; and a display unit that displays an image generated by the imaging device, wherein the imaging device comprises: a photoelectric conversion unit having a plurality of photoelectric conversion elements that each photoelectrically converts light irradiated at different positions on a two-dimensional imaging surface to generate an electrical signal; a light source unit that irradiates the imaging surface with irradiated light; and a first detection unit that detects a detection signal when an amount of change in the electrical signal generated by the plurality of photoelectric conversion elements exceeds a predetermined value.

[0026] 1 is a block diagram showing an example of a system configuration of an imaging system to which the technology according to the present disclosure is applied. A schematic diagram showing an example of the configuration of an imaging device. A plan view showing an example of the arrangement of an imaging surface of an imaging section and a light source section. A block diagram showing an example of the configuration of an imaging section. A block diagram showing an example of the configuration of a pixel array section 21. A circuit diagram showing an example of the circuit configuration of a pixel. A block diagram showing an example of the configuration of a control section. A diagram showing an example of generation of an address event detection signal from an address event detection section. A diagram schematically showing the number of times a detection signal is generated for each pixel. A diagram for explaining an example of gradation processing by an image generation section. A diagram showing an example of an image formed on an imaging surface via an imaging optical system. A diagram showing an example of an event image at a predetermined timing. A diagram showing an example of the reciprocal of the number of detections. A diagram showing an example of a gradation image generated by an image generation section. A diagram schematically showing features of an example of gradation processing. A plan view showing an example of the configuration of a plurality of light source sections. A flowchart showing an example of generation of a gradation image of an imaging device. A block diagram showing an example of the configuration of a light source section according to a second embodiment. A diagram showing an example of generation of an address event detection signal according to the second embodiment. 1 is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a mobile body control system to which the technology according to the present disclosure can be applied.

[0027] Hereinafter, embodiments of an imaging device, an imaging method, and an imaging system will be described with reference to the drawings. The following description will focus on the main components of the imaging device, the imaging method, and the imaging system, but the imaging device, the imaging method, and the imaging system may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0028] First Embodiment FIG. 1 is a block diagram showing an example of the system configuration of an imaging system 1 to which the technology according to the present disclosure is applied.

[0029] As shown in FIG. 1 , an imaging system 1 to which the technology according to the present disclosure is applied includes an imaging device 2, an operation unit 15, and a display unit 16. The imaging system 1 according to the present embodiment is, for example, an example of a surveillance system. The imaging device 2 according to the present embodiment is, for example, an asynchronous imaging device called a DVS (Dynamic Vision Sensor), and is capable of generating, for example, event images and still images in a surveillance area. The imaging device 2 is an example of an electronic device according to the present disclosure, and examples of such electronic devices can be used in surveillance systems, camera systems mounted on industrial robots, in-vehicle camera systems, and the like.

[0030] The imaging device 2 includes an imaging optical system 11 , a plurality of light source units 12 , a recording unit 13 , a control unit 14 , and an imaging unit 20 .

[0031] The operation unit 15 operates the imaging device 2, for example, via the network nw. For example, the operation unit 15 inputs an imaging mode switching signal. The display unit 16 is, for example, a monitor, and is capable of displaying event images and still images via the network nw. In other words, the monitor can monitor the monitoring area via the network nw.

[0032] Fig. 2 is a schematic diagram showing an example configuration of the imaging device 2. Fig. 3 is a plan view showing an example arrangement of the imaging surface 21a of the imaging unit 20 and the light source unit 12. In the embodiment, an X direction, a Y direction perpendicular to the X direction, and a Z direction perpendicular to the X direction and the Y direction are defined. As shown in Fig. 2, in the imaging device 2, the imaging optical system 11 is, for example, a single lens or multiple lenses, and captures incident light from a subject and forms an image on the imaging surface 21a of the imaging unit 20.

[0033] 2 and 3, the four light source units 12 irradiate the imaging surface 21a with light from four directions. The light source units 12 include an LED light source 120w and a light source drive unit 125. The LED light source 120w is, for example, a white LED. The light source drive unit 125 changes the amount of irradiation, including turning on and off the LED light source 120w, under the control of the control unit 14. The light source drive unit 125 can also independently control the amount of irradiation light from the four light source units 12.

[0034] The four light source units 12 monotonically increase or decrease the amount of light irradiated onto the imaging surface 21a while light is incident on the imaging surface 21a via the imaging optical system 11. That is, the light source driving unit 125 monotonically increases the amount of light irradiated onto the imaging surface 21a from the first irradiation amount to the second irradiation amount. Alternatively, the light source driving unit 125 monotonically decreases the amount of light irradiated onto the imaging surface 21a from the second irradiation amount to the first irradiation amount.

[0035] The imaging unit 20 performs predetermined signal processing, such as image recognition processing, on the captured image data, and outputs data indicating the processing result and an address event detection signal (hereinafter, sometimes simply referred to as a "detection signal") to the recording unit 13. A method for generating an address event detection signal will be described later. The recording unit 13 stores data supplied from the imaging unit 20 via a signal line. The control unit 14 is configured, for example, by a microcomputer, and controls the imaging operation and image generation in the imaging unit 20. The control unit 14 will be described in detail later.

[0036] [Image Capture Section (Arbiter System)] FIG. 4 is a block diagram showing an example of the configuration of the image capture section 20 in the image capture system 1 to which the technology according to the present disclosure is applied.

[0037] 4, the imaging unit 20 of the present disclosure is an asynchronous imaging unit called DVS, and includes a pixel array unit 21, a driving unit 22, an arbiter unit (arbitration unit) 23, a column processing unit 24, and a signal processing unit 25. The imaging surface 21a described above is the imaging surface of the pixel array unit 21. The pixel array unit 21 according to this embodiment corresponds to a photoelectric conversion unit.

[0038] In the imaging unit 20 having the above configuration, a plurality of pixels 30 are two-dimensionally arranged in a matrix (array) in the pixel array unit 21. Vertical signal lines VSL (see FIG. 5 ) are wired for each pixel column in this matrix-like pixel arrangement.

[0039] Each of the plurality of pixels 30 generates an analog signal of a voltage corresponding to the photocurrent as an electrical signal. Each of the plurality of pixels 30 detects the presence or absence of an address event based on whether or not the amount of change in the electrical signal exceeds a predetermined threshold. When an address event occurs, the pixel 30 outputs a request to the arbiter unit 23.

[0040] The driver 22 drives each of the plurality of pixels 30 arranged on the imaging surface 21 a to output an electrical signal generated by each pixel 30 to the column processor 24 .

[0041] The arbiter unit 23 arbitrates requests from each of the multiple pixels 30 and transmits a response based on the arbitration result to the pixel 30. Upon receiving the response from the arbiter unit 23, the pixel 30 supplies a detection signal indicating the detection result (address event detection signal) to the drive unit 22 and the signal processing unit 25. The detection signal from the pixel 30 can also be read out from multiple rows.

[0042] The column processing unit 24 is composed of, for example, an analog-to-digital converter, and performs processing to convert, into digital signals, analog electrical signals output from the pixels 30 in each pixel column of the pixel array unit 21. Then, the column processing unit 24 supplies the digital signals after analog-to-digital conversion to the signal processing unit 25.

[0043] The signal processing unit 25 performs predetermined signal processing such as CDS (Correlated Double Sampling) processing and image recognition processing on the digital signal supplied from the column processing unit 24. Then, the signal processing unit 25 supplies data indicating the processing result and the detection signal supplied from the arbiter unit 23 to the recording unit 13 (see FIG. 1 ) via a signal line.

[0044] [Configuration Example of Pixel Array Unit] FIG. 5 is a block diagram showing an example of the configuration of the pixel array unit 21. As shown in FIG.

[0045] In the pixel array section 21, which is made up of a plurality of pixels 30 arranged two-dimensionally in a matrix, each of the plurality of pixels 30 has a light receiving section 31, an electrical signal generating section 32, and an address event detecting section 33.

[0046] In the pixel 30 having the above configuration, the light receiving section 31 in the light receiving region performs photoelectric conversion on incident light to generate a photocurrent. Then, the light receiving section 31 supplies the photocurrent generated by photoelectric conversion to either the electrical signal generating section 32 or the address event detecting section 33 under the control of the driving section 22 (see FIG. 2 ).

[0047] The electrical signal generating unit 32 generates an electrical signal SIG, which is a voltage signal corresponding to the photocurrent supplied from the light receiving unit 31 in the light receiving region, and supplies this generated electrical signal SIG to the column processing unit 24 (see Figure 2) via the vertical signal line VSL.

[0048] The address event detector 33 detects the presence or absence of an address event by detecting whether the amount of change in the electrical signal exceeds a predetermined threshold value, based on the photocurrent from each of the light receiving sections 31 in the light receiving region. The photocurrent is proportional to the amount of light irradiated on the imaging surface 21 a.

[0049] An address event may be, for example, an ON event indicating that the amount of change in the electrical signal has exceeded a first upper threshold, and an OFF event indicating that the amount of change has fallen below a second lower threshold. The address event detection signal may be, for example, one bit indicating the detection result of an ON event and one bit indicating the detection result of an OFF event. The address event detection unit 33 may be configured to detect only ON events or only OFF events. For example, to detect only OFF events, a detection signal indicating an ON event is generated when the amount of change in the current electrical signal reaches 1.25. That is, a detection signal indicating an ON event is generated when the amount of light irradiated on the imaging surface 21a increases by 25 percent. The address event detection unit 33 in this embodiment corresponds to the first detection unit.

[0050] When an address event occurs, the address event detector 33 in the light-receiving region supplies a request for transmission of an address event detection signal to the arbiter unit 23 (see FIG. 2 ). Then, upon receiving a response to the request from the arbiter unit 23, the address event detector 33 supplies an address event detection signal to the driver unit 22 and the signal processor 25.

[0051] 6 is a circuit diagram showing an example of the circuit configuration of the pixel 30. As described above, each of the plurality of pixels 30 includes a light receiving unit 31, an electrical signal generating unit 32, and an address event detecting unit 33.

[0052] In the pixel 30 having the above configuration, the light receiving unit 31 includes a light receiving element (photoelectric conversion element) 311, a transfer transistor 312, and an OFG (Over Flow Gate) transistor 313. For example, an N-type MOS (Metal Oxide Semiconductor) transistor is used as the transfer transistor 312 and the OFG transistor 313. The transfer transistor 312 and the OFG transistor 313 are connected in series to each other.

[0053] The light receiving element 311 in the light receiving region is connected to a common connection node N １ and ground, and performs photoelectric conversion on incident light to generate charges in an amount corresponding to the amount of incident light.

[0054] A transfer signal TRG is supplied to the gate electrode of the transfer transistor 312 in the light receiving region from the drive unit 22 shown in Fig. 4. In response to the transfer signal TRG, the transfer transistor 312 supplies the charge photoelectrically converted by the light receiving element 311 to the electric signal generation unit 32.

[0055] A control signal OFG is supplied from the driving unit 22 to the gate electrode of the OFG transistor 313. In response to the control signal OFG, the OFG transistor 313 supplies the electrical signal generated by the light receiving element 311 to the address event detection unit 33. The electrical signal supplied to the address event detection unit 33 is a photocurrent made up of electric charges.

[0056] The electrical signal generating unit 32 includes a reset transistor 321, an amplifying transistor 322, a selecting transistor 323, and a floating diffusion layer 324. The reset transistor 321, the amplifying transistor 322, and the selecting transistor 323 are, for example, N-type MOS transistors.

[0057] The electric charge generated by photoelectric conversion in the light receiving element 311 is supplied from the light receiving unit 31 to the electric signal generating unit 32 by the transfer transistor 312. The electric charge supplied from the light receiving unit 31 is accumulated in the floating diffusion layer 324. The floating diffusion layer 324 generates a voltage signal having a voltage value corresponding to the amount of accumulated electric charge. In other words, the floating diffusion layer 324 converts the electric charge into a voltage.

[0058] The reset transistor 321 is connected to the power supply voltage V ＤＤ and the floating diffusion layer 324. A reset signal RST is supplied to the gate electrode of the reset transistor 321 from the drive unit 22. The reset transistor 321 initializes (resets) the amount of charge in the floating diffusion layer 324 in response to the reset signal RST.

[0059] The amplifier transistor 322 receives the power supply voltage V ＤＤ The amplifier transistor 322 is connected in series with the selection transistor 323 between the power supply line and the vertical signal line VSL. The amplifier transistor 322 amplifies the voltage signal converted from charge to voltage by the floating diffusion layer 324.

[0060] A selection signal SEL is supplied from the drive unit 22 to the gate electrode of the selection transistor 323. In response to the selection signal SEL, the selection transistor 323 outputs the voltage signal amplified by the amplification transistor 322 as an electrical signal SIG to the column processing unit 24 (see FIG. 4) via the vertical signal line VSL.

[0061] 7 is a block diagram showing an example of the configuration of the control unit 14. As shown in FIG. 7, the control unit 14 includes a mode control unit 130, a light source control unit 131, an image generation unit 132, and an input / output unit 133.

[0062] The mode control unit 130 changes between multiple imaging modes in the imaging device 2. The first mode is a mode in which the multiple light source units 12 (see FIG. 2) are turned off or the light intensity is maintained at a fixed value, and an event image is generated based on the detection signal of an address event from the address event detection unit 33 (see FIG. 6).

[0063] The second mode is a mode in which, while light is incident on the imaging surface 21a via the imaging optical system 11 (see FIG. 2), the illumination light irradiated onto the imaging surface 21a is monotonically increased or decreased, and a gradation image is also generated based on the number of occurrences of an address event detection signal from the address event detection unit 33 (see FIG. 6). The third mode is a mode in which, while in the second mode, an event image is not generated. As a result, in the first mode, an event image is generated, in the second mode, an event image and a gradation image are generated, and in the third mode, a gradation image is generated. For example, switching from Mode 1 to Mode 2 or Mode 3 can be performed by a switching signal from the operation unit 15 (see FIG. 1). Alternatively, the control unit 4 may normally continue monitoring in Mode 1, and when the number of detected events exceeds a predetermined number, switch to Mode 2 or Mode 3 for a predetermined confirmation time. This allows continuous monitoring of the occurrence of events.

[0064] The light source control unit 131 controls the light amounts of the plurality of light source units 12 under the control of the mode control unit 130. That is, in the first mode, the light source control unit 131 maintains a constant light amount of the plurality of light source units 12. Preferably, the light source control unit 131 turns off the plurality of light source units 12. In the second and third modes, the light source control unit 131 monotonically increases or decreases the amount of light irradiated onto the imaging surface 21a while light is incident on the imaging surface 21a via the imaging optical system 11 (see FIG. 2 ).

[0065] The image generation unit 132 generates a grayscale image based on the number of occurrences of the address event detection signal in the second mode and the third mode. The image generation unit 132 will be described in detail later. The input / output unit 133 is an interface that inputs and outputs image signals and operation signals via a network.

[0066] [First Mode] Referring to FIG. 6, in the first mode, the mode control unit 130 supplies a control signal OFG to the OFG transistor 313 of the light receiving unit 31 to drive the OFG transistor 313 and cause it to supply a photocurrent to the address event detection unit 33.

[0067] When an address event is detected in a pixel 30 in the light-receiving region, the drive unit 22 turns off the OFG transistor 313 of that pixel 30 to stop the supply of photocurrent to the address event detection unit 33. Next, the drive unit 22 drives the transfer transistor 312 by supplying a transfer signal TRG to the transfer transistor 312, causing the charge photoelectrically converted by the light-receiving element 311 to be transferred to the floating diffusion layer 324.

[0068] In this way, the imaging unit 20 having a pixel array section 21 in which pixels 30 of the above configuration are arranged two-dimensionally outputs only the electrical signals of the pixels 30 in the light-receiving area in which an address event is detected to the column processing section 24, and an event image is generated.

[0069] [Second Mode] The second mode will be described using Fig. 8 while referring to Fig. 6. Fig. 8 is a diagram showing an example of generation of an address event detection signal by the address event detection unit 33 (see Fig. 6) of the pixel 30. The horizontal axis represents time, and the vertical axis represents the amount of irradiation on the imaging surface 21a. The detection signal as represents the generation timing of the detection signal at the pixel 30a where the amount of irradiation is high and light is incident on the imaging surface 21a via the imaging optical system 11 (see Fig. 2). The detection signal bs represents the generation timing of the detection signal at the pixel 30b where the amount of irradiation is low and light is incident on the imaging surface 21a via the imaging optical system 11 (see Fig. 2).

[0070] In the second mode, the mode control unit 130 first supplies a control signal OFG to the OFG transistor 313 of the light receiving unit 31 to drive the OFG transistor 313 and cause it to supply a photocurrent to the address event detection unit 33. At this time, as shown in Fig. 8 , the mode control unit 130 causes incident light via the imaging optical system 11 to form an image on the imaging surface 21a, while controlling the light source control unit 131 to control the light amounts of the multiple light source units 12 so that the light amount monotonically increases from the first irradiation amount L1 to the second irradiation amount L2.

[0071] For example, the address event detector 33 outputs a detection signal when an electrical signal proportional to the photocurrent increases by 25 percent. The ratio of the address event detector 33 is not limited to 25 percent and can be set to any ratio depending on the purpose. For example, the mode controller 130 can set the ratio in response to an operation signal from the operation unit 15.

[0072] In the pixel 30b where the amount of light incident on the imaging surface 21a is low, when the light amount from the light source units 12 reaches the light amount Lb, the address event detector 33 generates a detection signal bs in time series every time the electrical signal proportional to the photocurrent increases by 25%. The detection signal bs is recorded in the recording unit 13 (see FIG. 1).

[0073] On the other hand, in the pixel 30a where the amount of incident light is high, when the light amount of the plurality of light source units 12 reaches the light amount La, the address event detection unit 33 generates a detection signal as in time series every time the electrical signal proportional to the photocurrent increases by 25%. The detection signal as is recorded in the recording unit 13. As can be seen from this, the number of times that a detection signal is generated increases as the irradiance of light incident on the imaging surface 21a via the imaging optical system 11 (see FIG. 2) decreases.

[0074] 9 is a diagram schematically illustrating the number of times a detection signal is generated for each pixel 30 in the pixel array unit 21 in the second mode. As described above, the number of times a detection signal is generated increases for a pixel 30 that receives a lower amount of light incident on the imaging surface 21 a.

[0075] On the other hand, in mode 2, the driving in mode 1 is also maintained. That is, when an address event is detected in a pixel 30 in the light-receiving region, the driving unit 22 turns off the OFG transistor 313 of that pixel 30 to stop the supply of photocurrent to the address event detection unit 33. Next, the driving unit 22 drives the transfer transistor 312 by supplying a transfer signal TRG to the transfer transistor 312, and causes the charge photoelectrically converted in the light-receiving element 311 to be transferred to the floating diffusion layer 324.

[0076] Next, the drive unit 22 turns off the transfer transistor 312 and supplies a control signal OFG to the OFG transistor 313 of the light receiving unit 31 to drive the OFG transistor 313 and cause the OFG transistor 313 to supply a photocurrent to the address event detection unit 33. For this reason, in mode 2, there is a limit to the shortening of the time taken for the irradiation amount to monotonically increase from the first irradiation amount L1 to the second irradiation amount L2 so as to prevent a decrease in the number of detection signals as and bs detected.

[0077] In this way, the imaging unit 20 having the pixel array unit 21 in which the pixels 30 having the above configuration are arranged two-dimensionally outputs only the electrical signals of the pixels 30 in the light-receiving region in which an address event is detected to the column processing unit 24, thereby generating an event image. Furthermore, in the second mode, a grayscale image is generated based on the number of occurrences of the detection signal.

[0078] 6 again, in the third mode, the mode control unit 130 constantly maintains the OFG transistor 313 of the pixel 30 in an off state. The mode control unit 130 also constantly supplies a control signal OFG to the OFG transistor 313 of the light receiving unit 31. As a result, in the third mode, the OFG transistor 313 is driven to constantly supply a photocurrent to the address event detection unit 33.

[0079] In the third mode, the mode control unit 130 controls the light source control unit 131 to control the light intensity of the plurality of light source units 12 so that the intensity monotonically increases from the first irradiation intensity L1 to the first irradiation intensity L2. The address event detection unit 33 outputs a detection signal when, for example, an electrical signal proportional to the photocurrent increases by 25 percent. The ratio of the address event detection unit 33 is not limited to 25 percent and can be set to any ratio depending on the purpose. For example, the mode control unit 130 can set the ratio in response to an operation signal from the operation unit 15.

[0080] In the pixel 30b with a low irradiation amount, when the light amount of the plurality of light source units 12 reaches the light amount Lb, the address event detection unit 33 generates a detection signal bs in time series every time the electrical signal proportional to the photocurrent increases by 25%. The detection signal bs is recorded in the recording unit 13 (see FIG. 1).

[0081] In the second mode, the OFG transistor 313 is turned off, causing a period of time during which the supply of photocurrent to the address event detection unit 33 is stopped, and therefore the detection signals as and bs tend to be generated less frequently as the time it takes for the signals to monotonically increase from the first irradiation amount L1 to the second irradiation amount L2 becomes shorter. However, in the third mode, the OFG transistor 313 of the pixel 30 is maintained in the off state, and the control signal OFG is constantly supplied to the OFG transistor 313 of the light receiving unit 31, thereby making it possible to suppress a decrease in the detection accuracy of the detection signals as and bs more than in the second mode.

[0082] Fig. 10 is a diagram illustrating an example of gradation processing by the image generating unit 132. Based on the characteristics shown in Fig. 8 and Fig. 9, the image generating unit 132 generates a gradation image based on the number of occurrences of address event detection signals.

[0083] The illuminance difference LW1, which is the difference between the second irradiation amount L2 and the irradiation amount La of pixel 30a, corresponds to the number of times the detection signal as is generated. Similarly, the illuminance difference LW1, which is the difference between the second irradiation amount L2 and the irradiation amount Lb of pixel 30b, corresponds to the number of times the detection signal bs is generated.

[0084] Therefore, the image generation unit 132 calculates the illuminance difference LW1 by raising the number of occurrences of the detection signal as to 1.25 and further multiplying it by a coefficient k1 to obtain D1. Similarly, the image generation unit 132 calculates the illuminance difference LW1 by raising the number of occurrences of the detection signal as to 1.25 and further multiplying it by a coefficient k1 to obtain D2. The coefficient k1 is a normalization coefficient for converting the illuminance difference to a pixel value. The image generation unit 132 then subtracts D1 from the pixel value Dmax corresponding to the second irradiation amount L2 to obtain the pixel value of pixel 30a, and subtracts D2 from the pixel value Dmax to obtain the pixel value of pixel 30b. Note that 1.25 is a value based on a ratio of 25 percent as the detection threshold setting value of the address event detection unit 33. Furthermore, as can be seen from FIG. 10 , even when the irradiation amount L2 is monotonically decreased from the second irradiation amount L2 to the first irradiation amount L1, the same number of detection signals can be generated by setting the threshold for off events in the address event detection unit 33. In this way, the image generation unit 132 generates image data having pixel values ​​corresponding to the number of detection signals detected by the address event detection unit 33 raised to the power of 1.25, a number based on the ratio (25 percent).

[0085] 11 to 14 are examples of image generation in mode 2. FIG. 11 is a diagram showing an example of an image formed on the imaging surface 21a via the imaging optical system 11 (see FIG. 2). FIG. 12 is a diagram showing an example of an event image at a predetermined timing. FIG. 13 is a diagram showing an example of the reciprocal of the number of detections. The whiter the color, the higher the value. FIG. 14 is a diagram showing an example of a gradation image generated by the image generation unit 132. In the event image shown in FIG. 12, image information for when no event has occurred cannot be obtained from the incident light incident via the imaging optical system 11. For this reason, it is difficult to distinguish, for example, an intruder who is stationary or hiding in the shadows. Furthermore, it is difficult to grasp the detailed information of the example image shown in FIG. 11.

[0086] In contrast, the grayscale image shown in Fig. 14 allows for grasping detailed information at the same level as the example image shown in Fig. 11. For example, the shading of clouds is also reproduced. This makes it possible to obtain image information even when no event has occurred, and to obtain detailed information during monitoring.

[0087] 15A and 15B are diagrams schematically illustrating the characteristics of the gradation processing example. FIG. 15A is a diagram similar to FIG. 8. The horizontal axis represents time, and the vertical axis represents the amount of irradiation onto the imaging surface 21a. The detection signal as represents the generation timing of a detection signal at a pixel 30a where light is incident on the imaging surface 21a via the imaging optical system 11 (see FIG. 2) and is subjected to a high amount of irradiation. The detection signal bs represents the generation timing of a detection signal at a pixel 30b where light is incident on the imaging surface 21a via the imaging optical system 11 (see FIG. 2) and is subjected to a low amount of irradiation.

[0088] FIG. 15(b) shows a case where the time taken for the monotonically increasing from the first irradiation amount L1 to the second irradiation amount L2 is longer than that shown in FIG. 15(a). FIG. 15(c) shows a case where the time taken for the monotonically increasing from the first irradiation amount L1 to the second irradiation amount L2 is longer than that shown in FIG. 15(b). The detection threshold in the address event detection unit 33 is set to 25 percent, which is the ratio of the current irradiation amount to the current irradiation amount. Therefore, the detection signals as and bs are generated each time the irradiation amount becomes 1.25 times larger. As can be seen from these figures, when the detection signals as and bs monotonically increase from the first irradiation amount L1 to the second irradiation amount L2, a decrease in accuracy is suppressed even if the time taken for the monotonically increasing from the first irradiation amount L1 to the second irradiation amount L2 is longer. Furthermore, when the time taken for the monotonically increasing from the first irradiation amount L1 to the second irradiation amount L2 can be maintained longer, a decrease in accuracy of the detection signals as and bs is suppressed even if the irradiation amount increases nonlinearly.

[0089] 15A and 15B are diagrams schematically illustrating the characteristics of the gradation processing example. FIG. 15A is a diagram similar to FIG. 8. The horizontal axis represents time, and the vertical axis represents the amount of irradiation onto the imaging surface 21a. The detection signal as represents the generation timing of a detection signal at a pixel 30a where light is incident on the imaging surface 21a via the imaging optical system 11 (see FIG. 2) and is subjected to a high amount of irradiation. The detection signal bs represents the generation timing of a detection signal at a pixel 30b where light is incident on the imaging surface 21a via the imaging optical system 11 (see FIG. 2) and is subjected to a low amount of irradiation.

[0090] FIG. 15(b) shows a case where the time taken for the monotonically increasing from the first irradiation amount L1 to the second irradiation amount L2 is longer than that shown in FIG. 15(a). FIG. 15(c) shows a case where the time taken for the monotonically increasing from the first irradiation amount L1 to the second irradiation amount L2 is longer than that shown in FIG. 15(b). The detection threshold in the address event detection unit 33 is set to 25 percent, which is the ratio of the current irradiation amount to the current irradiation amount. Therefore, the detection signals as and bs are generated each time the irradiation amount becomes 1.25 times larger. As can be seen from these figures, when the detection signals as and bs monotonically increase from the first irradiation amount L1 to the second irradiation amount L2, a decrease in accuracy is suppressed even if the time taken for the monotonically increasing from the first irradiation amount L1 to the second irradiation amount L2 is longer. Furthermore, when the time taken for the monotonically increasing from the first irradiation amount L1 to the second irradiation amount L2 can be maintained longer, a decrease in accuracy of the detection signals as and bs is suppressed even if the irradiation amount increases nonlinearly.

[0091] (Driving to reduce arbitration time) When the occurrence frequency of the address event detection unit 33 increases, arbitration in the arbiter unit 23 (see FIG. 4) may take time. For this reason, the imaging device 2 according to this embodiment has a light source driving mode in which the light sources 121 to 124 are turned on in sequence.

[0092] 16 is a plan view showing an example configuration of the plurality of light source units 121 to 124. The imaging device 2 has the plurality of light source units 121 to 124. In the light source drive mode, the light source unit 121 is turned on, and the irradiation amount is increased to the maximum light amount. Next, the light source unit 122 is turned on, and the irradiation amount is increased to the maximum light amount. Next, the light source unit 123 is turned on, and the irradiation amount is increased to the maximum light amount. Next, the light source unit 124 is turned on, and the irradiation amount is increased to the maximum light amount. As a result, the irradiation amount on the imaging surface 21a increases monotonically overall from the first irradiation amount L1 to the second irradiation amount L2, while the change in irradiation amount for each pixel 30 varies.

[0093] At this time, the occurrence frequency of the address event detection unit 33 first increases in the irradiation range of the light source unit 121, then increases in the irradiation range of the light source unit 122, then increases in the irradiation range of the light source unit 123, and then increases in the irradiation range of the light source unit 124. As a result, the arbitration occurrence region in the arbiter unit 23 (see FIG. 4) is dispersed when the irradiation amount monotonically increases from the first irradiation amount L1 to the second irradiation amount L2. Therefore, the arbitration time in the arbiter unit 23 (see FIG. 4) is shortened, and even if the time for monotonically increasing from the first irradiation amount L1 to the second irradiation amount L2 is further shortened, a decrease in the detection accuracy of the detection signals as and bs is suppressed.

[0094] Fig. 17 is a flowchart showing an example of generation of a gradation image by the imaging device 2. As shown in Fig. 16, the light source control unit 131 of the control unit 14 monotonically increases or decreases the light irradiated onto the imaging surface 21a from the plurality of light source units 12 while light is incident on the imaging surface 21a via the imaging optical system 11 (step S100).

[0095] Next, the mode control unit 130 of the control unit 14 causes the recording unit 13 to record the detection signals output from each pixel 30 (step S102). Then, the image generation unit 132 of the control unit 14 generates a gradation image based on the number of detection signals of each pixel 30 recorded in the recording unit 13 (step S104), and the gradation generation process ends.

[0096] As described above, the imaging device 2 according to this embodiment monotonically increases or decreases the amount of light irradiated onto the imaging surface 21 a from the plurality of light source units 12 while light is incident on the imaging surface 21 a via the imaging optical system 11, photoelectrically converts the irradiated light in the photoelectric conversion elements of the pixels 30 to generate electrical signals, and generates a gradation image based on the number of detection signals generated when the absolute value of the amount of change in the electrical signals exceeds a threshold value. As a result, the number of detection signals changes according to the amount of light incident on the imaging surface 21 a via the imaging optical system 11, and conversion into a luminance signal according to the number of detection signals becomes possible.

[0097] Second Embodiment An imaging device 2 according to a second embodiment differs from the imaging device 2 according to the first embodiment in that the light source unit is capable of irradiating the imaging surface 21 a with light of multiple wavelengths. The differences from the imaging device 2 according to the first embodiment will be described below.

[0098] 18 is a block diagram showing an example of the configuration of the light source unit 12 according to the second embodiment. The light source unit 12 includes a red LED light source 120r, a green LED light source 120g, a blue LED light source 120b, and a light source drive unit 125. The red LED light source 120r irradiates the imaging surface 21a with red light having a first wavelength. The green LED light source 120g irradiates the imaging surface 21a with green light having a second wavelength. The blue LED light source 120b irradiates the imaging surface 21a with blue light having a third wavelength. The light source drive unit 125 can independently control the irradiation amount of the green LED light source 120g, the blue LED light source 120b, and the light source drive unit 125.

[0099] FIG. 19 is a diagram showing an example of generation of an address event detection signal by the address event detection unit 33 (see FIG. 6) of the pixel 30. The horizontal axis represents time, and the vertical axis represents the amount of irradiation onto the imaging surface 21a. In FIG. 19(a), the detection signal rs represents the generation timing of the detection signal when the amount of irradiation from the red LED light source 120r is simply increased while light is incident on the imaging surface 11a via the imaging optical system 11 (see FIG. 2). In FIG. 19(b), the detection signal gs represents the generation timing of the detection signal when the amount of irradiation from the green LED light source 120g is simply increased while light is incident on the imaging surface 11a via the imaging optical system 11 (see FIG. 2). In FIG. 19(c), the detection signal bs represents the generation timing of the detection signal when the amount of irradiation from the blue LED light source 120b is simply increased while light is incident on the imaging surface 11a via the imaging optical system 11 (see FIG. 2).

[0100] As shown in Figure 19, the amounts of red light, green light, and blue light irradiated onto the pixel 30 are different. Therefore, as shown in Figures 19(a) to 19(c), by changing the light irradiated onto the imaging surface 21a to red light, green light, and blue light, the number of detection signals rs, gs, and bs detected varies. As can be seen from this, the image generation unit 132 (see Figure 7) can generate image data corresponding to R, G, and B based on the number of detection signals rs, gs, and bs detected. The display unit 16 (see Figure 1) can then display a color image using the image data corresponding to R, G, and B.

[0101] As described above, in the imaging device 2 according to this embodiment, when light is incident on the imaging surface 11a via the imaging optical system 11 (see FIG. 2), the red LED light source 120r, the green LED light source 120g, and the blue LED light source 120b of the light source unit 12 monotonically increase or decrease their irradiation amounts in that order, and the image generation unit 132 generates image data based on the number of detection signals rs, gs, and bs detected. The number of detection signals rs, gs, and bs detected changes depending on the irradiation amount of the color components irradiated on the imaging surface 11a, so it is possible to obtain luminance values ​​corresponding to the irradiation amounts of red light, green light, and blue light incident on the imaging surface 11a.

[0102] <Application Examples of the Technology According to the Present Disclosure> The technology according to the present disclosure can be applied to various products. More specific application examples are described below. For example, the technology according to the present disclosure may be realized as a distance measuring device mounted on any type of moving body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot, construction machinery, or agricultural machinery (tractor).

[0103] [Mobile Body] Fig. 20 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile body control system to which the technology according to the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected via a communication network 7010. In the example shown in Fig. 20, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an outside-vehicle information detection unit 7400, an inside-vehicle information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these multiple control units may be an in-vehicle communication network conforming to any standard, such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), or FlexRay (registered trademark).

[0104] Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a memory unit that stores the programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various controlled devices. Each control unit includes a network I / F for communicating with other control units via a communication network 7010, and a communication I / F for communicating with devices or sensors inside and outside the vehicle via wired or wireless communication. Figure 20 illustrates the functional configuration of the integrated control unit 7600, including a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, an audio / video output unit 7670, an in-vehicle network I / F 7680, and a memory unit 7690. The other control units also include a microcomputer, a communication I / F, a memory unit, and the like.

[0105] The drivetrain control unit 7100 controls the operation of devices related to the drivetrain of the vehicle according to various programs. For example, the drivetrain control unit 7100 functions as a control device for a drive force generating device for generating drive force for the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating braking force for the vehicle. The drivetrain control unit 7100 may also function as a control device for an ABS (Antilock Brake System) or an ESC (Electronic Stability Control), etc.

[0106] A vehicle state detection unit 7110 is connected to the drivetrain control unit 7100. The vehicle state detection unit 7110 includes at least one of a gyro sensor that detects the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, or a sensor that detects the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine rotation speed, the rotation speed of the wheels, etc. The drivetrain control unit 7100 performs arithmetic processing using signals input from the vehicle state detection unit 7110, and controls the internal combustion engine, the drive motor, the electric power steering device, the brake device, etc.

[0107] The body system control unit 7200 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 7200. The body system control unit 7200 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0108] The battery control unit 7300 controls the secondary battery 7310, which is the power supply source for the drive motor, in accordance with various programs. For example, information such as battery temperature, battery output voltage, or remaining battery capacity is input to the battery control unit 7300 from a battery device equipped with the secondary battery 7310. The battery control unit 7300 performs arithmetic processing using these signals, and controls the temperature regulation of the secondary battery 7310 or a cooling device or the like equipped in the battery device.

[0109] The outside vehicle information detection unit 7400 detects information outside the vehicle equipped with the vehicle control system 7000. For example, at least one of an imaging unit 7410 and an outside vehicle information detection unit 7420 is connected to the outside vehicle information detection unit 7400. The imaging unit 7410 includes at least one of a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The outside vehicle information detection unit 7420 includes at least one of an environmental sensor for detecting the current weather or climate, or a surrounding information detection sensor for detecting other vehicles, obstacles, pedestrians, etc. around the vehicle equipped with the vehicle control system 7000.

[0110] The environmental sensor may be, for example, at least one of a raindrop sensor that detects rain, a fog sensor that detects fog, a sunshine sensor that detects the degree of sunshine, and a snow sensor that detects snowfall. The surrounding information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device. The imaging unit 7410 and the outside vehicle information detection unit 7420 may be provided as independent sensors or devices, or may be provided as a device in which multiple sensors or devices are integrated.

[0111] 21 shows an example of the installation positions of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging units 7910, 7912, 7914, 7916, and 7918 are provided, for example, at least one of the front nose, side mirrors, rear bumper, back door, and upper part of the windshield inside the vehicle cabin of the vehicle 7900. The imaging unit 7910 provided on the front nose and the imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 provided on the side mirrors mainly acquire images of the sides of the vehicle 7900. The imaging unit 7916 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin is mainly used to detect leading vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0112] 21 shows an example of the imaging ranges of the imaging units 7910, 7912, 7914, and 7916. Imaging range a indicates the imaging range of the imaging unit 7910 provided on the front nose, imaging ranges b and c indicate the imaging ranges of the imaging units 7912 and 7914 provided on the side mirrors, respectively, and imaging range d indicates the imaging range of the imaging unit 7916 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 7910, 7912, 7914, and 7916, a bird's-eye view image of the vehicle 7900 viewed from above can be obtained.

[0113] The outside vehicle information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided on the front, rear, sides, corners, and above the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, ultrasonic sensors or radar devices. The outside vehicle information detection units 7920, 7926, and 7930 provided on the front nose, rear bumper, back door, and above the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, LIDAR devices. These outside vehicle information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, etc.

[0114] Returning to FIG. 20 , the explanation will be continued. The outside-vehicle information detection unit 7400 causes the imaging unit 7410 to capture an image outside the vehicle and receives the captured image data. The outside-vehicle information detection unit 7400 also receives detection information from the connected outside-vehicle information detection unit 7420. If the outside-vehicle information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the outside-vehicle information detection unit 7400 emits ultrasonic waves or electromagnetic waves and receives information on the received reflected waves. Based on the received information, the outside-vehicle information detection unit 7400 may perform object detection processing or distance detection processing for people, vehicles, obstacles, signs, text on the road, etc. Based on the received information, the outside-vehicle information detection unit 7400 may also perform environment recognition processing for recognizing rainfall, fog, road conditions, etc. Based on the received information, the outside-vehicle information detection unit 7400 may also calculate the distance to an object outside the vehicle.

[0115] The outside vehicle information detection unit 7400 may also perform image recognition processing or distance detection processing to recognize people, vehicles, obstacles, signs, or characters on the road surface based on the received image data. The outside vehicle information detection unit 7400 may perform processing such as distortion correction or alignment on the received image data, and may also generate an overhead image or a panoramic image by combining image data captured by different image capturing units 7410. The outside vehicle information detection unit 7400 may also perform viewpoint conversion processing using image data captured by different image capturing units 7410.

[0116] The interior information detection unit 7500 detects information inside the vehicle. A driver state detection unit 7510 that detects the driver's state is connected to the interior information detection unit 7500, for example. The driver state detection unit 7510 may include a camera that captures an image of the driver, a biosensor that detects the driver's biometric information, or a microphone that collects sound from within the vehicle cabin. The biosensor is provided, for example, on the seat or steering wheel, and detects the biometric information of a passenger sitting in the seat or the driver gripping the steering wheel. The interior information detection unit 7500 may calculate the driver's level of fatigue or concentration based on the detection information input from the driver state detection unit 7510, or may determine whether the driver is dozing off. The interior information detection unit 7500 may perform processing such as noise canceling on the collected audio signal.

[0117] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 in accordance with various programs. An input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 is realized by a device that can be operated by a passenger, such as a touch panel, a button, a microphone, a switch, or a lever. Data obtained by voice recognition of voice input through a microphone may be input to the integrated control unit 7600. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile phone or a personal digital assistant (PDA) that is compatible with the operation of the vehicle control system 7000. The input unit 7800 may be, for example, a camera, in which case the passenger can input information using gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the passenger may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by a passenger or the like using the input unit 7800 and outputs the signal to the integrated control unit 7600. The passenger or the like operates the input unit 7800 to input various data to the vehicle control system 7000 and to instruct processing operations.

[0118] The storage unit 7690 may include a ROM (Read Only Memory) that stores various programs executed by the microcomputer, and a RAM (Random Access Memory) that stores various parameters, calculation results, sensor values, etc. The storage unit 7690 may also be realized by a magnetic storage device such as a HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0119] The general-purpose communication I / F 7620 is a general-purpose communication I / F that mediates communication with various devices present in the external environment 7750. The general-purpose communication I / F 7620 may implement a cellular communication protocol such as GSM (Registered Trademark) (Global System of Mobile communications), WiMAX, LTE (Long Term Evolution), or LTE-A (LTE-Advanced), or other wireless communication protocols such as a wireless LAN (also referred to as Wi-Fi (Registered Trademark)) or Bluetooth (Registered Trademark). The general-purpose communication I / F 7620 may connect to a device (e.g., an application server or a control server) present on an external network (e.g., the Internet, a cloud network, or an operator-specific network) via, for example, a base station or an access point. In addition, the general-purpose communication I / F 7620 may connect to a terminal located near the vehicle (for example, a terminal of a driver, a pedestrian, or a store, or an MTC (Machine Type Communication) terminal) using, for example, P2P (Peer To Peer) technology.

[0120] The dedicated communication I / F 7630 is a communication I / F that supports a communication protocol designed for use in a vehicle. The dedicated communication I / F 7630 may implement a standard protocol such as WAVE (Wireless Access in Vehicle Environment), which is a combination of IEEE 802.11p at a lower layer and IEEE 1609 at an upper layer, DSRC (Dedicated Short Range Communications), or a cellular communication protocol. The dedicated communication I / F 7630 typically performs V2X communication, which is a concept including one or more of vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0121] The positioning unit 7640 performs positioning by receiving, for example, GNSS signals from GNSS (Global Navigation Satellite System) satellites (for example, GPS signals from GPS (Global Positioning System) satellites) and generates position information including the latitude, longitude, and altitude of the vehicle. Note that the positioning unit 7640 may identify the current position by exchanging signals with a wireless access point, or may obtain position information from a terminal such as a mobile phone, PHS, or smartphone that has a positioning function.

[0122] The beacon receiving unit 7650 receives, for example, radio waves or electromagnetic waves transmitted from radio stations or the like installed on the road, and acquires information such as the current location, congestion, road closures, required travel time, etc. The function of the beacon receiving unit 7650 may be included in the dedicated communication I / F 7630 described above.

[0123] The in-vehicle device I / F 7660 is a communication interface that mediates connections between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 may establish a wireless connection using a wireless communication protocol such as wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). The in-vehicle device I / F 7660 may also establish a wired connection using a protocol such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or MHL (Mobile High-Definition Link) via a connection terminal (and a cable, if necessary) not shown. The in-vehicle devices 7760 may include, for example, at least one of a mobile device or a wearable device owned by a passenger, or an information device carried into or attached to the vehicle. The in-vehicle devices 7760 may also include a navigation device that searches for a route to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0124] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals in accordance with a predetermined protocol supported by the communication network 7010.

[0125] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 in accordance with various programs based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. For example, the microcomputer 7610 may calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the acquired information inside and outside the vehicle, and output control commands to the drivetrain control unit 7100. For example, the microcomputer 7610 may perform cooperative control aimed at realizing functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the following distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc. In addition, the microcomputer 7610 may perform cooperative control for the purpose of autonomous driving, in which the vehicle travels autonomously without relying on driver operation, by controlling a driving force generating device, a steering mechanism, a braking device, etc. based on information acquired about the vehicle's surroundings.

[0126] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people, and create local map information including information about the vicinity of the vehicle's current location, based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. Furthermore, the microcomputer 7610 may predict dangers, such as a vehicle collision, the approach of a pedestrian, or entry into a closed road, based on the acquired information, and generate a warning signal. The warning signal may be, for example, a signal for generating a warning sound or turning on a warning lamp.

[0127] The audio / video output unit 7670 transmits at least one audio and / or visual output signal to an output device capable of visually or audibly notifying the vehicle occupants or the outside of the vehicle of information. In the example of FIG. 20 , an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are illustrated as examples of the output devices. The display unit 7720 may include, for example, at least one of an on-board display and a head-up display. The display unit 7720 may have an AR (Augmented Reality) display function. The output device may be other devices besides these devices, such as headphones, a wearable device such as an eyeglass-type display worn by the occupant, a projector, or a lamp. When the output device is a display device, the display device visually displays results obtained by various processes performed by the microcomputer 7610 or information received from other control units in various formats, such as text, images, tables, and graphs. Furthermore, when the output device is an audio output device, the audio output device converts an audio signal consisting of reproduced audio data or acoustic data into an analog signal and outputs it audibly.

[0128] In the example shown in FIG. 20 , at least two control units connected via the communication network 7010 may be integrated into a single control unit. Alternatively, each control unit may be composed of multiple control units. Furthermore, the vehicle control system 7000 may include another control unit not shown. In the above description, some or all of the functions performed by one of the control units may be assigned to another control unit. In other words, as long as information is transmitted and received via the communication network 7010, predetermined arithmetic processing may be performed by one of the control units. Similarly, a sensor or device connected to one of the control units may be connected to another control unit, and multiple control units may transmit and receive detection information to each other via the communication network 7010.

[0129] The above describes an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to, for example, the image capture units 7910, 7912, 7914, 7916, and 7918, the outside vehicle information detection units 7920, 7922, 7924, 7926, 7928, and 7930, and the driver state detection unit 7510, among the above-described components. Specifically, the image capture system 1 shown in FIG. 1 , which includes the image capture device according to the present disclosure, can be applied to these image capture units and detection units. Furthermore, by applying the technology according to the present disclosure, the effects of noise events such as sensor noise can be mitigated, and the occurrence of a true event can be reliably and quickly detected, thereby enabling safe vehicle driving.

[0130] The present technology can be configured as follows:

[0131] (1) An imaging device comprising: a photoelectric conversion unit having a plurality of photoelectric conversion elements that each photoelectrically converts light irradiated at different positions on a two-dimensional imaging surface to generate an electrical signal; a light source unit that irradiates the imaging surface with the irradiating light; and a first detection unit that detects a detection signal when a change in the electrical signal generated by the plurality of photoelectric conversion elements exceeds a predetermined value.

[0132] (2) The imaging device according to (1), further comprising an imaging optical system that directs light onto the imaging surface.

[0133] (3) The imaging device according to (2), wherein the light source unit monotonically increases or monotonically decreases the amount of illumination light irradiated onto the imaging surface while light is incident on the imaging surface via the imaging optical system.

[0134] (4) The imaging device according to (2) or (3), further comprising: an image generating unit that generates image data in which values ​​relating to the number of times of detection signals detected by the first detecting unit are arranged in two-dimensional coordinates.

[0135] (5) The imaging device according to (4), further comprising a light source driving unit that controls the light source unit.

[0136] (6) The imaging device according to (5), wherein the light source driving unit monotonically increases the irradiation amount of light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount.

[0137] (7) The imaging device according to (6), wherein the light source driving unit monotonically decreases the amount of irradiation light irradiated onto the imaging surface from the second irradiation amount to the first irradiation amount.

[0138] (8) The imaging device according to (7), wherein the first detection unit detects the detection signal when a change amount of the electrical signal exceeds a predetermined ratio.

[0139] (9) The imaging device according to (8), wherein the image generation unit generates, as the image data, a pixel value corresponding to a value obtained by raising the number of times the detection signal is detected by the first detection unit to a power of a numerical value based on the ratio.

[0140] (10) The imaging device according to any one of (1) to (9), wherein the light source unit has a plurality of light sources, and the light source driving unit monotonically increases or monotonically decreases the light sources in different time ranges.

[0141] (11) The imaging device described in (10), wherein the light source unit has a first light source of a first wavelength and a second light source of a second wavelength different from the first wavelength, the light source driving unit monotonically increases the irradiation light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount using the irradiation light of the first light source, and the image generating unit generates the image data as first image data.

[0142] (12) The imaging device described in (11), wherein, after generating the first image data, the light source driving unit monotonically increases the amount of irradiation light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount using irradiation light from the second light source, and the image generating unit generates the image data as second image data.

[0143] (13) The imaging device described in (12), wherein the light source unit further has a third light source of a third wavelength different from the first wavelength and the second wavelength, and after generating the second image data, the light source driving unit monotonically increases the irradiation light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount using the irradiation light of the third light source, and the image generating unit generates the image data as third image data.

[0144] (14) The imaging device according to (13), wherein the image generation unit generates color image data based on the first image data, the second image data, and the third image data.

[0145] (15) The imaging device according to (14), wherein the first to third light sources are any one of a red LED, a blue LED, and a green LED.

[0146] (16) The imaging device according to any one of (1) to (15), wherein the first detection unit detects the detection signal when an absolute value of an amount of change in a direction in which the signal level of the electrical signal increases exceeds a first threshold, and detects the detection signal when an absolute value of an amount of change in a direction in which the signal level of the electrical signal decreases exceeds a second threshold.

[0147] (17) The imaging device according to any one of (2) to (15), further comprising a control unit that switches between a first mode in which the light source of the light source unit is turned off or the amount of light is maintained constant and an image is generated based on the detection signal, and a second mode in which the light source of the light source unit is turned on and, while light is incident on the imaging surface via the imaging optical system, the amount of irradiation light irradiated onto the imaging surface is monotonically increased or monotonically decreased and an image is generated based on the number of detection signals.

[0148] (18) The imaging device according to (17), further including a third mode in which, in the second mode, an image based on the detection signal is not generated.

[0149] (19) An imaging method comprising: monotonically increasing or monotonically decreasing the amount of irradiated light onto an imaging surface while light is incident on the imaging surface via an imaging optical system; photoelectrically converting the irradiated light using a plurality of photoelectric conversion elements to generate electrical signals; and detecting the number of times a detection signal is generated when the absolute value of the amount of change in the electrical signals exceeds a predetermined threshold.

[0150] (20) An imaging system comprising: an imaging device; and a display unit that displays an image generated by the imaging device, wherein the imaging device comprises: a photoelectric conversion unit having a plurality of photoelectric conversion elements that each photoelectrically converts light irradiated at different positions on a two-dimensional imaging surface to generate an electrical signal; a light source unit that irradiates the imaging surface with irradiation light; and a first detection unit that detects a detection signal when an amount of change in the electrical signal generated by the plurality of photoelectric conversion elements exceeds a predetermined value.

[0151] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0152] 1: imaging system, 2: imaging device, 10: second light source unit, 11: imaging optical system, 12: light source unit, 14: control unit, 15: operation unit, 16: display unit, 20: imaging unit, 21a: imaging surface, 33: address event detection unit (first detection unit), 40: control unit, 100: second light source driving unit, 120b: blue LED light source, 120g: green LED light source, 120r: ​​red LED light source, 120w: LED light source, 125: light source driving unit, 130: mode control unit, 131: light source control unit, 132: image generation unit, 311: photoelectric conversion element.

Claims

1. An imaging device comprising: a photoelectric conversion unit having a plurality of photoelectric conversion elements that each photoelectrically converts light irradiated to different positions on a two-dimensional imaging surface to generate an electrical signal; a light source unit that irradiates the imaging surface with irradiating light; and a first detection unit that detects a detection signal when a change in the electrical signal generated by the plurality of photoelectric conversion elements exceeds a predetermined value.

2. The imaging device according to claim 1, further comprising an imaging optical system for directing light onto the imaging surface.

3. The imaging device according to claim 2, wherein the light source unit monotonically increases or decreases the amount of light irradiated onto the imaging surface when light is incident on the imaging surface via the imaging optical system.

4. The imaging device according to claim 2, further comprising an image generating unit that generates image data in which values ​​relating to the number of times a detection signal is detected by the first detection unit are arranged in two-dimensional coordinates.

5. The imaging device according to claim 4, further comprising a light source driving section for controlling the light source section.

6. The imaging device according to claim 5, wherein the light source driving section monotonically increases the amount of light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount.

7. The imaging device according to claim 6, wherein the light source driving section monotonically decreases the amount of light irradiated onto the imaging surface from the second irradiation amount to the first irradiation amount.

8. The imaging device according to claim 7, wherein the first detection unit detects the detection signal when an amount of change in the electrical signal exceeds a predetermined ratio.

9. The imaging device according to claim 8, wherein the image generation unit generates, as the image data, a pixel value corresponding to a value obtained by raising the number of times the detection signal is detected by the first detection unit to a power of a numerical value based on the ratio.

10. The imaging device according to claim 9, wherein the light source section has a plurality of light sources, and the light source driving section causes the plurality of light sources to monotonically increase or decrease in different time ranges.

11. The imaging device described in claim 10, wherein the light source unit has a first light source of a first wavelength and a second light source of a second wavelength different from the first wavelength, the light source driving unit monotonically increases the amount of irradiation light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount using the irradiation light of the first light source, and the image generating unit generates the image data as first image data.

12. The imaging device described in claim 11, wherein after generating the first image data, the light source driving unit monotonically increases the amount of light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount using light from the second light source, and the image generating unit generates the image data as second image data.

13. The imaging device described in claim 12, wherein the light source unit further has a third light source having a third wavelength different from the first wavelength and the second wavelength, and after generating the second image data, the light source driving unit monotonically increases the irradiation light irradiated onto the imaging surface from a first irradiation amount to a second irradiation amount using the irradiation light of the third light source, and the image generating unit generates the image data as third image data.

14. The imaging device according to claim 13, wherein the image generating section generates color image data based on the first image data, the second image data, and the third image data.

15. The imaging device according to claim 14, wherein the first, second and third light sources are any one of a red LED, a blue LED and a green LED.

16. The imaging device of claim 1, wherein the first detection unit detects the detection signal when an absolute value of an amount of change in a direction in which the signal level of the electrical signal increases exceeds a first threshold, and detects the detection signal when an absolute value of an amount of change in a direction in which the signal level of the electrical signal decreases exceeds a second threshold.

17. The imaging device of claim 2, further comprising a control unit that switches between a first mode in which the light source of the light source unit is turned off or the amount of light is maintained at a constant level and an image is generated based on the detection signal, and a second mode in which the light source of the light source unit is turned on and, while light is incident on the imaging surface via the imaging optical system, the amount of light irradiated onto the imaging surface is monotonically increased or decreased to generate an image based on the number of detection signals.

18. The imaging device according to claim 17, further comprising a third mode in which, in the second mode, no image is generated based on the detection signal.

19. An imaging method comprising the steps of: while light is incident on an imaging surface via an imaging optical system, monotonically increasing or decreasing the amount of irradiated light onto the imaging surface; photoelectrically converting the irradiated light using a plurality of photoelectric conversion elements to generate an electrical signal; and detecting the number of times a detection signal is generated when the absolute value of the amount of change in the electrical signal exceeds a predetermined threshold value.

20. An imaging system comprising: an imaging device; and a display unit that displays an image generated by the imaging device, wherein the imaging device has a photoelectric conversion unit having a plurality of photoelectric conversion elements that each photoelectrically converts light irradiated to different positions on a two-dimensional imaging surface to generate an electrical signal; a light source unit that irradiates the imaging surface with irradiating light; and a first detection unit that detects a detection signal when an amount of change in the electrical signal generated by the plurality of photoelectric conversion elements exceeds a predetermined value.

Images