Imaging element and imaging method

Abstract

The present invention relates to an imaging element and an imaging method that make it possible to assign a highly reliable timestamp. The present invention is provided with: a pixel array unit in which a plurality of pixels for detecting an event are arranged in a matrix; an arbiter that arbitrates requests from the pixels; a difference computation unit that computes the difference between an occurrence time at which an event occurred at a pixel and a read time at which the event is read; and a timestamp unit that assigns a timestamp. The present invention can be applied to an arbiter-type event-based vision sensor (EVS) for detecting events.

Description

Image sensor and imaging method

[0001] The present technology relates to an imaging element and an imaging method, and more particularly to an imaging element and an imaging method that are capable of supplying event data to which highly reliable information is added, for example.

[0002] While an image sensor is a sensor for acquiring an image including a subject, an event sensor (EVS: Event-based Vision Sensor) is a sensor for detecting changes in the subject. By limiting the sensing target from "images" to "changes in the subject," an event sensor can set a higher frame rate than an image sensor. Like an image sensor, an event sensor is realized by a solid-state imaging device such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0003] Japanese Patent Application Laid-Open No. 2020-136811

[0004] Since an arbiter-type EVS processes events in the order in which they are fired, if multiple events fire simultaneously, there is a possibility that a processing wait time will occur while these events are arbitrated. In this case, there is a possibility that an error due to the wait time will occur between the timing when an event occurs and the timing when a timestamp is assigned to the event, which may reduce the reliability of the timestamp.

[0005] The present technology has been made in view of such circumstances, and makes it possible to provide highly reliable information.

[0006] An imaging element according to one aspect of the present technology is an imaging element including: a pixel array unit in which a plurality of pixels that detect events are arranged in a matrix; an arbiter that arbitrates requests from the pixels; a difference calculation unit that calculates the difference between the occurrence time when the event occurs in the pixel and the read time when the event is read; and a timestamp unit that assigns a timestamp.

[0007] An imaging method according to one aspect of the present technology is an imaging method in which an imaging element includes a pixel array unit in which a plurality of pixels are arranged in a matrix, an arbiter, a difference calculation unit provided between the pixel array unit and the arbiter, and an output unit, wherein the pixel in the pixel array unit detects an event, the difference calculation unit calculates the difference between the occurrence time at which the event occurred in the pixel and the read time at which the event is read, and the output unit outputs the difference and an assigned timestamp.

[0008] An imaging element according to one aspect of the present technology includes a pixel array unit in which a plurality of pixels for detecting an event are arranged in a matrix, an arbiter that arbitrates requests from the pixels, a difference calculation unit that calculates the difference between the occurrence time when the event occurs in the pixel and the read time when the event is read, and a timestamp unit that assigns a timestamp.

[0009] In an imaging method according to one aspect of the present technology, an event is detected by a pixel of the pixel array section of an imaging element that includes a pixel array section in which a plurality of pixels are arranged in a matrix, an arbiter, a difference calculation section that is provided between the pixel array section and the arbiter, and an output section, and the difference between the occurrence time when the event occurred in the pixel and the read time when the event is read is calculated by the difference calculation section, and the timestamp that is assigned and the difference are output by the output section.

[0010] The imaging device may be an independent device or an internal block constituting a single device.

[0011] Fig. 1 is a block diagram showing a configuration of an embodiment of an imaging device to which the present technology is applied. Fig. 2 is a block diagram showing a schematic configuration example of an imaging element. Fig. 3 is a diagram showing a detailed configuration example of an imaging element. Fig. 4 is a diagram showing a configuration example of a pixel array unit. Fig. 5 is a diagram for explaining data to be transmitted and received. Fig. 6 is a diagram for explaining a format of event data. Fig. 7 is a timing chart for explaining the operation of the imaging element. Fig. 8 is a block diagram showing an example of a schematic configuration of a vehicle control system. Fig. 9 is an explanatory diagram showing an example of installation positions of an outside vehicle information detection unit and an imaging unit.

[0012] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described.

[0013] <Configuration Example of Imaging Device> The imaging device 10 is a camera equipped with an event sensor that outputs, as event data, a temporal change in an electrical signal obtained by photoelectrically converting an optical signal. Such an event sensor is also called an EVS (event-based vision sensor). A camera equipped with a general image sensor captures images in synchronization with a vertical synchronization signal and outputs frame data, which is image data for one frame (screen), at the period of the vertical synchronization signal. However, the imaging device 10 outputs event data only when an event occurs, and therefore can be said to be an asynchronous or address-controlled camera.

[0014] The imaging device 10 shown in FIG. 1 includes an optical unit 11, an imaging element 12, a control unit 13, and a data processing unit 14.

[0015] The optical unit 11 collects light from a subject and makes it incident on the image sensor 12. The image sensor 12 photoelectrically converts the incident light that has entered through the optical unit 11 to generate event data, and supplies the event data to the data processing unit 14. The image sensor 12 is a light-receiving element that regards a change in pixel luminance as an event and outputs event data that indicates the occurrence of an event.

[0016] The control unit 13 controls the imaging element 12. For example, the control unit 13 instructs the imaging element 12 to start and end imaging.

[0017] The data processing unit 14 is configured with, for example, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a microprocessor, etc., and executes predetermined processing. The data processing unit 14 includes an event data processing unit 21 and a recording unit 22. For example, the event data processing unit 21 performs event data processing using event data supplied from the image sensor 12, and image data processing using event images. The recording unit 22 records and accumulates the predetermined data on a predetermined recording medium as needed.

[0018] 2 is a block diagram showing a schematic configuration example of the image sensor 12. The image sensor 12 includes a pixel array unit 41, a drive unit 42, an arbiter 43, a difference calculation unit 44, and an output unit 45.

[0019] A plurality of pixels 61 are arranged in a two-dimensional lattice pattern in the pixel array section 41. Each pixel 61 includes a photodiode 71 as a photoelectric conversion element and an address event detection circuit 72.

[0020] The address event detection circuit 72 detects a change in photocurrent as an event when a change exceeding a predetermined threshold occurs in the photocurrent, which is an electrical signal generated by photoelectric conversion of the photodiode 71. When an event is detected, the address event detection circuit 72 outputs a request to the arbiter 43 via the difference calculation unit 44, requesting output of event data indicating the occurrence of the event.

[0021] The drive unit 42 drives the pixel array unit 41 by supplying control signals to each pixel 61 of the pixel array unit 41. The output unit 45 performs necessary processing on the event data output by each pixel 61 constituting the pixel array unit 41, and supplies the data to the data processing unit 14 (FIG. 1).

[0022] Fig. 3 is a diagram showing a more detailed example configuration of the image sensor 12. As shown in Fig. 3, a difference calculation unit 44 is provided for each row of the pixel array unit 41. In the example shown in Fig. 3, the pixel array unit 41 has 25 pixels (5 x 5) and five rows, so that difference calculation units 44-1 to 44-5 are provided for each row.

[0023] Hereinafter, when there is no need to distinguish between the difference calculation units 44-1 to 44-5 individually, or when it is desired to collectively describe the difference calculation units 44-1 to 44-5 as one difference calculation unit (as shown in Figure 2), they will be described as difference calculation unit 44.

[0024] Here, in order to explain the case of reading one row at a time, a difference calculation unit 44 is provided for each row. However, if reading is performed column by column, a difference calculation unit 44 is provided for each column, and if reading is performed in units of a predetermined number of pixels, for example, 16 pixels (4 x 4), a difference calculation unit 44 is provided for each of the predetermined number of pixels.

[0025] The difference calculation unit 44 is provided between the pixel array unit 41 and the arbiter 43. In the following description, the embodiment in which the difference calculation unit 44 is provided between the pixel array unit 41 and the arbiter 43 as shown in Fig. 3 will be described, but the present technology can also be applied to the configuration shown in Fig. 4.

[0026] 4 is a diagram showing another example of the configuration of the image sensor 12, specifically a diagram showing an example of the configuration of the pixel array unit 41. A difference calculation unit 44 is provided for each pixel 61 arranged in the pixel array unit 41. As will be described later, the difference calculation unit 44 calculates the time elapsed from when the pixel 61 issues a request until it receives read approval. The request is issued from the pixel 61, and the read approval is received by the pixel 61. Therefore, a configuration in which a difference calculation unit 44 that calculates the time elapsed from when the request is issued until it receives read approval can be provided for each pixel 61.

[0027] Returning to the description of the configuration of the image sensor 12 shown in Fig. 3, for example, as shown in Fig. 3, when an event occurs in pixel A and a request (Req) is issued, the request is supplied to the difference calculation unit 44-2 connected to the row in which pixel A is arranged. The difference calculation unit 44-2 supplies the supplied Req to the arbiter 43 and starts timing.

[0028] The arbiter 43 outputs a read acknowledge signal (Ack) to pixel A that has issued the request via the difference calculation unit 44. When the difference calculation unit 44 receives the Ack, it stops clocking and outputs the Ack to pixel A. The difference calculation unit 44 starts clocking when the request (Req) is input as a trigger, and stops clocking when the read acknowledge signal (Ack) is input as a trigger.

[0029] The difference calculation unit 44 calculates the difference between the time when an event occurs in the pixel 61 and the time when the data of that event is read out from the pixel 61. The difference calculation unit 44 may be configured to be able to calculate such a difference, and may be configured, for example, as a counter that counts from when a request (Req) is input until when a read acknowledge signal (Ack) is input.

[0030] As the counter, for example, a ripple counter can be used. A ripple counter is a counter that operates by connecting flip-flops in series. A ripple counter operates by providing a pulse to the first stage, so that counter operation can be performed with reduced power consumption. A ripple counter may be used as the difference calculation unit 44, or another counter may be used. The difference calculation unit 44 may be configured as a unit other than a counter as long as it is capable of performing the above-mentioned timekeeping.

[0031] The output unit 45 of the image sensor 12 shown in Figure 3 includes a global timer 51, a timestamp latch 52, a format conversion unit 53, and a latch 54. The latch 54 latches the read event data. The latched event data is supplied to the format conversion unit 53. The format conversion unit 53 converts the event data and other data into a format suitable for output to the downstream data processing unit 14 (Figure 1), and outputs the converted data to the data processing unit 14.

[0032] The time (timestamp) at which the arbiter 43 returned an Ack is supplied to the format conversion unit 53 from the timestamp latch 52. The Ack returned from the arbiter 43 is input to the timestamp latch 52. The timestamp latch 52 is supplied with time data from the global timer 51, and latches the supplied time data at the time when the Ack is input, and supplies the latched time data (timestamp) to the format conversion unit 53.

[0033] The format conversion unit 53 is also supplied with difference data supplied from the difference calculation unit 44 via the selector 46. The selector 46 selects the difference calculation unit 44 to which the Ack returned from the arbiter 43 has been input, and outputs difference data of the difference calculated by the selected difference calculation unit 44 to the format conversion unit 53.

[0034] The format conversion unit 53 is supplied with event data via the latch 54, with differential data via the selector 46, and with a timestamp from the timestamp latch 52. The format conversion unit 53 supplies these pieces of data to the data processing unit 14 ( FIG. 2 ). The data supplied from the image sensor 12 (format conversion unit 53) to the data processing unit 14 will be described below with reference to FIG. 5 .

[0035] 5A, the image sensor 12 supplies event data, timestamps, and difference data to the data processing unit 14. The timestamps are the readout times of the events supplied from the timestamp latch 52. The data processing unit 14 can identify the readout times of the events from the timestamps and subtract the difference represented by the difference data from the timestamps to identify the firing times of the events.

[0036] In the example shown in B of Fig. 5, event data, timestamp, and difference data are supplied from the image sensor 12 to the data processing unit 14. The timestamp is the event firing time. The event firing time is calculated by subtracting the difference time indicated by the difference data supplied from the difference calculation unit 44 via the selector 46 from the event readout time supplied from the timestamp latch 52. The data processing unit 14 can determine the event firing time from the timestamp and add the difference indicated by the difference data to the timestamp, thereby determining the event readout time.

[0037] In the example shown in Fig. 5C, event data, a first timestamp, and a second timestamp are supplied from the image sensor 12 to the data processing unit 14. The first timestamp is the readout time of the event supplied from the timestamp latch 52. The second timestamp is the firing time of the event. The firing time of the event is calculated by subtracting the difference time indicated by the difference data supplied from the difference calculation unit 44 via the selector 46 from the readout time of the event supplied from the timestamp latch 52. The data processing unit 14 can identify the readout time of the event from the first timestamp and the firing time of the event from the second timestamp.

[0038] In any of the cases A to C in FIG. 5, the data processing unit 14 can identify the time when the event is fired and the time when the event is read, thereby improving the reliability of the timestamp.

[0039] For example, in a system where event data is read row by row, the read data for one row is made up of events that occurred somewhere within the range from the occurrence time of the first event to the read time of the event data. In a system where only the firing time or the read time of an event can be identified, only one side of the time range can be identified, which reduces the reliability of the timestamp.

[0040] As described above, this technology can identify the time at which an event is fired and the time at which the event is read, thereby improving the reliability of timestamps. A system capable of obtaining such highly reliable timestamps can be obtained with a configuration such as that shown in Figure 3, which also reduces the circuit configuration resources. For example, to obtain a timestamp, it is conceivable to provide a timer mechanism with a timestamp bit width for each row, but such a configuration would require large circuit resources.

[0041] As shown in FIG. 3 , according to the present technology, it is possible to configure the device as a whole, rather than for each row, with a single timer mechanism, and to configure the device with a difference calculation unit 44 that requires relatively small circuit resources to calculate the time difference. This allows for a significant reduction in circuit resources, and a configuration that can obtain a highly reliable timestamp even with the reduced circuit resources.

[0042] The total amount of data supplied from the image sensor 12 to the data processing unit 14 can also be reduced by including differential data as shown in Fig. 5A and Fig. 5B. A specific format example will now be described with reference to Fig. 6.

[0043] <Data Format Example> FIG. 6 is a diagram showing an example of the frame structure of one frame of event data transmitted from the image sensor 12 (format conversion unit 53) to the data processing unit 14. As shown in FIG.

[0044] As shown in Fig. 6, one frame's worth of event data is stored in multiple long packets arranged in a line between a frame start FS, a short packet indicating the start of the frame, and a frame end FE, a short packet indicating the end of the frame. In the example shown in Fig. 6, a long packet storing embedded data (EBD) is arranged at the beginning of the long packet storing the event data. The long packet has a packet header PH and a packet footer PF.

[0045] As the event data, for example, event polarity information can be used, which indicates positive P for pixels whose brightness values ​​have changed from the reference value to the positive side, and negative N for pixels whose brightness values ​​have changed from the reference value to the negative side. Note that data other than event polarity information may also be used as the event data. When one frame is made up of (N-1) rows, event data E(0) to E(N) are written as shown in FIG. 6.

[0046] A line header LH is added to each event data E. A timestamp and differential data are added to this line header LH. If the line header LH is composed of, for example, 64 bits, the following number of bits can be allocated to add the timestamp and differential data: 10 bits event address information 34 bits timestamp (read time or firing time) 10 bits differential data 10 bits other

[0047] The timestamp is made up of 34 bits, but the difference data is made up of 10 bits. Even when the data processing unit 14 transmits and receives data that can identify both the time when an event was fired and the time when the event was read, it is sufficient to transmit a 34-bit timestamp and 10-bit difference data. Even when the data processing unit 14 transmits and receives data that can identify both the time when an event was fired and the time when the event was read, the amount of data can be reduced.

[0048] <Operation of Image Sensor> The operation of the image sensor 12 shown in FIG. 3 will be further described with reference to FIG.

[0049] At time t1, an event occurs in pixel A, and pixel A outputs a request signal (Req) to the arbiter 43. When pixel A outputs Req, the difference calculation unit 44 starts counting time.

[0050] At time t2, the difference calculation unit 44 stops clocking when it receives a read acknowledge signal (Ack) from the arbiter 43. If the time at which clocking is stopped is taken as time T, this time T is supplied as difference data to the format conversion unit 53 via the selector 46.

[0051] At time t2, when the timestamp latch 52 receives a read acknowledge signal (Ack) from the arbiter 43, it latches the global time supplied from the global timer 51 and outputs it to the format conversion unit 53 as a timestamp (Ack timestamp in Figure 7).

[0052] When a read acknowledge signal (Ack) output from the arbiter 43 is returned to pixel A, the event data is read from pixel A, temporarily latched in the latch 54 , and then supplied to the format conversion unit 53 .

[0053] The format conversion unit 53 converts the event data, timestamp, and difference data acquired in this manner into data in the format described with reference to Fig. 6, for example, and outputs the converted data to the data processing unit 14. At this time, if the time of event firing is to be transmitted as the timestamp, the format conversion unit 53 subtracts the difference from the Ack timestamp.

[0054] In this way, according to the present technology, a configuration can be achieved in which circuit resources are reduced, and even with such a configuration, it is possible to add a highly reliable timestamp and supply the event data to a processing unit.

[0055] <Application to a Mobile Body> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0056] FIG. 8 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0057] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 8, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown in the figure are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053 as functional components of the integrated control unit 12050.

[0058] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0059] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0060] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0061] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0062] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0063] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0064] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0065] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0066] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 8, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0067] FIG. 9 is a diagram showing an example of the installation position of the imaging unit 12031.

[0068] In FIG. 9, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0069] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0070] 9 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0071] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0072] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0073] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0074] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0075] In this specification, a system refers to an entire device made up of multiple devices.

[0076] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0077] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the present technology.

[0078] The present technology may also be configured as follows. (1) An imaging element including: a pixel array unit in which a plurality of pixels that detect an event are arranged in a matrix; an arbiter that arbitrates requests from the pixels; a difference calculation unit that calculates a difference between an occurrence time when the event occurs in the pixel and a readout time when the event is read out; and a timestamp unit that assigns a timestamp. (2) The imaging element described in (1), in which the difference calculation unit is provided between the pixel array unit and the arbiter. (3) The imaging element described in (1) or (2), in which the difference calculation unit starts clocking when the request is input and stops clocking when a signal that authorizes the reading of event data is input. (4) The imaging element described in any of (1) to (3), in which the difference calculation unit is a counter. (5) The imaging element described in (4), in which the counter is a ripple counter. (6) The imaging element described in any of (1) to (5), in which the timestamp unit assigns the readout time. (7) The image sensor according to any one of (1) to (6), further comprising an output unit that converts the event data read out from the pixel, difference data of the difference from the difference calculation unit, and the timestamp from the timestamp unit into a predetermined format and outputs the converted data. (8) The image sensor according to (7), wherein the timestamp from the timestamp unit is the readout time, and the output unit calculates the occurrence time by subtracting the difference from the readout time, and outputs the calculated occurrence time as the timestamp. (9) The image sensor according to (8), wherein the output unit converts the timestamp indicating the readout time from the timestamp unit and the timestamp indicating the calculated occurrence time into the format and outputs the converted data. (10) The image sensor according to (7), wherein the timestamp and the difference data are stored in a header added to a packet in which data of the event is stored. (11) The image sensor according to any one of (1) to (10), wherein the difference calculation unit is provided for each row of the pixel array unit.(12) An imaging method comprising: an imaging element including: a pixel array section in which a plurality of pixels are arranged in a matrix; an arbiter; a difference calculation section provided between the pixel array section and the arbiter; and an output section; wherein the pixel of the pixel array section detects an event; the difference calculation section calculates the difference between the occurrence time when the event occurs in the pixel and the read time when the event is read; and the output section outputs the added timestamp and the difference.

[0079] REFERENCE SIGNS LIST 10 Imaging device, 11 Optical unit, 12 Imaging element, 13 Control unit, 14 Data processing unit, 21 Event data processing unit, 22 Recording unit, 41 Pixel array unit, 42 Driver unit, 43 Arbiter, 44 Difference calculation unit, 45 Output unit, 46 Selector, 51 Global timer, 52 Time stamp latch, 53 Format conversion unit, 54 Latch, 61 Pixel, 71 Photodiode, 72 Address event detection circuit

Claims

1. An imaging device comprising: a pixel array unit in which a plurality of pixels for detecting events are arranged in a matrix; an arbiter for arbitrating requests from the pixels; a difference calculation unit for calculating the difference between the occurrence time of the event at the pixel and the read-out time of the event; and a timestamp unit for assigning a timestamp.

2. The imaging device according to claim 1, wherein the difference calculation unit is provided between the pixel array unit and the arbiter.

3. The imaging device according to claim 1, wherein the difference calculation unit starts clocking when the request is input, and stops clocking when a signal approving the reading of the event data is input.

4. The imaging device according to claim 1, wherein the difference calculation unit is a counter.

5. The imaging device according to claim 4, wherein the counter is a ripple counter.

6. The imaging device according to claim 1, wherein the time stamp section assigns the readout time.

7. The imaging device according to claim 1, further comprising an output section that converts the event data read out from the pixels, the difference data of the difference from the difference calculation section, and the timestamp from the timestamp section into a predetermined format and outputs the converted data.

8. The imaging element according to claim 7, wherein the timestamp from the timestamp unit is the readout time, and the output unit calculates the occurrence time by subtracting the difference from the readout time, and outputs the calculated occurrence time as the timestamp.

9. The imaging element according to claim 8, wherein the output section converts the timestamp indicating the read time from the timestamp section and the timestamp indicating the calculated generation time into the format and outputs the converted timestamp.

10. The imaging device according to claim 7, wherein the timestamp and the difference data are stored in a header added to a packet in which the event data is stored.

11. The imaging device according to claim 1, wherein the difference calculation unit is provided for each row of the pixel array unit.

12. An imaging method comprising: an imaging element comprising: a pixel array section in which a plurality of pixels are arranged in a matrix; an arbiter; a difference calculation section provided between the pixel array section and the arbiter; and an output section; wherein a pixel of the pixel array section detects an event; the difference calculation section calculates the difference between the occurrence time when the event occurs in the pixel and the read time when the event is read; and the output section outputs an assigned timestamp and the difference.

Images