Imaging device and imaging method

Abstract

The present technology relates to an imaging device and an imaging method that make it possible to reduce processing related to event detection. The imaging device is provided with: a pixel array unit in which a plurality of pixels for detecting an event are arranged in a matrix; an arbiter for arbitrating requests from pixels; and an adjustment unit for adjusting the timing at which a request from a pixel that has detected an event is supplied to the arbiter. The adjustment unit starts measuring time from a time point at which the request from the pixel is input, and outputs the request to the arbiter after a predetermined time has elapsed. The present technology is applicable to event-based vision sensors (EVS), for example.

Description

Imaging device and imaging method

[0001] The present technology relates to an imaging device and an imaging method, and more particularly to an imaging device and an imaging method that are capable of shortening processing time and reducing power consumption, for example.

[0002] One known event-driven imaging device is called an Event-based Vision Sensor (EVS). When an event (e.g., movement) occurs in a scene, the EVS outputs data on the portion of the scene where the luminance level has changed due to the event. Patent Document 1 proposes an imaging device that outputs data related to luminance changes of incident light at high speed.

[0003] Japanese Patent Application Laid-Open No. 2003-3308

[0004] An arbiter-based EVS processes events in the order in which they occur. It is desirable to reduce the processing required when an event occurs.

[0005] The present technology has been made in consideration of such circumstances, and makes it possible to reduce the amount of processing required when an event occurs.

[0006] An imaging device according to one aspect of the present technology is an imaging device including: a pixel array unit in which a plurality of pixels that detect an event are arranged in a matrix; an arbiter that arbitrates requests from the pixels; and an adjustment unit that adjusts the timing at which the request from the pixel that detected the event is supplied to the arbiter.

[0007] An imaging method according to one aspect of the present technology is an imaging method in which an imaging device includes a pixel array unit in which a plurality of pixels are arranged in a matrix, an arbiter, and an adjustment unit provided between the pixel array unit and the arbiter, in which a pixel of the pixel array unit detects an event, the adjustment unit adjusts a timing at which a request from the pixel that detected the event is supplied to the arbiter, and the arbiter arbitrates the request input via the adjustment unit.

[0008] An imaging device according to one aspect of the present technology includes a pixel array unit in which a plurality of pixels that detect events are arranged in a matrix, an arbiter that arbitrates requests from the pixels, and an adjustment unit that adjusts the timing at which the requests from the pixels that have detected the events are supplied to the arbiter.

[0009] In an imaging method according to one aspect of the present technology, an imaging device is provided with a pixel array unit in which a plurality of pixels are arranged in a matrix, an arbiter, and an adjustment unit provided between the pixel array unit and the arbiter, wherein a pixel in the pixel array unit detects an event, the adjustment unit adjusts the timing at which a request from the pixel that detected the event is supplied to the arbiter, and the arbiter arbitrates the requests input via the adjustment unit.

[0010] The imaging device may be an independent device or an internal block constituting a single device.

[0011] 1 is a block diagram showing a configuration of an embodiment of an imaging device to which the present technology is applied; FIG. 2 is a block diagram showing a schematic configuration example of an imaging element; FIG. 3 is a diagram showing an example of a pixel position at which an event is fired; FIG. 4 is a diagram showing adjustment of timing in the row direction; FIG. 5 is a diagram showing adjustment of timing in the column direction; FIG. 6 is a diagram showing an example configuration of a row timing adjustment unit; FIG. 7 is a diagram showing an example configuration of a column timing adjustment unit; FIG. 8 is a diagram showing operation of a column timing adjustment unit; FIG. 9 is a diagram showing readout in address order; FIG. 10 is a block diagram showing an example of a schematic configuration of a vehicle control system; FIG. 11 is an explanatory diagram showing an example of installation positions of an outside vehicle information detection unit and an imaging unit;

[0012] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described.

[0013] <Configuration Example of Imaging Device> The imaging device 10 is a camera equipped with an event sensor that outputs, as event data, a temporal change in an electrical signal obtained by photoelectrically converting an optical signal. Such an event sensor is also called an EVS (event-based vision sensor). A camera equipped with a general image sensor captures images in synchronization with a vertical synchronization signal and outputs frame data, which is image data for one frame (screen), at the period of the vertical synchronization signal. However, the imaging device 10 outputs event data only when an event occurs, and therefore can be said to be an asynchronous or address-controlled camera.

[0014] The imaging device 10 shown in FIG. 1 includes an optical unit 11, an imaging element 12, a control unit 13, and a data processing unit 14.

[0015] The optical unit 11 collects light from a subject and makes it incident on the image sensor 12. The image sensor 12 photoelectrically converts the incident light that has entered through the optical unit 11 to generate event data, and supplies the event data to the data processing unit 14. The image sensor 12 is a light-receiving element that regards a change in pixel luminance as an event and outputs event data that indicates the occurrence of an event.

[0016] The control unit 13 controls the imaging element 12. For example, the control unit 13 instructs the imaging element 12 to start and end imaging.

[0017] The data processing unit 14 is configured with, for example, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a microprocessor, etc., and executes predetermined processing. The data processing unit 14 includes an event data processing unit 21 and a recording unit 22. For example, the event data processing unit 21 performs event data processing using event data supplied from the image sensor 12, and image data processing using event images. The recording unit 22 records and accumulates the predetermined data on a predetermined recording medium as needed.

[0018] 2 is a block diagram showing a schematic configuration example of the image sensor 12. The image sensor 12 includes a pixel array section 41, a drive section 42, a row arbiter 43, a column arbiter 44, and an output section 45. A row timing adjustment section 46 is disposed between the row arbiter 43 and the pixel array section 41, and a column timing adjustment section 47 is disposed between the column arbiter 44 and the pixel array section 41.

[0019] Here, the row arbiter 43 and the row timing adjustment unit 46 are described as separate entities, but the row timing adjustment unit 46 may be included in the row arbiter 43. Similarly, the column arbiter 44 and the column timing adjustment unit 47 are described as separate entities, but the column arbiter 44 may be configured as including the column timing adjustment unit 47.

[0020] A plurality of pixels 61 are arranged in a two-dimensional lattice pattern in the pixel array section 41. Each pixel 61 includes a photodiode 71 as a photoelectric conversion element and an address event detection circuit 72.

[0021] When a change exceeding a predetermined threshold occurs in the photocurrent, which is an electrical signal generated by photoelectric conversion of the photodiode 71, the address event detection circuit 72 detects the change in the photocurrent as an event. When an event is detected, the address event detection circuit 72 outputs a request requesting the output of event data indicating the occurrence of the event to the row arbiter 43 via the row timing adjustment unit 46 and to the column arbiter 44 via the column timing adjustment unit 47.

[0022] The drive unit 42 drives the pixel array unit 41 by supplying a control signal to each pixel 61 of the pixel array unit 41 .

[0023] As will be described later, the row timing adjustment unit 46 adjusts the timing of outputting requests generated in the row direction to the row arbiter 43. The row arbiter 43 arbitrates requests from pixels 61 in the same row within the pixel array unit 41, and returns a response indicating whether or not output of event data is permitted to the pixel 61 that sent the request.

[0024] As will be described later, the column timing adjustment unit 47 adjusts the timing of outputting requests generated in the row direction to the column arbiter 44. The column arbiter 44 arbitrates requests from pixels 61 in the same column within the pixel array unit 41, and returns a response indicating whether output of event data is permitted or not to the pixel 61 that sent the request. A pixel 61 that receives a permission response from both the row arbiter 43 and the column arbiter 44 can output event data to the output unit 45.

[0025] The output unit 45 performs necessary processing on the event data output from each pixel 61 that constitutes the pixel array unit 41, and supplies the data to the data processing unit 14 (FIG. 1).

[0026] <Regarding Timing Adjustment by Timing Adjustment Unit> The following describes the adjustment of timing related to request output performed by the row timing adjustment unit 46 and the column timing adjustment unit 47. Fig. 3 is a diagram illustrating a portion of the image sensor 12 shown in Fig. 2 .

[0027] A pixel 61 in which an event is fired in the pixel array unit 41 issues a request (Req) to the row arbiter 43 and the column arbiter 44. In the example shown in Fig. 3, an event is fired at each timing in pixel 61A, pixel 61B, pixel 61C, and pixel 61D, and ReqA, ReqB, ReqC, and ReqD are issued to the row timing adjustment unit 46 and the column timing adjustment unit 47, respectively.

[0028] The row timing adjustment unit 46 processes requests ReqA, ReqB, ReqC, and ReqD supplied from pixels 61 of the pixel array unit 41 that are from the same row and are supplied within a predetermined time T as requests that arrived at the same time, and outputs Req' to the row arbiter 43.

[0029] The column timing adjustment unit 47 processes requests ReqA, ReqB, ReqC, and ReqD supplied from pixels 61 of the pixel array unit 41 that are supplied within a predetermined time T' as requests that arrived at the same time, and outputs Req' to the column arbiter 44.

[0030] A dash is added to indicate that the request has been processed by the row timing adjustment unit 46 or the column timing adjustment unit 47.

[0031] The timing adjustment performed by the row timing adjustment unit 46 will be described with reference to Fig. 4. Since timing adjustment is performed in the row direction (left and right direction in the figure), an example will be described in which ReqA from pixel 61A and ReqD from pixel 61D in the same row are to be processed.

[0032] 4 , at time t1, an event occurs in pixel 61A, causing ReqA to be output and input to row timing adjustment unit 46. At time t2, an event occurs in pixel 61D, causing ReqD to be output and input to row timing adjustment unit 46. Row timing adjustment unit 46 receives ReqA and ReqD at different times, but treats them as requests supplied at the same time, and at time t3, outputs Req′ (referred to as Req′AD) to row arbiter 43, indicating that a request has been made from pixel 61A and pixel 61D (a request has been made from the row in which pixel 61A and pixel 61D are arranged).

[0033] Time t3 is the same as time t2 or a time slightly later than time t2. The time from time t1 to time t3 corresponds to time T. When row timing adjustment unit 46 inputs ReqA at time t1, it does not immediately output ReqA to row arbiter 43, but rather outputs ReqA (Req'AD) after time T has elapsed since time t1. In other words, ReqA is delayed by row timing adjustment unit 46 and then output.

[0034] If a Req is input during the time T after receiving ReqA at time t1, the row timing adjustment unit 46 also outputs information about that Req along with ReqA.

[0035] The row timing adjustment unit 46 waits for requests to be input within a predetermined time, and issues Req' to the row arbiter 43 so that the requests input within the predetermined time are treated as requests input at the same time.

[0036] As a process for Req'AD issued at time t3, reading from pixel 61A and pixel 61D is performed at time t4.

[0037] If timing adjustment by the row timing adjustment unit 46 were not performed (in a configuration without the row timing adjustment unit 46), the row arbiter 43 would receive ReqA from pixel 61A at time t1 and ReqD from pixel 61D at time t2, and readout would occur in that order. In this case, readout from pixel 61A and readout from pixel 61D would occur at different times. In this case, readout would occur twice.

[0038] As described above, when timing adjustment is performed by the row timing adjustment section 46, reading is performed only once, and the number of read operations can be reduced.

[0039] For example, if pixel 61A and pixel 61D have different sensitivities due to variations in transistor performance, even if events occur in pixel 61A and pixel 61D at the same time, the timing of event detection will differ, and as a result, ReqA and ReqD will be issued at different times as described above. In such a case, the row timing adjustment unit 46 executes a waiting process for ReqA and ReqD, so that the pixels 61 can be treated as pixels 61 in which events occurred at the same time.

[0040] This makes it possible to perform processing that absorbs the latency between pixels, and also makes it possible to read out events with fewer readout times.

[0041] The time T can be set so that even if events occurring within the time T are treated as occurring at the same time, the final accuracy is not affected.

[0042] If pixel 61A and pixel 61D have the same sensitivity, then pixel 61A and pixel 61D will detect events at different times, and the timing for issuing ReqA and ReqD at different times will be correct, as explained with reference to Figure 4. Even in such a case, the processing in row timing adjustment unit 46 will enable ReqA and ReqD to be processed as requests issued at the same time, and by adjusting time T to a time that does not affect accuracy as described above, the number of readouts can be reduced without reducing accuracy.

[0043] By being able to read out events with fewer read times, the time required for reading can be shortened, the amount of processing can be reduced, and power consumption can be reduced.

[0044] Next, the timing adjustment performed by the column timing adjustment unit 47 in the column direction will be described with reference to Fig. 5. For timing adjustment in the column direction (the vertical direction in the figure), the following example will be described in which ReqA from pixel 61A, ReqB from pixel 61B, and ReqC from pixel 61C are output within time T'.

[0045] 4 , at time t11, an event occurs in pixel 61B, and ReqB is output and input to the column timing adjustment unit 47. At time t12, an event occurs in pixel 61A, and ReqA is output and input to the column timing adjustment unit 47. At time t13, an event occurs in pixel 61C, and ReqC is output and input to the column timing adjustment unit 47.

[0046] The column timing adjustment unit 47 receives ReqB, ReqA, and ReqC at different times, but treats them as requests supplied at the same time, and at time t14, Req'A indicating that a request has been made from pixel 61A, Req'B indicating that a request has been made from pixel 61B, and Req'C indicating that a request has been made from pixel 61C are output to the column arbiter 44.

[0047] Time T' can be a signal that instructs the output unit 45 to read a signal from the pixel 61 where an event has occurred and output the signal, for example, a signal synchronized with the vertical synchronization signal. The vertical synchronization signal can be used as a trigger, and the period of the vertical synchronization signal can be used as time T'. When the column timing adjustment unit 47 receives ReqB at time t11 within time T', it does not immediately issue ReqB to the column arbiter 44, but waits until a trigger is input before outputting. ReqA and ReqC are treated in the same way as ReqB, and output by the column timing adjustment unit 47 is delayed until a trigger is input.

[0048] The column timing adjustment unit 47 waits for requests to be input within a predetermined time, and issues Req' to the column arbiter 44 so that the requests input within the predetermined time are treated as requests input at the same time.

[0049] As processing for Req'A, Req'B, and Req'C issued at time t14, signals are read out from pixels 61A, 61B, and 61C at times t15, t16, and t17, respectively. This reading can be performed, for example, in address order. In the example shown in FIG. 3, pixels 61A, 61B, and 61C are located in order from the top of the pixel array unit 41. In this case, when reading out in address order, the pixels are pixel 61A, pixel 61B, and pixel 61C, and they can be read out in this order.

[0050] If timing adjustment by the column timing adjustment unit 47 is not performed (in a configuration without the column timing adjustment unit 47), the column arbiter 44 receives ReqB from pixel 61B at time t11, ReqA from pixel 61A at time t12, and ReqC from pixel 61C at time t13, and so readout is arranged in this order. In this case, pixel 61B is read from the row in which pixel 61B is located (row B), pixel 61A in row A above row B is read, and pixel 61C below row A and row B is read.

[0051] When reading is performed in this order of requests, there is no correlation between the reading order and the pixel positions (addresses) in the pixel array section 41, so when performing signal processing, evaluation, measurement, or combining with a gradation sensor in the subsequent stages, it is necessary to separately rearrange the pixels in address order beforehand.

[0052] As described above, when timing adjustment by the column timing adjustment unit 47 is performed in a queue, it is possible to read out in address order rather than in request order, so there is no need to take measures such as rearranging the data in address order during subsequent signal processing, evaluation, measurement, or when combined with a gradation sensor.

[0053] Since it is also possible to read out data in the order of addresses in the column direction, the amount of processing can be reduced, and power consumption can also be reduced.

[0054] <Configuration Example and Operation of Row Timing Adjustment Unit> Figure 6 is a diagram showing a configuration example of the row timing adjustment unit 46. The row timing adjustment unit 46 includes timing control units 101 in the same number as the number of rows in the pixel array unit 41 (Figure 2). When the pixel array unit 41 is made up of M rows, the row timing adjustment unit 46 includes timing control units 101-1 to 101-M that execute request queuing for each row.

[0055] Since the timing control units 101-1 to 101-M have the same configuration, the configuration of the timing control unit 101-1 will be described as an example. The timing control unit 101-1 includes a signal latch unit 111, a latch timing control unit 112, and a timer unit 113.

[0056] When an event occurs in a pixel 61 arranged in a predetermined row, the signal latch unit 111 is supplied with a request signal from that pixel 61. The signal latch unit 111 latches the input request signal until a latch trigger is input from the latch timing control unit 112, and when the latch trigger is input, the signal latch unit 111 outputs a request signal (Req1' in FIG. 6) to the column arbiter 44 (FIG. 2).

[0057] A request signal is supplied to the latch timing control unit 112 at the same timing as the request signal is input to the signal latch unit 111. When the latch timing control unit 112 inputs the request signal, it notifies the timing unit 113. When the timing unit 113 receives an instruction from the latch timing control unit 112, it starts timing.

[0058] The timer unit 113 continues to measure time until it reaches the threshold value P. This measured time corresponds to the time T described above. The threshold value P is configured to be supplied from the register 114. By configuring the register 114 to supply the threshold value P, it becomes possible to change and set the measured time T in the timer unit 113 to a desired time.

[0059] When the time being measured reaches time T, timer unit 113 notifies latch timing control unit 112 of this fact. Upon receiving the notification from timer unit 113, latch timing control unit 112 outputs a latch trigger to signal latch unit 111. Signal latch unit 111 latches one or more input request signals until it receives the latch trigger, and outputs the request signal latched at the time it receives the latch trigger to row arbiter 43 in the subsequent stage.

[0060] Although the timing at which latch timing control section 112 outputs a latch trigger has been described as being notified by clock section 113, it may also be notified from a source other than clock section 113. Referring to Fig. 6, row timing adjustment section 46 includes selection section 115, and selection section 115 may also be configured to receive an internally generated trigger or an externally input trigger.

[0061] The internally generated trigger is, for example, a trigger generated within the image sensor 12, and the externally input trigger is, for example, a trigger generated outside the image sensor 12 and input into the image sensor 12. The selection unit 115 selects the internally generated trigger or the externally input trigger and supplies it to the latch timing control unit 112. The latch timing control unit 112 outputs a latch trigger to the signal latch unit 111 at the time the trigger is supplied.

[0062] When the selection unit 115 is provided, it is possible to configure the system so that selection between the internally generated trigger and the externally input trigger can be switched depending on some conditions. It is also possible to configure the system so that the selection unit 115 is not provided and only either the internally generated trigger or the externally input trigger is supplied to the latch timing control unit 112. When the system is configured so that either the internally generated trigger or the externally input trigger is supplied, the timekeeping unit 113 and the register 114 may be omitted.

[0063] The operation of the row timing adjustment unit 46 having such a configuration will be described with reference to the timing chart shown in Fig. 7. The timing chart shown in Fig. 7 is a timing chart for the example of the situation described with reference to Figs. 3 and 4, in which timing is performed by the timing unit 113.

[0064] At time t1, pixel 61A fires and outputs a request signal (Req1 is set to High), which is then supplied to signal latch unit 111 and latch timing control unit 112. Latch timing control unit 112 instructs timing unit 113 to start timing, causing timing unit 113 to start timing.

[0065] The timer unit 113 counts from 0 to P. At time t2 while the timer unit 113 is timing, the pixel 61D fires, a request signal is output, and the request signal is supplied to the signal latch unit 111 and the latch timing control unit 112. However, since Req1 is in a High state, the signal latch unit 111 maintains the state in which it latches the signal, and the latch timing control unit 112 maintains the state in which it waits for a notification from the timer unit 113.

[0066] At time t3, when the timer unit 113 counts up to P, this fact is notified to the latch timing control unit 112. Upon receiving the notification from the timer unit 113, the latch timing control unit 112 outputs a latch trigger to the signal latch unit 111. When the latch trigger is issued at time t3, the signal latch unit 111 outputs a request signal (Req1′) to the row arbiter 43.

[0067] In this way, by performing queueing in the row direction, events can be read out with fewer read operations, shortening the read time and reducing power consumption.

[0068] <Configuration Example and Operation of Column Timing Adjustment Unit> Fig. 8 is a diagram showing a configuration example of the column timing adjustment unit 47. The configuration of the column timing adjustment unit 47 can basically be the same as the configuration of the row timing adjustment unit 46 shown in Fig. 6.

[0069] The column timing adjustment unit 47 includes timing control units 201 in the same number as the number of columns in the pixel array unit 41 (FIG. 2). If the pixel array unit 41 is made up of N columns, the column timing adjustment unit 47 includes timing control units 201-1 to 201-N that execute request queuing for each column. Since the timing control units 201-1 to 201-N have the same configuration, the configuration of the timing control unit 201-1 will be described here as an example. The timing control unit 201-1 includes a signal latch unit 211 and a latch timing control unit 212.

[0070] When an event occurs in a pixel 61 arranged in a predetermined column, the signal latch unit 211 is supplied with a request signal from that pixel 61. The signal latch unit 211 latches the input request signal until a latch trigger is input from the latch timing control unit 212, and when the latch trigger is input, outputs a request signal (Req1' in FIG. 8) to the column arbiter 44 (FIG. 2).

[0071] A request signal is supplied to the latch timing control unit 212 at the same timing as the request signal is input to the signal latch unit 211. An internally generated trigger or an externally input trigger selected by the selection unit 215 is input to the latch timing control unit 212. A configuration may be adopted in which the selection unit 215 is not provided and only either the internally generated trigger or the externally input trigger is supplied to the latch timing control unit 212.

[0072] When adjusting the timing (waiting) in the column direction, the timing at which the latch timing control section 212 issues a latch trigger is set to the same timing for the timing control sections 201-1 to 201-N, so an internally generated trigger or an externally input trigger is supplied in common to the timing control sections 201-1 to 201-N. The timing control section 201 included in the column timing adjustment section 47 can be configured without the timer section 113 and the register 114 (FIG. 6).

[0073] The internal trigger or external input trigger can use a vertical synchronization signal and is a trigger that is output at a predetermined interval (period). Since the trigger is input at each predetermined interval, the latch timing control unit 212 outputs a latch trigger to the signal latch unit 211 at each predetermined interval.

[0074] The signal latch unit 211 latches one or more input request signals until it receives a latch trigger, and outputs the request signal that has been latched at the time it receives the latch trigger to the column arbiter 44 at the subsequent stage.

[0075] The operation of the column timing adjustment unit 47 having such a configuration will be described with reference to the timing chart shown in Fig. 9. The timing chart shown in Fig. 9 is a timing chart for an example of the situation described with reference to Figs. 3 and 5. In Fig. 9, the time from time t21 to time t22 is time T', which is the trigger period (Vsync trig).

[0076] At time t11, pixel 61B fires and a request signal is output from pixel 61B (Req2 is set to High), and the request signal is supplied to the signal latch unit 211 and latch timing control unit 212 of timing control unit 201-2. At time t12, pixel 61A fires and a request signal is output from pixel 61A (Req1 is set to High), and the request signal is supplied to the signal latch unit 211 and latch timing control unit 212 of timing control unit 201-1.

[0077] At time t13, the pixel 61C fires and outputs a request signal (Req3 goes High), which is then supplied to the signal latch unit 211 and latch timing control unit 212 of the timing control unit 201-3.

[0078] At time t22, when a time T' has elapsed since time t21, a trigger is collectively supplied to each of the latch timing control sections 212 of the timing control sections 201-1 to 202-N. At time t14, which is the same time as time t22 or a little later, each of the latch timing control sections 212 of the timing control sections 201-1 to 202-N outputs a latch trigger to each of the signal latch sections 211.

[0079] When a latch trigger is issued at time t14, each of the latch timing control sections 212 of the timing control sections 201-1 to 202-N outputs a request signal to the column arbiter 44. In the example shown in Figure 9, the timing control section 201-1 outputs a request signal (Req1') to the column arbiter 44, the timing control section 201-2 outputs a request signal (Req2') to the column arbiter 44, and the timing control section 201-3 outputs a request signal (Req3') to the column arbiter 44.

[0080] In this way, by performing queueing in the column direction, it becomes possible to read out events that have occurred within a certain period in address order. This will be further explained with reference to FIG.

[0081] FIG. 10 is a timing chart for explaining the output timing of request signals Req1′, Req2′, and Req3′ output from the timing control unit 201 and read acknowledge signals Ack1, Ack2, and Ack3 output from the column arbiter 44.

[0082] Since Req1', Req2', and Req3' are output simultaneously at time t14 by the column timing adjustment unit 47, the column arbiter 44 simultaneously inputs these request signals at time t14. The column arbiter 44 operates based on rule 1, which requires that requests be read in the order of their input time (and that ACKs be returned). In addition to rule 1, rule 2 is also established, which requires that requests be read in address order (and that ACKs be returned) when they are input at the same time.

[0083] Since the column arbiter 44 returns Ack based on rule 2, Ack1 is returned at time t14, Ack2 is returned at time t31 after time t14, and Ack3 is returned at time t32 after time t31.

[0084] In this way, by simply setting a simple rule that requests input to the column arbiter 44 at the same time are read in address order, it is possible to read events that occur within a certain period of time in address order.

[0085] By reading out in address order, there is no need to rearrange the addresses separately beforehand when performing signal processing, evaluation, measurement, or when combining with a gradation sensor, which makes it possible to reduce processing, shorten processing time, and reduce power consumption.

[0086] <Application to a Mobile Body> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0087] FIG. 11 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0088] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 11, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053.

[0089] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0090] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0091] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0092] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0093] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0094] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0095] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0096] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0097] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 11, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0098] FIG. 12 is a diagram showing an example of the installation position of the imaging unit 12031.

[0099] In FIG. 12, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0100] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0101] 12 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0102] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0103] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0104] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0105] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0106] In this specification, a system refers to an entire device made up of multiple devices.

[0107] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0108] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the present technology.

[0109] Note that the present technology may also be configured as follows. (1) An imaging device comprising: a pixel array unit in which a plurality of pixels that detect an event are arranged in a matrix; an arbiter that arbitrates requests from the pixels; and an adjustment unit that adjusts timing at which the request from the pixel that detected the event is supplied to the arbiter. (2) The imaging device described in (1), wherein the adjustment unit starts clocking from the time when the request from the pixel is input, and outputs the request to the arbiter after a predetermined time has elapsed. (3) The imaging device described in (1), wherein the adjustment unit starts clocking from the time when the request from the pixel is input, and if a plurality of the requests are input during the predetermined time, outputs the plurality of the requests to the arbiter as requests input at the same time. (4) The imaging device described in any of (1) to (3), wherein the adjustment unit is provided for each row of the pixel array unit. (5) The imaging device according to any one of (1) to (4), wherein the adjustment unit receives a trigger at a predetermined cycle, and when a plurality of requests are received between the time when the request from the pixel is received and the time when the trigger is received, outputs the plurality of requests to the arbiter as requests received at the same time. (6) The imaging device according to (5), wherein the adjustment unit is provided for each column of the pixel array unit, and the trigger is supplied collectively to the adjustment unit provided for each column. (7) The imaging device according to (5) or (6), wherein when a plurality of requests are received at the same time, the arbiter outputs a response permitting output of event data in the order of the pixel addresses. (8) An imaging method comprising: an imaging device comprising: a pixel array section in which a plurality of pixels are arranged in a matrix; an arbiter; and an adjustment section provided between the pixel array section and the arbiter; a pixel of the pixel array section detects an event; the adjustment section adjusts the timing of supplying a request from the pixel that detected the event to the arbiter; and the arbiter arbitrates the request input via the adjustment section.

[0110] REFERENCE SIGNS LIST 10 Imaging device, 11 Optical unit, 12 Imaging element, 13 Control unit, 14 Data processing unit, 21 Event data processing unit, 22 Recording unit, 41 Pixel array unit, 42 Driving unit, 43 Row arbiter, 44 Column arbiter, 45 Output unit, 46 Row timing adjustment unit, 47 Column timing adjustment unit, 61 Pixel, 71 Photodiode, 72 Address event detection circuit, 101 Timing control unit, 111 Signal latch unit, 112 Latch timing control unit, 113 Time counting unit, 114 Register, 115 Selection unit, 201 Timing control unit, 211 Signal latch unit, 212 Latch timing control unit, 215 Selection unit

Claims

1. An imaging device comprising: a pixel array section in which a plurality of pixels that detect an event are arranged in a matrix; an arbiter that arbitrates requests from the pixels; and an adjustment section that adjusts the timing at which the request from the pixel that detected the event is supplied to the arbiter.

2. The imaging device according to claim 1, wherein the adjustment unit starts timing from the time when the request from the pixel is input, and outputs the request to the arbiter after a predetermined time has elapsed.

3. The imaging device described in claim 1, wherein the adjustment unit starts timing from the time when the request from the pixel is input, and if multiple requests are input within a predetermined time, outputs the multiple requests to the arbiter as requests input at the same time.

4. The imaging device according to claim 1, wherein the adjustment section is provided for each row of the pixel array section.

5. The imaging device of claim 1, wherein the adjustment unit receives a trigger at a predetermined cycle, and when multiple requests are received between the time the request from the pixel is received and the time the trigger is received, outputs the multiple requests to the arbiter as requests received at the same time.

6. The imaging device according to claim 5, wherein the adjustment unit is provided for each column of the pixel array unit, and the trigger is supplied collectively to the adjustment units provided for each column.

7. The imaging device according to claim 5, wherein the arbiter outputs a response permitting output of event data in the order of the pixel addresses when a plurality of requests are input at the same time.

8. An imaging method comprising: an imaging device comprising: a pixel array section in which a plurality of pixels are arranged in a matrix; an arbiter; and an adjustment section provided between the pixel array section and the arbiter; a pixel of the pixel array section detects an event; the adjustment section adjusts the timing at which a request from the pixel that detected the event is supplied to the arbiter; and the arbiter arbitrates the requests input via the adjustment section.

Images