Imaging device and equipment

Abstract

To allow an event-based imaging device to perform event detection on a moving subject with improved accuracy.SOLUTION: An imaging device includes: a pixel portion that includes a plurality of unit pixels and is configured to detect as an event signal a change in a light intensity of each of the unit pixels; an event readout portion configured to read out the event signal from the pixel portion; an output portion configured to output, regarding at least one region relating to the event signal of the pixel portion, motion information relating to motion of the at least one region; and a determination portion configured to determine a predicted region after a change in the at least one region, the predicted region being predicted on the basis of the motion information. The event readout portion is configured to read out the event signal from the determined predicted region.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The techniques of this disclosure relate to event-based imaging devices and equipment. [Background technology]

[0002] In recent years, IoT (Internet of Things), AI (Artificial Intelligence), and autonomous driving With the spread of image sensors such as those mentioned above, event-based sensors have been proposed as image sensors that consume less power and are faster than conventional sensors. Event-based sensors are also called dynamic vision sensors. These sensors monitor changes in the amount of light at each pixel in a two-dimensional pixel array and detect the changes as events. Event-based sensors can detect and output events at the timing when a change in the amount of light occurs, regardless of object detection processing or image processing, making them capable of low power consumption and high-speed operation.

[0003] With the aim of detecting more events from objects using an event-based sensor, for example, Patent Document 1 proposes an event-based sensor that sets the probability of detecting an event based on the results of object detection using pattern recognition of pixel values. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-136958 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the time resolution of the object detection process is lower than the time resolution of the output data of the event-based sensor. Therefore, in the above event-based sensor, when a specific area in the pixel array is set based on the detection result of the object as the subject and an event signal is read out from the specific area, the readout of the event signal may be delayed due to the delay associated with the object detection process. In addition, in the above event-based sensor, if the object to be detected moves, the object may not fit within the set specific area, and an event of the object outside the specific area may not be read out. As a result, the above event-based sensor may not be able to detect the change of the subject in the pixel array as an event with high accuracy.

[0006] The technology disclosed herein has been made in consideration of the above-mentioned problems, and has an object to perform more accurate event detection for a moving subject in an event-based imaging device. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the imaging device disclosed herein includes an imaging device having a pixel unit having a plurality of unit pixels and detecting a change in light amount for each unit pixel as an event signal, an event readout unit that reads out the event signal from the pixel unit, an output unit that outputs motion information regarding the motion of at least one region in the pixel unit related to the event signal, and a determination unit that determines a prediction region after the change in the at least one region, which is predicted based on the motion information, wherein the event readout unit reads out the event signal from the determined prediction region.

[0008] In order to achieve the above object, an apparatus according to the present disclosure is an apparatus including the above imaging device, the apparatus including an optical device corresponding to the imaging device, a control device for controlling the imaging device, and a control circuit for controlling the imaging device. The present invention includes an apparatus further comprising at least one of a processing device that processes signals output from the imaging device, a display device that displays information obtained by the imaging device, a storage device that stores information obtained by the imaging device, and a mechanical device that operates based on information obtained by the imaging device. Effect of the Invention

[0009] According to the technology of the present disclosure, in an event-based imaging device, event detection for a moving subject can be performed with higher accuracy. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an imaging device according to a first embodiment; [Diagram 2] FIG. 1 is a diagram showing a schematic configuration of a pixel according to a first embodiment; [Diagram 3] FIG. 1 is a diagram for explaining a process for determining an area of ​​interest executed by an imaging device according to a first embodiment; [Figure 4] FIG. 1 is a diagram for explaining a motion prediction process executed by an imaging device according to a first embodiment; [Diagram 5] FIG. 1 is a diagram for explaining a process for determining a prediction region executed by an imaging device according to a first embodiment; [Figure 6] FIG. 1 is a diagram showing a schematic configuration of an event readout unit of an image capture device according to a first embodiment; [Figure 7] FIG. 13 is a diagram showing a schematic configuration of an event readout unit of an image capture device according to a second embodiment; [Figure 8] FIG. 13 is a diagram showing a schematic configuration of an arbitration unit of an imaging device according to a second embodiment; [Figure 9] FIG. 13 is a diagram showing a schematic configuration of an apparatus including a semiconductor device according to a third embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each embodiment described below, an event-based imaging device will be mainly described as an example of a photoelectric conversion device. However, each embodiment is not limited to an event-based imaging device, and can be applied to other examples of photoelectric conversion devices. For example, there are distance measuring devices (devices for measuring distance using focus detection or TOF (Time Of Flight)), photometry devices (devices for measuring the amount of incident light, etc.), etc.

[0012] First Embodiment Fig. 1 shows a schematic configuration of an event-based imaging device 100 according to the first embodiment. As shown in Fig. 1, the event-based imaging device 100 includes a pixel unit 101, an event readout unit 103, a memory 105, a region of interest determination unit 106, a motion prediction unit 107, a prediction region determination unit 108, and an event rate control unit 109.

[0013] The pixel section 101 is configured by arranging a plurality of unit pixels 110 in a two-dimensional array, for example, similar to a general image sensor. The unit pixels 110 may be arranged in a one-dimensional line sensor shape or other shapes. Each unit pixel 110 is a pixel capable of detecting a change in the amount of incident light as an event signal, and is configured by a circuit exemplified in FIG. 2. As shown in FIG. 2, one unit pixel 110 has a photodiode 200, a logarithmic I / V conversion section 201, a subtraction section 202, and a comparison section 203.

[0014] In the unit pixel 110, the photodiode 200 generates a photocurrent I p For example, the logarithmic I / V converter 201 generates a photocurrent I P is converted to a potential, and a logarithmic potential V1 is obtained by conversion using a logarithmic function. The output V1 of the logarithmic I / V converter 201 does not necessarily have to be a logarithmic potential, and may be a potential that changes linearly according to the amount of incident light, or another value that changes according to the amount of light.

[0015] The subtraction unit 202 calculates the difference V between the logarithmic potential V1 and the reference potential. dIn addition, the comparison unit 203 calculates the difference V d The comparison unit 203 uses two thresholds, a positive threshold T1 and a negative threshold T2, and the difference V d When the value of the positive threshold is exceeded or when the value of the negative threshold is exceeded, an event signal with a value other than 0 is generated. Here, the event signal E of a pixel P(x, y) at time t can be expressed by the following equation (1).

number

[0016] Here, the time resolution of event detection for the unit pixel 110 at time t is, for example, 1 μs. When the value of the event signal E is 1, it indicates that a change has occurred in the unit pixel 110 to brighten it, i.e., the occurrence of a positive event. When the value of the event signal E is −1, it indicates that a change has occurred in the unit pixel 110 to darken it, i.e., the occurrence of a negative event.

[0017] The above formula (1) is an example of an expression of the event signal E, and other expressions may be used, such as assigning different numerical values ​​to each state of a positive event, a negative event, and no event, or treating positive and negative events as one type of event signal without distinguishing between them. The event signal E is a signal that is generated independently (asynchronously) of the frame synchronization signal used in normal image sensors, and the use of the event signal E enables event detection with high time resolution. The detected event signal E is sent to the event readout unit 103. The reference potential is updated using a response signal Ack from the event readout unit 103, which will be described later, as a trigger.

[0018] The pixel unit 101 is configured to output a transmission request signal Req for an event signal on a row-by-row basis to the event readout unit 103. The transmission request signal Req from each pixel row 102 is output when, for example, at least one of the unit pixels 110 in each column in the pixel row 102 detects an event signal, that is, when the value of the event signal E becomes 1 or −1.

[0019] In this embodiment, the pixel unit 101 sends transmission requests in row units, but pixel groups may be formed in any other units, such as column units or rectangular area units having a certain row or column width, and a transmission request may be output for each unit.

[0020] The event readout unit 103 reads out an event signal from the pixel unit 101. Specifically, the event readout unit 103 has an arbitration unit 600 that arbitrates transmission request signals Req from, for example, each pixel row 102 of the pixel unit 101. Details of the arbitration unit 600 will be described later with reference to FIG. 6. The arbitration unit 600 reads out an event signal E from each pixel in a selected pixel row 102. When the event signal E is read out by the arbitration unit 600, the value of the event signal E of each unit pixel 110 in the pixel row 102 is returned to 0.

[0021] Thereafter, a response signal Ack is returned from the event readout unit 103 to the pixel row 102, and each unit pixel 110 of the pixel row 102 resumes event detection using the logarithmic potential V1 at the time of receiving the response signal Ack as a new reference potential. The event signal of the pixel unit 101 read out by the event readout unit 103 is used as an input for controlling an event rate, which is the event readout frequency described later, and is also used to generate final output data of the event-based imaging device 100. Information indicating, for example, polarity (positive or negative of the event signal) as well as the coordinate value of the unit pixel 110 from which the event signal was read out and a timestamp at the time of reading out is added to each event signal, and the event signal is output from the event readout unit 103 as event data. The coordinate value and timestamp can be added to the event signal by the event readout unit 103. The coordinate value is also added to the event readout signal. The event readout unit 103 determines the row and column positions of the event signal that it has read out and adds them to the event signal. The event readout unit 103 also has, for example, a timer circuit (not shown), and adds the output value of the timer at the time the event is read out to the event signal as a timestamp. In this way, the event readout unit 103 generates and outputs event data.

[0022] The bus 104 is used for data transfer between the event reading unit 103 and a memory 105, an attention area determining unit 106, a motion predicting unit 107, a prediction area determining unit 108, and an event rate control unit 109, which will be described later.

[0023] The memory 105 temporarily stores intermediate processing data, such as event data output from the event reading unit 103, and input / output data for the attention area determination unit 106, motion prediction unit 107, prediction area determination unit 108, and event rate control unit 109, which will be described later.

[0024] The attention area determination unit 106 performs processing for detecting an attention area, such as determining whether or not an object of interest exists, from the read event data. In the following description, the attention area detected by the attention area determination unit 106 is referred to as a "pre-attention area." This processing can use object detection processing using, for example, a CNN (Convolutional Neural Network). The attention area determination unit 106 performs the pre-attention area detection processing at regular time intervals, for example, depending on the time required for the object detection processing.

[0025] However, CNN requires input data in an array structure such as frame image data output by a general image sensor. Therefore, the output event sequence of the event readout unit 103 may be converted into an array structure and stored in the memory 105, and the attention area determination unit 106 may be configured to read out the stored data from the memory 105 and process it.

[0026] An example of an array structure representation of an event sequence is a two-dimensional histogram format in which a value corresponding to a pixel value of output image data from a general image sensor is the number of events that occurred at that pixel within a certain period of time. Alternatively, a Time Surface format in which a value corresponding to a pixel value of output image data from a general image sensor is the latest timestamp of events that occurred at each pixel may be used. These are just examples, and any method may be adopted to convert an event sequence into array structure data.

[0027] Furthermore, the attention region determining unit 106 outputs, for example, region information of the detected pre-attention region and the event rate of the corresponding region. The detected pre-attention region may be a plurality of regions.

[0028] FIG. 3 is a diagram for explaining an example of the attention area detected by the above process. In the example of FIG. 3, a vehicle 301 traveling on a road surface 300 is imaged by the event-based imaging device 100. The pre-attention area 302 is an attention area determined by the attention area determination unit 106, which detects the vehicle 301 as an object of attention. The area information of the pre-attention area 302 is expressed by a pair of values ​​of the upper left coordinate and the lower right coordinate of the area, assuming that the pre-attention area 302 is a rectangular area in the pixel unit 101. The event rate of the event used to determine the pre-attention area 302 may be assigned according to the type of object of attention (such as a person, a vehicle, or a traffic light), for example, and is set to a value between 0 and 1. The pre-attention area 302 is determined by the attention area determination unit 106 repeatedly performing processing for a certain period of time, for example, every 10 ms.

[0029] In this way, the time period (e.g., 10 ms) for updating the pre-attention region 302 is generally larger than the time resolution (e.g., 1 μs) for event detection. This does not impose any constraints on the realization of processing.

[0030] The motion prediction unit 107 is an output unit that outputs motion information related to the motion of each of the pre-attention regions output by the attention region determination unit 106. Here, the motion information may include, for example, information indicating the type of motion of the event-based imaging device 100, such as two-dimensional translation (x-axis direction (horizontal direction) and y-axis direction (vertical direction)), rotation, yaw, pitch, zoom, etc. Additionally or alternatively, the motion information may include information indicating the type of motion of the target of interest imaged by the event-based imaging device 100, such as two-dimensional translation (x-axis direction (horizontal direction) and y-axis direction (vertical direction)), rotation, yaw, pitch, zoom, etc. Additionally or alternatively, the motion information may include information indicating the speed and acceleration of the target, which are the amounts of these motions.

[0031] Then, the prediction region determination unit 108 determines a prediction region after the pre-attention region changes from the current time to a future time, based on the region information of the pre-attention region 302 output by the attention region determination unit 106 and the motion information output by the motion prediction unit 107. The prediction region determination unit 108 outputs the region information of the determined prediction region.

[0032] The processing of the motion prediction unit 107 will be described based on the examples shown in Fig. 3 and Fig. 4. The time shown in Fig. 3 is the current time t, and the time at which the pre-region of interest is calculated immediately before the time (time t) in Fig. 3 is the past time t-1. Moreover, the time at which the pre-region of interest is calculated immediately after the time (time t) in Fig. 3 is the future time t+1.

[0033] FIG. 4A shows an example of a vehicle 401 and a pre-attention region 402 at time t-1. In FIG. 4A, the vehicle 401 captured by the event-based imaging device 100 and the pre-attention region 402 determined by the attention region determination unit 106 are each shown as a rectangular region. FIG. 4B shows an example of a vehicle 411 and a prediction region 412 at time t+1 predicted by the prediction region determination unit 108 based on the motion information output by the motion prediction unit 107. In FIG. 4B, the vehicle 401 captured by the event-based imaging device 100 and the pre-attention region 402 determined by the attention region determination unit 106 are each shown as a rectangular region drawn with a dashed line. Note that the pre-attention region 402 determined by the attention region determination unit 106, which is the first region determination unit, at time t-1 is an example of a first region.

[0034] The motion prediction unit 107 calculates an optical flow, for example, using region information of the pre-attention region 402 at time t-1 and events detected in the event-based imaging device 100 between time t-1 and time t. The optical flow is vector information representing a change for each pixel or each attention region, and is information calculated based on a change in an occurrence position of an event occurring in a certain region within a certain period of time. Furthermore, the motion prediction unit 107 calculates a type of motion and / or an amount of motion of the pre-attention region 402 from the optical flow. Then, the motion prediction unit 107 calculates region information of the prediction region 412 at time t+1 by interpolating the change in the pre-attention region 402 up to future time t+1 using the optical flow. Note that the prediction region 412 at time t+1 is an example of a prediction region after a change in the first region determined by the prediction region determination unit 108, which is a second region determination unit, at time t+1.

[0035] In the event-based imaging device 100, a method for calculating an optical flow from an event signal detected in the pixel unit 101 and a method for calculating the type and amount of movement of a pre-attention region from the optical flow can be realized by a known algorithm. The method described here is one example, and any other method for acquiring the type and amount of movement of a region of interest based on an event signal detected in the pixel unit and event data generated by the event readout unit may be adopted.

[0036] Fig. 5A shows an example of a prediction region determined by the prediction region determination unit 108 in the example shown in Fig. 3 and Fig. 4. In Fig. 5A, the prediction region 412 shown in Fig. 4B is shown in the image shown in Fig. 3. The prediction region determination unit 108 outputs, for example, region information of the smallest rectangular region including the pre-attention region 302 at the current time t and the prediction region 412 at future time t+1 as region information of the prediction region after the change of the attention region. This prediction region 502 is also shown in Fig. 5A.

[0037] However, the above calculation method of the prediction region 502 is an example, and for example, the prediction region determination unit 108 may calculate the reliability of the motion prediction of the pre-attention region by the motion prediction unit 107. Then, when the calculated reliability is lower than a predetermined threshold, the prediction region determination unit 108 may use a prediction region 512 obtained by adding margins to the top, bottom, left and right of the prediction region 502 in FIG. 5A instead of the prediction region 502, as shown in FIG. 5B. Here, the value of the reliability of the motion prediction may be set to a higher value as the difference between the prediction region at the current time predicted based on the past pre-attention region and motion information and the pre-attention region at the current time is smaller. Examples of the difference between the prediction region and the pre-attention region include the difference in the coordinates of the region, the difference in the width of the region in the x-axis direction, and the difference in the width of the region in the y-axis direction.

[0038] The event rate control unit 109 controls an event rate, which is the frequency at which the event reading unit 103 reads events in a prediction region, based on the region information for each prediction region.

[0039] Fig. 6 shows an example of the configuration of the event readout unit 103. As shown in Fig. 6, in Fig. 6, the arbitration unit 600 arbitrates the transmission request signals Req output for each row of the pixel unit 101. More specifically, the arbitration unit 600 repeatedly executes the reading of the event for the row, the return of the response signal Ack, and the output of the readout event to the subsequent stage, for example, in the order of arrival of the transmission request signals Req.

[0040] The arbitration process by the arbitration unit 600 described above is not limited to this. For example, the arbitration unit 600 may check whether there is a transmission request starting from the top row of the pixel unit 101, and sequentially read out the rows for which there is a transmission request.

[0041] Based on the event rate control signal input from the event rate control unit 109, the thinning unit 601 thins out the event signal E output from the arbitration unit 600 in the spatial direction (for example, row direction, column direction), time direction, etc., and outputs it.

[0042] An example of event rate control will now be described with reference to Fig. 5A. Here, it is assumed that the event rate in the region of interest of a vehicle imaged by the event-based imaging device 100 is set to 1, and the event rate in the region outside the region of interest is set to 0.2. Note that these event rate settings are merely examples, and can be set arbitrarily depending on the type of subject that is set as the region of interest, etc.

[0043] 5A, since the setting value of the event rate for the prediction region 502 is 1, the event rate control unit 109 controls the thinning processing unit 601 not to thin out the event signals of the region output from the arbitration unit 600. On the other hand, since the setting value of the event rate for the region other than the prediction region 502 is 0.2, the event rate control unit 109 controls the thinning processing unit 601 to output 20% of the event signals of the region output from the arbitration unit 600.

[0044] For example, the event rate control unit 109 instructs the thinning unit 601 to perform row-wise thinning. Specifically, the event rate control unit 109 notifies the row thinning rate C y , the column thinning rate is C x , the time decimation rate is C t Then, the thinning rate is adjusted so as to satisfy the following formula (2), and each thinning rate is notified to the thinning processing unit 601.

number

[0045] The thinning unit 601 thins out the event signals and time stamps of the rows and columns in a certain pattern according to the thinning rates adjusted as described above. As another example, the thinning unit 601 may execute thinning processing in a random pattern that stochastically determines whether or not to thin out each row, column, and time stamp.

[0046] In this embodiment, as described above, intermediate data of the processing of each processing unit is temporarily stored in the memory 105, but this is not a necessary configuration for implementing this embodiment, and each processing unit may directly exchange data without storing data output from each processing unit in the memory 105. For example, an SNN (Spiking Neural Network) can use an event signal generated as a time-series spike signal as an input signal, and can also be used for the object detection processing in this embodiment. Therefore, for example, the event data output by the event reading unit 103 may be directly input in chronological order to the attention area determination unit 106 that performs object detection based on the SNN. Similarly, the motion prediction unit 107, the prediction area determination unit 108, and the event rate control unit 109 can also realize the above processing without using the memory 105 by processing the input in chronological order.

[0047] In the above description, it is assumed that the attention region determining unit 106, the motion predicting unit 107, the prediction region determining unit 108, and the event rate control unit 109 are arranged inside the event-based imaging device 100. However, these processing units may be arranged outside the event-based imaging device 100. In this case, event data is transmitted from the event-based imaging device 100 to an information processing device (not shown) via an external interface, and the information processing device that receives the event data executes the above-mentioned prediction region determination process. Then, the event rate control signal is fed back to the event-based imaging device 100 side, thereby implementing the event rate control of the present invention. Note that it may be appropriately determined which processing units are arranged outside the event-based imaging device 100.

[0048] The above is a description of the event-based imaging device 100 according to this embodiment. In the event-based imaging device according to the conventional technology, when an event signal is generated, a signal is issued from the event detection circuit of each pixel to the arbitration unit requesting the arbitration unit to read out the event signal, and the arbitration unit selects the row to be read out. However, since the event signal is read out row by row, if a transmission request signal is issued from two or more rows, there will be pixels whose event signal is made to wait to be read out by the arbitration unit. In particular, when the event-based imaging device is mounted on a moving object, events can be detected from all positions within the angle of view of the event-based imaging device. Therefore, for example, if the number of rows of the pixel unit is 720, the maximum waiting time is equivalent to 720×H. Here, H is the time required to read out the events of one row of the pixel unit. For example, if H=1.2 μs, a waiting time (delay time) of just under 1 ms occurs in the above example.

[0049] Furthermore, in an event-based imaging device according to the prior art, even if the timestamp added to the event signal has a high time resolution, such as 1 μs, the time delay in reading out will result in an error in the timestamp. In other words, the timestamp has a time resolution equivalent to 1 ms, and the low latency performance of the event-based imaging device (as compared to a general image sensor) is not satisfactory. The frame rate (corresponding to the frame rate at which the image is captured) decreases. Also, artifacts due to time stamp errors (for example, an actual straight line is observed as a curved line) may occur. Artifacts can lead to, for example, a decrease in the accuracy of recognizing an object of interest in an event-based imaging device. Furthermore, if a next event occurs at the same pixel between the issuance of a transmission request signal and the reading of an event, the previous event signal may be lost.

[0050] However, according to the event-based imaging device 100 of the present embodiment, a prediction region is determined based on the movement of the event-based imaging device 100 and the target object of interest, and the target object of interest can be continuously captured within the prediction region. Therefore, in the pixel unit, event detection in a region not related to the target object of interest can be reduced, and event detection in a region related to the target object of interest can be performed with higher time resolution. As a result, the event-based imaging device 100 of the present embodiment can reduce the occurrence of time stamp errors and loss of event signals, which are concerns in event-based imaging devices according to conventional techniques. In addition, the event-based imaging device 100 of the present embodiment can reduce the amount of information by reducing event detection in a region not related to the target object of interest, and can also be expected to reduce power consumption associated with signal transmission.

[0051] <Second embodiment> Next, a description will be given of an event-based imaging device according to a second embodiment of the present invention. In the following description, the same components as those in the above embodiment are given the same reference numerals, and detailed description thereof will be omitted.

[0052] The configuration of the event-based imaging device 100 of this embodiment is similar to that of the event-based imaging device 100 of the first embodiment shown in Fig. 1. Fig. 7 shows an example of the configuration of the event readout unit 103 of the event-based imaging device 100 of this embodiment. As shown in Fig. 7, in the event readout unit 103 of this embodiment, an event rate control signal is input to an arbitration unit 600 and a thinning-out processing unit 601.

[0053] Fig. 8 shows an example of the configuration of the arbitration unit 600 of this embodiment. As shown in Fig. 8, the arbitration unit 600 employs a four-input, one-output configuration, and outputs an event signal for one selected row while controlling the order of readout from the four rows of the pixel unit 101. In the example shown in Fig. 8, the transmission request signals for the four rows are Req0, Req1, Req2, and Req3. Furthermore, the response signals corresponding to the respective transmission request signals are Ack0, Ack1, Ack2, and Ack3. Furthermore, the event signals corresponding to the respective transmission request signals are E0, E1, E2, and E3.

[0054] The arbitration circuit 800 employs a two-input, one-output configuration, and when transmission request signals Req0 and Req1 are input simultaneously, it selects one of the transmission request signals and outputs the selected transmission request signal (shown as Req01 in the figure) to the subsequent stage. Similarly, the arbitration circuit 801 selects one of the transmission request signals Req2 and Req3 and outputs the selected transmission request signal (shown as Req23 in the figure) to the subsequent stage.

[0055] The arbitration circuit 802 receives transmission request signals Req01 and Req23 from the arbitration circuits 800 and 801, respectively, and outputs corresponding response signals Ack01 and Ack23 to the arbitration circuits 800 and 801. The arbitration circuit 802 selects one of the transmission request signals Req01 and Req23, and returns a response signal Ack01 or Ack23 corresponding to the selected transmission request signal. The arbitration circuit 800 that receives Ack01 or the arbitration circuit 801 that receives Ack23 outputs to the pixel unit 101 any of the response signals Ack0, Ack1, Ack2, and Ack3 corresponding to the transmission request signal selected when outputting the transmission request signals Req01 and Req23. For example, when the arbitration circuit 800 selects the transmission request signal Req1 and outputs the transmission request signal Req01 to the subsequent stage, the arbitration circuit 800 outputs the response signal Ack1 corresponding to the transmission request signal Req1 to the pixel unit 101.

[0056] In this way, the response signal is returned in the reverse order of the transmission request signal relayed from the previous arbitration circuit, and the row to be read out is determined. Then, the event signals of the pixels for one row that received the response signal are stored in the buffer 803. The buffer 803 outputs the event signals for this one row in sequence to the subsequent stages.

[0057] In the example shown in Fig. 8, the arbitration circuits 800, 801, and 802 arbitrate the transmission request signals by selecting the transmission request signals according to the ratio of the priority values. As an example, in the arbitration circuit 800, the priority in the selection of the transmission request signals Req0 and Req1 is both 1, that is, the ratio of the priority values ​​is fixed at "Req0:Req1=1:1". Therefore, when both Req0 and Req1 are input, the arbitration circuit 800 performs arbitration so that Req0 and Req1 are output to the subsequent arbitration circuit 802 with the same frequency. Here, also in the arbitration circuit 801, the priority in the selection of the transmission request signals Req2 and Req3 is both 1, that is, the ratio of the priority values ​​is fixed at "Req2:Req3=1:1".

[0058] In addition, in the arbitration circuit 802, the priority in the selection of the transmission request signals Req01 and Req23 is assumed to be 10 for Req01 and 2 for Req23, i.e., the ratio of the priority values ​​is "Req01:Req23=10:2". Therefore, when both Req01 and Req23 are input, the arbitration circuit 802 arbitrates so that Req23 is selected two times while Req01 is selected ten times. As shown in the figure, the priority determination unit 804 sets the priority of each of the signals Req01 and Req23 of the arbitration circuit 802 based on the event rate control signal input from the event rate control unit 109.

[0059] In the above example, the event signals E0 and E1 of the top two rows of the pixel unit 101 are output from the buffer 803 of the arbitration unit 600 more frequently than the event signals E2 and E3 of the bottom two rows of the pixel unit 101.

[0060] The thinning-out processing unit 601 performs the same thinning-out processing in the column direction and the time direction as in the first embodiment. In the above example, it is assumed that the number of rows of the pixel unit 101 input to the arbitration unit 600 is four, but the number of rows may be any number. In addition, it is assumed that the arbitration circuit constituting the arbitration unit 600 has a two-stage configuration of the first-stage arbitration circuits 800, 8001 and the second-stage arbitration circuit 802, but the number of stages of the arbitration circuit may be changed according to the number of rows of the transmission request signals input from the pixel unit 101. In addition, it may be configured as a circuit in which three or more rows of transmission request signals are input to one arbitration circuit. Furthermore, the priority determination unit 804 performs priority setting only for the arbitration circuit 802, but in addition to this or instead of this, the priority determination unit 804 may be configured to perform priority setting for the arbitration circuit 800 and / or the arbitration circuit 801.

[0061] Also in this embodiment, similarly to the first embodiment, it is assumed that the transmission request signals Req0, Req1, Req2, and Req3 are input to the arbitration unit 600 in units of rows of the pixel unit 101. However, the transmission request signals may be input to the arbitration unit 600 in units of columns, units of rectangular regions defined by a certain width in the row direction and a certain width in the column direction, or for each pixel group of any unit in the pixel unit 101.

[0062] Furthermore, the event readout unit 103 of this embodiment may be configured without including the thinning-out unit 601. When the thinning-out unit 601 is not used, the arbitration unit 600 performs the above-mentioned event rate control based on the event rate control signal.

[0063] As described above, according to this embodiment, the readout frequency of the event signals detected in the pixel unit 101 can be controlled by the arbitration unit 600 for each pixel group (row unit, column unit, rectangular area unit, etc.) of the pixel unit 101. By thinning out the signals, some of the event signals input to the event readout unit 103 are discarded and not output to a subsequent stage. On the other hand, in this embodiment, the event signals that are not output from the event readout unit 103 to a subsequent stage are controlled by arbitration by the arbitration unit 600 so as not to be input to the event readout unit 103. As a result, the event-based imaging device 100 of this embodiment can further reduce the processing delay caused by reading out the event signals compared to the first embodiment.

[0064] <Third embodiment> Next, a description will be given of an event-based imaging device according to a second embodiment of the present invention. In the following description, the same components as those in the above embodiment are given the same reference numerals, and detailed description thereof will be omitted.

[0065] The configuration of the event-based imaging device 100 of this embodiment is similar to that of the event-based imaging device 100 of the first embodiment shown in Fig. 1. In this embodiment, the processing of the event rate control unit 109 is different from that of the event rate control unit 109 in the first embodiment. The processing of the event rate control unit 109 in this embodiment will be described with reference to Fig. 5.

[0066] In this embodiment, the event rate control unit 109 controls the event rate according to the area ratio between the pre-attention region 302 calculated by the attention region determination unit 106 and the prediction region 502 calculated by the prediction region determination unit 108 .

[0067] Specifically, a first event rate corresponding to the pre-attention region 302 is I1, a second event rate corresponding to the prediction region 502 is I2, the area of ​​the pre-attention region 302 is S1, and the area of ​​the prediction region 502 is S2. Note that the event rate I1 is an example of a first read frequency, and the event rate I2 is an example of a second read frequency. In this case, I2 can be calculated by the following formula (3).

number

[0068] This allows event rate control section 109 to perform control such that event rate I2 becomes smaller as area S2 relative to area S1 increases.

[0069] The method of controlling the event rate using the above formula (3) is just one example, and other calculation formulas and processing methods may be used as long as the event rate I2 is controlled to be smaller as the value of area S2 relative to area S1 increases.

[0070] In the example of FIG. 5, the faster the speed of the vehicle 301 moves, the larger the proportion of the prediction region 502 in the entire captured image becomes, and the larger the area S2 becomes. In this case, if the first event rate I1 is applied to the prediction region 502, the number of event signals generated by the pixel unit 101 increases, and there is a possibility that timestamp errors and unintended event loss may occur in event detection within the prediction region 502. However, in this embodiment, the event rate control unit 109 adaptively lowers the second event rate I2 according to the area S2, thereby making it possible to reduce the number of events detected by the pixel unit 101 compared to the first embodiment. In addition, in this embodiment, it is also expected to have the effect of reducing timestamp errors and event loss associated with event detection.

[0071] <Fourth embodiment> Any of the first to third embodiments can be applied to the fourth embodiment. 9 is a schematic diagram for explaining a device 9191 including a semiconductor device 930. The semiconductor device 930 can be any one of the event-based imaging devices explained in the first to third embodiments, or an event-based imaging device combining a plurality of the embodiments. The device 9191 including the semiconductor device 930 will be explained in detail. The semiconductor device 930 can include a package 920 that houses the semiconductor device 910 in addition to a semiconductor device 910 having a semiconductor layer. The package 920 can include a base to which the semiconductor device 910 is fixed, and a cover such as glass that faces the semiconductor device 910. The package 920 can further include a bonding member such as a bonding wire or a bump that connects a terminal provided on the base and a terminal provided on the semiconductor device 910.

[0072] The device 9191 can include at least one of an optical device 940, a control device 950, a processing device 960, a display device 970, a storage device 980, and a mechanical device 990. The optical device 940 corresponds to the semiconductor device 930. The optical device 940 is, for example, a lens, a shutter, or a mirror. The control device 950 controls the semiconductor device 930. The control device 950 is, for example, a semiconductor device such as an ASIC.

[0073] The processing device 960 processes the signal output from the semiconductor device 930. The processing device 960 is a semiconductor device such as a CPU or ASIC for configuring an AFE (analog front end) or a DFE (digital front end). The display device 970 is an EL display device or a liquid crystal display device that displays information (images) obtained by the semiconductor device 930. The storage device 980 is a magnetic device or a semiconductor device that stores information (images) obtained by the semiconductor device 930. The storage device 980 is a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive.

[0074] The mechanical device 990 has a moving part or a propulsion part such as a motor or an engine. In the device 9191, the signal output from the semiconductor device 930 is displayed on the display device 970, or transmitted to the outside by a communication device (not shown) included in the device 9191. For this purpose, the device 9191 preferably further includes a memory device 980 and a processing device 960 in addition to the memory circuit and arithmetic circuit included in the semiconductor device 930. The mechanical device 990 may be controlled based on the signal output from the semiconductor device 930.

[0075] The device 9191 is also suitable for electronic devices such as information terminals (e.g., smartphones and wearable devices) with a photographing function and cameras (e.g., interchangeable lens cameras, compact cameras, video cameras, and surveillance cameras). The mechanical device 990 in the camera can drive components of the optical device 940 for zooming, focusing, and shutter operation. Alternatively, the mechanical device 990 in the camera can move the semiconductor device 930 for vibration isolation operation.

[0076] The device 9191 may be transportation equipment such as a vehicle, a ship, or an aircraft. The mechanical device 990 in the transportation equipment may be used as a moving device. The device 9191 as a transportation equipment is suitable for transporting the semiconductor device 930 or for assisting and / or automating driving (piloting) by using a photographing function. The processing device 960 for assisting and / or automating driving (piloting) can perform processing for operating the mechanical device 990 as a moving device based on information obtained by the semiconductor device 930. Alternatively, the device 9191 may be a medical device such as an endoscope, a measuring device such as a distance measuring sensor, an analytical device such as an electron microscope, an office machine such as a copier, or an industrial device such as a robot.

[0077] According to the fourth embodiment described above, it is possible to obtain good pixel characteristics. Therefore, the value of the semiconductor device 930 can be increased. Increasing the value here means adding functions, improving performance, improving characteristics, improving reliability, improving manufacturing yield, reducing environmental impact, and the like. This includes at least one of the following: reduction in size, cost reduction, size reduction, and weight reduction.

[0078] Therefore, if the semiconductor device 930 according to the fourth embodiment is used in the equipment 9191, the value of the equipment can be improved. For example, by mounting the semiconductor device 930 on a transport equipment, excellent performance can be obtained when photographing the outside of the transport equipment or measuring the external environment. Therefore, in manufacturing and selling the transport equipment, it is advantageous to decide to mount the semiconductor device 930 according to the fourth embodiment on the transport equipment in order to improve the performance of the transport equipment itself. In particular, the semiconductor device 930 is suitable for transport equipment that performs driving assistance and / or automatic driving of the transport equipment using information obtained by the semiconductor device 930.

[0079] Each of the above-described embodiments can be modified as appropriate without departing from the technical concept. The disclosure of this specification includes not only what is described in this specification, but also all matters that can be understood from this specification and the drawings attached hereto. The disclosure of this specification also includes the complement of the concepts described in this specification. In other words, if this specification contains a statement that "A is greater than B," for example, this specification can be said to disclose that "A is not greater than B" even if the statement that "A is not greater than B" is omitted. This is because when it is stated that "A is greater than B," it is assumed that the case where "A is not greater than B" is taken into consideration.

[0080] The disclosure of this embodiment includes the following configuration. (Configuration 1) a pixel section having a plurality of unit pixels and configured to detect a change in a light amount for each of the unit pixels as an event signal; an event readout unit that reads out the event signal from the pixel unit; an output unit configured to output, for at least one region in the pixel unit associated with the event signal, motion information related to motion of the at least one region; a determination unit that determines a predicted region after the at least one region is changed, the predicted region being predicted based on the motion information; having The event reading unit reads the event signal from the determined prediction region. 1. An imaging device comprising: (Configuration 2) A control unit is further configured to control a read frequency of the event signal for one region or each of a plurality of regions of the at least one region based on the determined prediction region, The event reading unit reads the event signal from the prediction region of the one region of the at least one region or each of the plurality of regions based on the reading frequency controlled by the control unit. 2. The imaging device according to configuration 1, (Configuration 3) The imaging device according to the configuration 1 or 2, wherein a time period during which the determination unit determines the prediction region is greater than a time resolution for detecting the event signal by the pixel unit. (Configuration 4) 4. The imaging device according to any one of configurations 1 to 3, wherein the motion information includes at least one of a type of motion and an amount of motion of the at least one region. (Configuration 5) The imaging device according to configuration 4, wherein the type of movement includes at least one of horizontal movement, vertical movement, yaw, pitch, zoom, and rotation of the imaging device. (Configuration 6) 6. The imaging device according to claim 4 or 5, wherein the amount of movement includes at least one of a speed of movement and an acceleration of movement in the imaging device. (Configuration 7) The determination unit is a first region determination unit that determines at least one region in the pixel unit based on the event signal read by the event readout unit; a second region determination unit that determines a prediction region after the change of the at least one region based on the at least one region determined by the first region determination unit and the motion information; 7. The imaging device according to any one of configurations 1 to 6, comprising: (Configuration 8) the event readout unit generates event data using the event signal and a coordinate value indicating a position of the unit pixel that outputs the event signal; The determination unit is a first region determination unit that determines at least one region in the pixel unit using the event data; a second region determination unit that determines a prediction region after the change of the at least one region based on the at least one region determined by the first region determination unit and the motion information; 7. The imaging device according to any one of configurations 1 to 6, comprising: (Configuration 9) the output unit outputs a predicted region of the first region predicted based on the first region determined by the first region determination unit at a time earlier than a current time and the motion information output by the output unit in the event signal read by the event read unit between the past time and the current time; The second region determination unit determines a region including the at least one region determined by the first region determination unit at the current time and the prediction region after the change of the first region as a prediction region after the change of the at least one region. 8. The imaging device according to configuration 7. (Configuration 10) the output unit outputs a predicted region of the first region predicted based on a first region determined by the first region determination unit at a time earlier than a current time and the movement information output by the output unit in the event data generated by the event reading unit between the earlier time and the current time; The second region determination unit determines a region including the at least one region determined by the first region determination unit at the current time and the prediction region after the change of the first region as a prediction region after the change of the at least one region. 9. The imaging device according to configuration 8, (Configuration 11) The output unit outputs a reliability of the predicted region after the change of the at least one region determined by the determination unit; The second region determination unit adjusts a prediction region after the change of the at least one region based on the reliability. 8. The imaging device according to configuration 7. (Configuration 12) The output unit outputs a reliability of the predicted region after the change of the at least one region determined by the determination unit; The second region determination unit adjusts a prediction region after the change of the at least one region based on the reliability. 9. The imaging device according to configuration 8, (Configuration 13) The second region determination unit performs an adjustment such that the area of ​​the predicted region after the change of the at least one region becomes larger as the reliability becomes lower. 12. The imaging device according to claim 11, (Configuration 14) The second region determination unit performs an adjustment such that the area of ​​the predicted region after the change of the at least one region becomes larger as the reliability becomes lower. 13. The imaging device according to configuration 12, (Configuration 15) The imaging device described in configuration 11, characterized in that the reliability decreases as the difference between the predicted area of ​​the first area output by the output unit and the at least one area determined by the first area determination unit at the current time increases. (Configuration 16) 16. The imaging device of claim 1, wherein the event readout unit thins out and reads out the event signals in at least one of the spatial and temporal directions based on a readout frequency for each of the at least one region. (Configuration 17) an arbitration unit that arbitrates an order in which event signals are read out from a plurality of pixel groups each including one or more unit pixels; The control unit controls a frequency of reading the event signal based on the order arbitrated by the arbitration unit. 3. The imaging device according to configuration 2. (Configuration 18) A control unit is further configured to control a read frequency of the event signal for one region or each of a plurality of regions of the at least one region based on the determined prediction region, The imaging device described in configuration 7 or 8, characterized in that the control unit assigns a first readout frequency to the at least one region determined by the first region determination unit, and assigns a second readout frequency, which is obtained by adjusting the first readout frequency in accordance with an area ratio between the at least one region and a prediction region of the at least one region determined by the second region determination unit, to the prediction region of the at least one region. (Configuration 19) The imaging device described in configuration 18, characterized in that the control unit reduces the second readout frequency as the area of ​​the prediction region of the at least one region determined by the second region determination unit relative to the at least one region determined by the first region determination unit becomes larger. (Configuration 20) An apparatus including the imaging device according to any one of configurations 1 to 19, an optical device corresponding to the imaging device; A control device for controlling the imaging device; a processing device that processes a signal output from the imaging device; a display device for displaying information obtained by the imaging device; a storage device that stores information obtained by the imaging device; and and a mechanical device that operates based on information obtained by the imaging device. [Explanation of symbols]

[0081] 100 Event-based imaging device, 101 Pixel unit, 103 Event readout unit, 107 Motion prediction unit, 108 Prediction region determination unit, 110 Unit pixel

Claims

1. A pixel unit having multiple unit pixels, which detects changes in light intensity as an event signal for each unit pixel, An event readout unit that reads the event signal from the pixel unit, An output unit that outputs motion information relating to the movement of at least one region in the pixel unit that is related to the event signal, A determination unit that determines the predicted region after the change of at least one region, based on the motion information, It has, The event readout unit reads out the event signal from the unit pixels included in the determined prediction region. An imaging device characterized by the following features.

2. The system further includes a control unit that controls the readout frequency of the event signal for each of the at least one region or more regions of the predicted region, based on the determined predicted region. The event reading unit reads the event signal from the unit pixels included in the prediction region of each of the at least one region or the plurality of regions, based on the reading frequency controlled by the control unit. The imaging apparatus according to feature 1.

3. The imaging apparatus according to claim 1, characterized in that the time period for determining the prediction region by the determination unit is greater than the time resolution for detecting the event signal by the pixel unit.

4. The imaging apparatus according to claim 1, characterized in that the motion information includes at least one of the type of motion and the amount of motion in the at least one region.

5. The imaging apparatus according to claim 4, characterized in that the type of movement includes at least one of horizontal movement, vertical movement, yaw, pitch, zoom, and rotation in the imaging apparatus.

6. The imaging apparatus according to claim 4, characterized in that the amount of motion includes at least one of the velocity of motion and the acceleration of motion in the imaging apparatus.

7. The aforementioned determination unit, A first region determination unit determines at least one region in the pixel unit based on the event signal read out by the event readout unit, A second region determination unit determines a predicted region after a change in the at least one region based on the at least one region determined by the first region determination unit and the motion information, The imaging device according to claim 1, characterized by having the following features.

8. The event reading unit generates event data using the event signal and the coordinate value indicating the position of the unit pixel that output the event signal. The aforementioned determination unit, A first region determination unit that determines at least one region in the pixel portion using the event data, A second region determination unit determines a predicted region after a change in the at least one region based on the at least one region determined by the first region determination unit and the motion information, The imaging device according to claim 1, characterized by having the following features.

9. The output unit outputs a predicted region of the first region, which is predicted based on the first region determined by the first region determination unit at a time prior to the current time, and the motion information output by the output unit in the event signals read by the event reading unit between the past time and the current time. The second region determination unit determines a region including the at least one region determined by the first region determination unit at the current time and the predicted region after the change of the first region as the predicted region after the change of the at least one region. The imaging apparatus according to feature 7.

10. The output unit outputs a predicted region of the first region, which is predicted based on the first region determined by the first region determination unit at a time prior to the current time, and the motion information output by the output unit in the event data generated by the event reading unit between the past time and the current time. The second region determination unit determines a region including the at least one region determined by the first region determination unit at the current time and the predicted region after the change of the first region as the predicted region after the change of the at least one region. The imaging apparatus according to feature 8.

11. The output unit outputs the confidence level of the predicted region after the change of the at least one region determined by the determination unit. The second region determination unit adjusts the predicted region after the change of at least one region based on the confidence level. The imaging apparatus according to feature 7.

12. The output unit outputs the confidence level of the predicted region after the change of the at least one region determined by the determination unit. The second region determination unit adjusts the predicted region after the change of at least one region based on the confidence level. The imaging apparatus according to feature 8.

13. The second region determination unit performs an adjustment such that the area of ​​the predicted region after the change in at least one region increases as the reliability decreases. The imaging apparatus according to feature 11.

14. The second region determination unit performs an adjustment such that the area of ​​the predicted region after the change in at least one region increases as the reliability decreases. The imaging apparatus according to feature 12.

15. The imaging apparatus according to claim 11, characterized in that the reliability decreases as the difference between the predicted region of the first region output by the output unit and the at least one region determined by the first region determination unit at the current time increases.

16. The imaging apparatus according to claim 1, characterized in that the event readout unit reads out the event signals by thinning them in at least one direction, either spatially or temporally, based on the readout frequency for each of the at least one region.

17. It further includes an arbitration unit that arbitrates the order in which event signals are read from multiple pixel groups consisting of one or more unit pixels, The control unit controls the readout frequency of the event signals based on the order arbitrated by the arbitration unit. The imaging device according to feature 2.

18. The system further includes a control unit that controls the readout frequency of the event signal for each of the at least one region or more regions of the predicted region, based on the determined predicted region. The imaging apparatus according to claim 7, characterized in that the control unit assigns a first readout frequency to the at least one region determined by the first region determination unit, and assigns a second readout frequency to the predicted region of the at least one region, which is obtained by adjusting the first readout frequency according to the area ratio of the at least one region and the predicted region of the at least one region determined by the second region determination unit.

19. The imaging apparatus according to claim 18, characterized in that the control unit lowers the second readout frequency as the area of ​​the predicted region of the at least one region determined by the determination unit increases.

20. An apparatus comprising an imaging device according to any one of claims 1 to 19, An optical device corresponding to the aforementioned imaging device, Control device for controlling the imaging device, A processing unit that processes the signal output from the imaging device, A display device that displays information obtained by the aforementioned imaging device. A storage device for storing information obtained by the imaging device, and The apparatus is characterized by further comprising at least one of the following: a mechanical device that operates based on information obtained by the imaging device.

Images