Imaging apparatus and imaging method

Abstract

[Problem] To reliably and effectively detect an event. [Solution] The present invention provides an imaging device including: a plurality of photoelectric conversion elements each photoelectrically converting incident light into an electric signal; a plurality of detection units each outputting a detection signal when an absolute value of a variation amount of the electric signal generated by each of the plurality of photoelectric conversion elements exceeds a predetermined threshold value; and a threshold value adjustment unit that adjusts the threshold value based on a detection state of the detection signal in each of the plurality of detection units.

Description

Technical Field

[0001] This disclosure relates to imaging apparatus and imaging methods. Background Technology

[0002] An imaging device is known that acquires data only on the portion of the brightness level that changes due to an event when the event occurs in the imaging scene. This type of imaging device can be called an event-based vision sensor (EVS).

[0003] Reference List

[0004] Patent documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2017-535999. Summary of the Invention

[0006] The problem to be solved by the present invention

[0007] In traditional EVS, since the event detection threshold is essentially fixed, the number of events detected can increase or decrease dramatically depending on the imaging scene. For example, in an imaging scene where flickering occurs, there is a possibility that events other than flickering might be ignored as a result of detecting numerous noise events caused by flickering. Furthermore, in imaging scenes in dark places such as at night, there is a possibility that objects that have entered the imaging scene may not be accurately detected due to the small number of events detected.

[0008] Furthermore, traditional EVS detects events in all pixels, and when events occur only in a portion of the pixel area, there is a possibility that detecting events may take time or that events in that portion of the pixel area may not be detected.

[0009] Therefore, this disclosure provides an imaging apparatus and imaging method capable of reliably and effectively detecting events.

[0010] Solution to the problem

[0011] To address the aforementioned problems, according to this disclosure, an imaging apparatus is provided, comprising:

[0012] Multiple photoelectric conversion elements, each configured to photoelectrically convert incident light to generate an electrical signal.

[0013] Multiple detectors, each configured to output a detection signal if the absolute value of the change in the electrical signal generated by each of the multiple photoelectric conversion elements exceeds a predetermined threshold; and

[0014] The threshold adjustment unit is configured to adjust the threshold based on the detection status of the corresponding detection signals from multiple detectors.

[0015] The detection status of multiple detectors may include at least one of the following: the number of detection signals detected in a predetermined area within a predetermined period, the ratio of the number of detectors that output detection signals to the total number of detectors in the predetermined area, and the signal level of the detection signals.

[0016] The threshold adjustment unit can adjust the threshold in at least one of the following situations: the number of detected signals in a predetermined area within a predetermined period is greater than or equal to a first reference value, and the number of detected signals in a predetermined area within a predetermined period is less than a second reference value.

[0017] If the number of detected signals in a predetermined area within a predetermined period is equal to or greater than the first reference value, the threshold adjustment unit can adjust the threshold to make the detector's detection sensitivity low.

[0018] If the number of detected signals in a predetermined area within a predetermined period is less than the second reference value, the threshold adjustment unit can adjust the threshold to make the detector have high detection sensitivity.

[0019] The threshold may include a first threshold and a second threshold.

[0020] The detector may include:

[0021] The first detector is configured to detect the first detection signal when the absolute value of the change in the electrical signal along the increasing direction exceeds a first threshold.

[0022] The second detector is configured to detect a second detection signal when the absolute value of the change in the electrical signal along a decreasing direction exceeds a second threshold; and

[0023] The threshold adjustment unit can adjust the first threshold and the second threshold based on the detection status in the first detector and the second detector.

[0024] The imaging apparatus may further include: a first determining unit configured to determine whether the detection status in the first detector is within a first permissible range, and

[0025] The second determining unit is configured to determine whether the detection status in the second detector is within a second permissible range, wherein...

[0026] When the first determining unit determines that the detection condition is not within the first allowable range, the threshold adjustment unit may adjust the first threshold, and when the second determining unit determines that the detection condition is not within the second allowable range, the threshold adjustment unit may adjust the second threshold.

[0027] According to this disclosure, an imaging apparatus is provided, comprising:

[0028] Multiple first photoelectric conversion elements, each configured to photoelectrically convert incident light to generate an electrical signal.

[0029] Multiple detectors, each configured to output a detection signal if the absolute value of the change in the electrical signal generated by each of the multiple first photoelectric conversion elements exceeds a predetermined threshold; and

[0030] The region extraction unit is configured to extract a portion of the viewing area within which multiple first photoelectric conversion elements perform photoelectric conversion based on the detection status of detection signals from multiple detectors.

[0031] The region extraction unit can extract a portion of the region based on the position of the output detection signal within the viewpoint of multiple first photoelectric conversion elements performing photoelectric conversion.

[0032] The imaging apparatus may include: an object recognition unit configured to recognize an object existing within a viewing angle where photoelectric conversion is performed by a plurality of first photoelectric conversion elements based on a detection status from detection signals from a plurality of detectors, wherein a region extraction unit extracts a portion of the object recognized by the object recognition unit.

[0033] The imaging apparatus may include a notification unit configured to notify at least one of a plurality of first photoelectric conversion elements and a plurality of detectors of information about a portion of a region.

[0034] The notification unit can notify the detectors of the appropriate number of detection signals to be detected by multiple detectors, as well as information about a portion of the area.

[0035] Multiple detectors can adjust the threshold based on an appropriate amount of information about the number of detection signals notified by the notification unit.

[0036] The imaging device may further include: an information processing unit configured to generate operating condition information based on detection signals from multiple detectors, the operating condition information including at least one of the following: photoelectric conversion speeds of multiple first photoelectric conversion elements, the ratio of the first photoelectric conversion element performing the photoelectric conversion to the multiple first photoelectric conversion elements, and activation frequencies of the multiple first photoelectric conversion elements.

[0037] The notification unit can notify at least one of the multiple first photoelectric conversion elements and multiple detectors of operating condition information.

[0038] The region extraction unit can output event information based on the detection signal output from the detector associated with the first photoelectric conversion element located in the partial region.

[0039] The region extraction unit can output information about the coordinates of a portion of the region, as well as event information.

[0040] The imaging device may include an imaging unit comprising a plurality of second photoelectric conversion elements, each second photoelectric conversion element photoelectrically converting incident light to generate an electrical signal, and the imaging unit being configured to output image data in a portion region based on the electrical signal.

[0041] The imaging apparatus may include an image correction unit configured to correct image data based on event information, which is based on a detection signal output from a detector associated with a first photoelectric conversion element located in a portion of the region.

[0042] According to this disclosure, an imaging method is provided, comprising:

[0043] A detection signal is output when the absolute value of the change in the electrical signal generated by multiple photoelectric conversion elements exceeds a predetermined threshold. Each of these multiple photoelectric conversion elements photoelectrically converts the incident light to generate an electrical signal.

[0044] The threshold is adjusted based on the detection status of the detection signal.

[0045] The method may further include: extracting a portion of the viewing area within which multiple photoelectric conversion elements perform photoelectric conversion based on the detection status of the detection signal. Attached Figure Description

[0046] Figure 1 This is a block diagram illustrating an example system configuration of an imaging system applying the technology according to this disclosure.

[0047] Figure 2 This is a block diagram illustrating an example configuration of an imaging apparatus according to a first configuration example of this disclosure.

[0048] Figure 3 This is a block diagram illustrating an example configuration of pixel array units.

[0049] Figure 4 This is a circuit diagram showing an example of the circuit configuration of a pixel.

[0050] Figure 5 This is a block diagram illustrating a first configuration example of an address event detector.

[0051] Figure 6 This is a circuit diagram illustrating an example configuration of the current-to-voltage conversion unit in an address event detector.

[0052] Figure 7 This is a circuit diagram showing an example configuration of the subtractor and quantizer in an address event detector.

[0053] Figure 8 This is a block diagram illustrating a second configuration example of the address event detector.

[0054] Figure 9 This is a block diagram illustrating an example configuration of an imaging apparatus according to a second configuration example of this disclosure.

[0055] Figure 10 It is an exploded perspective view schematically showing the stacked chip structure of the imaging device.

[0056] Figure 11 This is a block diagram illustrating an example configuration of the column processing unit of an imaging apparatus according to a first configuration example.

[0057] Figure 12 This is a block diagram illustrating the internal configuration of the signal processing unit according to the first embodiment.

[0058] Figure 13 This is a flowchart illustrating the processing operations of the pixel array unit and the signal processing unit according to the first embodiment.

[0059] Figure 14 This is an illustration showing an example of setting multiple pixel regions in an imaging scene.

[0060] Figure 15 It is shown Figure 9 A flowchart of the event counting process in the imaging device of the scanning method.

[0061] Figure 16 It is shown Figure 9 A flowchart illustrating the modification of the event counting processing procedure in a scanning imaging device.

[0062] Figure 17 It is shown Figure 2 A flowchart of the event counting process in the asynchronous imaging device.

[0063] Figure 18 This is a diagram illustrating an example of a first captured image used to adjust the threshold of the address event detector, where the entire imaging scene is initially set to a predetermined range.

[0064] Figure 19 This is an illustration of an example of a second captured image used to adjust the first and second reference values ​​for a portion of a pixel region in an imaging scene.

[0065] Figure 20 This is a flowchart illustrating the processing operation of the signal processing unit according to the second embodiment.

[0066] Figure 21 It is a diagram used to illustrate the permissible range.

[0067] Figure 22 This is a block diagram illustrating the internal configuration of the signal processing unit according to the third embodiment.

[0068] Figure 23 This is a flowchart of the processing operation of the signal processing unit according to the third embodiment.

[0069] Figure 24 This is a block diagram illustrating the connection relationship between the signal processing unit and the CPU according to the fourth embodiment.

[0070] Figure 25 It is shown Figure 24 The flowchart shows the signal processing unit and CPU processing operations.

[0071] Figure 26 This is a diagram showing examples of objects included in an imaging scene.

[0072] Figure 27 This is a block diagram illustrating the connection relationship between the signal processing unit and the CPU according to the fifth embodiment.

[0073] Figure 28 It is shown Figure 27 The flowchart shows the signal processing unit and CPU processing operations.

[0074] Figure 29 This is a block diagram illustrating the connection relationship between the signal processing unit and the CPU according to the sixth embodiment.

[0075] Figure 30 It is shown Figure 29 The flowchart shows the signal processing unit and CPU processing operations.

[0076] Figure 31 This is a block diagram illustrating a schematic configuration of an imaging system according to the seventh embodiment.

[0077] Figure 32 This is a block diagram illustrating the connection relationship between the signal processing unit and the CPU according to the seventh embodiment.

[0078] Figure 33 It is shown Figure 32 The flowchart shows the signal processing unit and CPU processing operations.

[0079] Figure 34 This is a block diagram illustrating a schematic configuration example of a vehicle control system, which is an example of a mobile body control system to which the technology according to this disclosure can be applied.

[0080] Figure 35 This is a diagram showing an example of the mounting location of the imaging unit. Specific Implementation

[0081] In the following description, embodiments of the imaging apparatus and imaging method will be described with reference to the accompanying drawings. Although the main components of the imaging apparatus and imaging method will be described primarily below, the imaging apparatus and imaging method may have components and functions not shown or described. The following description does not exclude components and functions not shown or described.

[0082] Figure 1 This is a block diagram illustrating an example system configuration of an imaging system applying the technology according to this disclosure.

[0083] like Figure 1 As shown, the imaging system 10 applying the technology of this disclosure includes an imaging lens 11, an imaging device 20, a recording unit 12, a controller 13, and a CPU 55. The imaging system 10 is an example of the electronic device of this disclosure, and examples of electronic devices include camera systems mounted on industrial robots, vehicle-mounted camera systems, etc.

[0084] In the imaging system 10 with the above configuration, the imaging lens 11 captures incident light from the object and forms an image on the imaging surface of the imaging device 20. The imaging device 20 photoelectrically converts the incident light captured by the imaging lens 11 in pixels to obtain imaging data. The imaging device of this disclosure, described later, is used as the imaging device 20.

[0085] Imaging device 20 performs predetermined signal processing (such as image recognition processing) on ​​the captured image data to output data representing the processing result and a detection signal of an address event (hereinafter referred to simply as the "detection signal"), described later, to recording unit 12. The method for generating the detection signal of the address event will be described later. Recording unit 12 stores data provided from imaging device 20 via signal line 14. Controller 13 includes, for example, a microcomputer and controls the imaging operations in imaging device 20. CPU 55 performs various types of information processing based on the event signals output from imaging device 20. Note that CPU 55 may be located inside imaging device 20.

[0086] [Imaging device (arbitration method) according to the first configuration example]

[0087] Figure 2 This is a block diagram illustrating an example configuration of an imaging device according to a first configuration example, which is used as an imaging device 20 in an imaging system 10 applying the technology according to this disclosure.

[0088] like Figure 2 As shown, the imaging device 20, which is a first configuration example of the imaging device of this disclosure, is an asynchronous imaging device called EVS, and includes a pixel array unit 21, a driving unit 22, an arbitration unit (arbitration unit) 23, a column processing unit 24, and a signal processing unit 25.

[0089] In the imaging apparatus 20 with the above configuration, a plurality of pixels 30 are arranged in a matrix (array) two-dimensionally in pixel array unit 21. The vertical signal line VSL, which will be described later, is wired relative to this matrix-like pixel array for each pixel column.

[0090] Each of the plurality of pixels 30 generates an analog signal corresponding to the voltage of the photocurrent as a pixel signal. Furthermore, each of the plurality of pixels 30 detects the presence or absence of an address event based on whether the change in photocurrent exceeds a predetermined threshold. Then, when an address event occurs, pixel 30 outputs a request to arbitration unit 23.

[0091] The driving unit 22 drives each of the plurality of pixels 30 to output the pixel signal generated in each pixel 30 to the column processing unit 24.

[0092] Arbitration unit 23 arbitrates requests from each of the plurality of pixels 30 and sends a response to pixel 30 based on the arbitration result. Pixel 30 receiving the response from arbitration unit 23 provides a detection signal (an address event detection signal) indicating the detection result to drive unit 22 and signal processing unit 25. Reading the detection signal from pixel 30 can be performed by reading multiple rows.

[0093] The column processing unit 24 includes, for example, an analog-to-digital converter, and performs a process for each pixel column of the pixel array unit 21 to convert the analog pixel signal output from the pixel 30 of the column into a digital signal. Then, the column processing unit 24 provides the analog-to-digital converted digital signal to the signal processing unit 25.

[0094] The signal processing unit 25 performs predetermined signal processing on the digital signal provided by the column processing unit 24, such as correlated double sampling (CDS) processing or image recognition processing. Then, the signal processing unit 25 provides data indicating the processing result and the detection signal provided by the arbitrator unit 23 to the recording unit 12 via signal line 14 (see [link to recording unit 12]). Figure 1 ).

[0095] [Pixel array unit configuration example]

[0096] Figure 3 This is a block diagram illustrating an example configuration of pixel array unit 21.

[0097] In the pixel array unit 21 in which multiple pixels 30 are arranged in a matrix two-dimensional arrangement, each of the multiple pixels 30 includes a light receiving unit 31, a pixel signal generating unit 32, and an address event detector 33.

[0098] In the pixel 30 with the above configuration, the light receiving unit 31 photoelectrically converts the incident light to generate a photocurrent. Then, under the control of the driving unit 22, the light receiving unit 31 provides the photocurrent generated by the photoelectric conversion to either the pixel signal generation unit 32 or the address event detector 33 (see [link to relevant documentation]). Figure 2 ).

[0099] The pixel signal generation unit 32 generates a signal corresponding to the voltage of the photocurrent provided from the light receiving unit 31 as a pixel signal SIG, and provides the generated pixel signal SIG to the column processing unit 24 via the vertical signal line VSL (see [link]). Figure 2 ).

[0100] The address event detector 33 detects the presence or absence of an address event based on whether the change in photocurrent from each optical receiving unit 31 exceeds a predetermined threshold. Address events include, for example, a turn-on event indicating that the change in photocurrent exceeds an upper threshold and a turn-off event indicating that the change falls below a lower threshold. Furthermore, the address event detection signal includes, for example, a bit indicating the detection result of a turn-on event and a bit indicating the detection result of a turn-off event. Note that the address event detector 33 can be configured to detect only turn-on events.

[0101] When an address event occurs, the address event detector 33 provides a request to the arbitration unit 23 to send a detection signal for the address event (see [link]). Figure 2 Then, when a response to the request is received from the arbitration unit 23, the address event detector 33 provides the address event detection signal to the drive unit 22 and the signal processing unit 25.

[0102] [Exemplary circuit configuration for a pixel]

[0103] Figure 4 This is a circuit diagram illustrating an example of the circuit configuration of pixel 30. As described above, each of the plurality of pixels 30 includes a light receiving unit 31, a pixel signal generating unit 32, and an address event detector 33.

[0104] In the pixel 30 with the above configuration, the light receiving unit 31 includes a light receiving element (photoelectric conversion element) 311, a transmission transistor 312, and an overcurrent gate (OFG) transistor 313. For example, N-type metal-oxide-semiconductor (MOS) transistors are used as the transmission transistor 312 and the OFG transistor 313. The transmission transistor 312 and the OFG transistor 313 are connected in series with each other.

[0105] The light receiving element 311 is connected between the common connection node N1 of the transmission transistor 312 and the OFG transistor 313 and ground, and photoelectrically converts the incident light to generate a charge corresponding to the amount of incident light.

[0106] Transmit signal TRG from Figure 2 The driving unit 22 shown provides power to the gate electrode of the transmission transistor 312. In response to the transmission signal TRG, the transmission transistor 312 provides the charge converted by the light receiving element 311 to the pixel signal generation unit 32.

[0107] A control signal OFG is provided from the driving unit 22 to the gate electrode of the OFG transistor 313. In response to the control signal OFG, the OFG transistor 313 provides an electrical signal generated by the photoreceiving element 311 to the address event detector 33. The electrical signal provided to the address event detector 33 is a photocurrent including charge.

[0108] The pixel signal generation unit 32 includes a reset transistor 321, an amplification transistor 322, a selection transistor 323, and a floating diffusion layer 324. For example, an N-type MOS transistor can be used as the reset transistor 321, the amplification transistor 322, and the selection transistor 323.

[0109] The charge converted by the light receiving element 311 from the light receiving unit 31 is supplied to the pixel signal generating unit 32 by the transfer transistor 312. The charge supplied from the light receiving unit 31 accumulates in the floating diffusion layer 324. The floating diffusion layer 324 generates a voltage signal with a voltage value corresponding to the amount of accumulated charge. That is, the floating diffusion layer 324 converts charge into voltage.

[0110] Reset transistor 321 is connected to the power supply voltage V. DD The power supply line is between the floating diffusion layer 324 and the power supply line. The reset signal RST is provided from the drive unit 22 to the gate electrode of the reset transistor 321. The reset transistor 321 initializes (resets) the charge of the floating diffusion layer 324 in response to the reset signal RST.

[0111] Amplifying transistor 322 and selecting transistor 323 are connected in series at power supply voltage V. DD Between the power supply line and the vertical signal line VSL. The amplifying transistor 322 amplifies the voltage signal that has undergone charge-voltage conversion by the floating diffusion layer 324.

[0112] The selection signal SEL is provided from the driving unit 22 to the gate electrode of the selection transistor 323. In response to the selection signal SEL, the selection transistor 323 outputs the voltage signal amplified by the amplifying transistor 322 as a pixel signal SIG to the column processing unit 24 via the vertical signal line VSL (see [link to column processing unit 24]). Figure 2 ).

[0113] In the imaging apparatus 20, which includes a pixel array unit 21 with pixels 30 arranged in a two-dimensional configuration as described above, when an address event is detected... Figure 1When the controller 13 indicates, the drive unit 22 provides the control signal OFG to the OFG transistor 313 of the optical receiver unit 31, thereby driving the OFG transistor 313 to provide photocurrent to the address event detector 33.

[0114] Then, when an address event is detected in a specific pixel 30, the driving unit 22 turns off the OFG transistor 313 of the pixel 30 and stops providing photocurrent to the address event detector 33. Next, the driving unit 22 drives the transmission transistor 312 by providing the transmission signal TRG to the transmission transistor 312, and transfers the charge photoelectrically converted by the light receiving element 311 to the floating diffusion layer 324.

[0115] In this way, the imaging apparatus 20, which includes a pixel array unit 21 with pixels 30 arranged in a two-dimensional configuration as described above, outputs only the pixel signals of the pixels 30 that have detected address events to the column processing unit 24. Therefore, compared to the case where pixel signals of all pixels are output regardless of the presence or absence of address events, the power consumption and image processing workload of the imaging apparatus 20 can be reduced.

[0116] Note that the configuration of pixel 30 illustrated here is an example and is not limited to this configuration example. For example, the pixel configuration may not include the pixel signal generation unit 32. In the case of this pixel configuration, the OFG transistor 313 is omitted in the light receiving unit 31, and only the transmission transistor 312 needs to have the function of the OFG transistor 313.

[0117] [First Configuration Example of Address Event Detector]

[0118] Figure 5 This is a block diagram illustrating a first configuration example of the address event detector 33. (See diagram for example.) Figure 5 As shown, the address event detector 33 according to this configuration example includes a current-to-voltage conversion unit 331, a buffer 332, a subtractor 333, a quantizer 334, and a transmission unit 335.

[0119] The current-to-voltage conversion unit 331 converts the photocurrent from the light receiving unit 31 of the pixel 30 into a logarithmic voltage signal. The current-to-voltage conversion unit 331 provides the converted voltage signal to the buffer 332. The buffer 332 buffers the voltage signal provided from the current-to-voltage conversion unit 331 and provides the voltage signal to the subtractor 333.

[0120] A row drive signal is provided from drive unit 22 to subtractor 333. Subtractor 333 reduces the level of the voltage signal provided from buffer 332 according to the row drive signal. Then, subtractor 333 provides the voltage signal after the level has decreased to quantizer 334. Quantizer 334 quantizes the voltage signal provided from subtractor 333 into a digital signal, and outputs the digital signal as a detection signal for an address event to transmission unit 335.

[0121] The transmission unit 335 transmits the address event detection signal provided by the quantizer 334 to the arbitration unit 23, etc. When an address event is detected, the transmission unit 335 provides a request to the arbitration unit 23 to send the address event detection signal. Then, when a response to the request is received from the arbitration unit 23, the transmission unit 335 provides the address event detection signal to the drive unit 22 and the signal processing unit 25.

[0122] Next, a configuration example of the current-to-voltage conversion unit 331, subtractor 333, and quantizer 334 in the address event detector 33 will be described.

[0123] (Configuration example of a current-to-voltage conversion unit)

[0124] Figure 6 This is a circuit diagram illustrating an example configuration of the current-to-voltage conversion unit 331 in the address event detector 33. (See diagram for example.) Figure 6 As shown, the current-to-voltage conversion unit 331 according to this embodiment has a circuit configuration including an N-type transistor 3311, a P-type transistor 3312, and an N-type transistor 3313. These transistors 3311 to 3313 are, for example, MOS transistors.

[0125] The N-type transistor 3311 is connected to the power supply voltage V. DD The power supply line is connected between the power supply line and the signal input line 3314. The P-type transistor 3312 and the N-type transistor 3313 are connected in series at the power supply voltage V. DD The power line is connected to ground. Then, the common connection node N2 of the P-type transistor 3312 and the N-type transistor 3313 is connected to the gate electrode of the N-type transistor 3311 and... Figure 5 The input terminals of the buffer 332 shown.

[0126] Predetermined bias voltage V bIas The current is applied to the gate electrode of the P-type transistor 3312. Therefore, the P-type transistor 3312 provides a constant current to the N-type transistor 3313. The photocurrent is input from the light receiving unit 31 to the gate electrode of the N-type transistor 3313 via the signal input line 3314.

[0127] The drain electrodes of N-type transistors 3311 and 3313 are connected to a power supply, and such a circuit is called a source follower. The photocurrent from the photoreceiving unit 31 is converted into a logarithmic voltage signal by two source followers connected in a loop.

[0128] (Configuration example of subtractor and quantizer)

[0129] Figure 7 This is a circuit diagram showing an example configuration of the subtractor 333 and quantizer 334 in the address event detector 33.

[0130] The subtractor 333 according to this example includes a capacitor element 3331, an inverter circuit 3332, a capacitor element 3333, and a switching element 3334.

[0131] One end of capacitor element 3331 is connected to Figure 5 The output terminal of the buffer 332 is shown, and its other end is connected to the input terminal of the inverter circuit 3332. A capacitor element 3333 is connected in parallel with respect to the inverter circuit 3332. A switching element 3334 is connected between the two ends of the capacitor element 3333. A row drive signal is provided from the drive unit 22 to the switching element 3334 as an on / off control signal. The switching element 3334 opens and closes the path connecting the two ends of the capacitor element 3333 according to the row drive signal. The inverter circuit 3332 reverses the polarity of the voltage signal input via the capacitor element 3331.

[0132] In the subtractor 333 with the above configuration, when the switching element 3334 is turned on (off), the voltage signal V init The input is fed to the terminal of capacitor element 3331 on the buffer 332 side, and its opposite terminal is a virtual ground terminal. For convenience, the potential of the virtual ground terminal is set to 0. At this time, the charge Q accumulated in capacitor element 3331 is... init This is represented by the following expression (1), where C1 is the capacitance value of capacitor element 3331. On the other hand, since the two ends of capacitor element 3333 are short-circuited, the accumulated charge is 0.

[0133] Q init =C1×V InIt (1)

[0134] Next, consider the voltage change of the terminal of the capacitor element 3331 on the buffer 332 side when the switching element 3334 is turned off (on). after In this case, the charge Q accumulated in capacitor element 3331 after It is represented by the following expression (2).

[0135] Q after =C1×Vafter (2)

[0136] On the other hand, the charge Q2 accumulated in capacitor element 3333 is represented by the following expression (3), where C2 is the capacitance value of capacitor element 3333, and V out It is the output voltage.

[0137] Q2=-C2×V out (3)

[0138] At this time, since the total charge of capacitor element 3331 and capacitor element 3333 remains unchanged, the following expression (4) is established.

[0139] Q init =Q after +Q2 (4)

[0140] When expressions (1) to (3) are substituted into expression (4) and transformed, the following expression (5) is obtained.

[0141] V out =-(C1 / C2)×(V) after -V init (5)

[0142] Expression (5) represents the subtraction operation of the voltage signal, and the gain of the subtraction result is C1 / C2. Since it is generally desirable to maximize the gain, it is preferable to design C1 to be large and C2 to be small. On the other hand, when C2 is too small, kTC noise increases and the noise characteristics may deteriorate. Therefore, the reduction of C2 is limited to a range that can tolerate noise. In addition, since an address event detector 33 including a subtractor 333 is installed for each pixel 30, the capacitor elements 3331 and 3333 have area limitations. Taking these into consideration, the capacitance values ​​C1 and C2 of capacitor elements 3331 and 3333 are determined.

[0143] exist Figure 7 In this circuit, quantizer 334 includes comparator 3341. Comparator 3341 sets the output signal of inverter circuit 3332 (i.e., the voltage signal from subtractor 333) as a non-inverting (+) input and sets a predetermined threshold voltage V. th Set to inverting (-) input. Then, comparator 3341 compares the voltage signal from subtractor 333 with a predetermined threshold voltage V. th The comparison is performed, and the signal representing the comparison result is output to the transmission unit 335 as an address event detection signal.

[0144] [Second Configuration Example for Address Event Detector]

[0145] Figure 8This is a block diagram illustrating a second configuration example of the address event detector 33. (See diagram for example.) Figure 8 As shown, in addition to the current-to-voltage conversion unit 331, buffer 332, subtractor 333, quantizer 334 and transmission unit 335, the address event detector 33 according to this configuration example also includes a storage unit 336 and a controller 337.

[0146] Storage unit 336 is disposed between quantizer 334 and transmission unit 335, and accumulates the output of quantizer 334 (i.e., the comparison result of comparator 3341) based on the sample signal provided from controller 337. Storage unit 336 may be a sampling circuit such as a switch, plastic or capacitor, or may be a digital storage circuit such as a latch or flip-flop.

[0147] Controller 337 will set a predetermined threshold voltage V th The inverting (-) input of comparator 3341 is provided. The threshold voltage V supplied to comparator 3341 from controller 337 is... th Different voltage values ​​can be specified in a time-division manner. For example, controller 337 provides a threshold voltage V at different timings associated with a conduction event indicating that the change in photocurrent exceeds an upper limit threshold. th1 and the threshold voltage V associated with a turn-off event representing a change in value below a lower threshold. th2 This allows a comparator 3341 to detect multiple types of address events.

[0148] For example, memory cell 336 can associate a threshold voltage V with a shutdown event. th2 During the cycle from the controller 337 to the inverting (-) input terminal of the comparator 3341, the threshold voltage V associated with the conduction event is used. th1 The comparison results of comparator 3341 are accumulated. Note that storage unit 336 can be inside or outside pixel 30. Furthermore, storage unit 336 is not a necessary component of address event detector 33; that is, storage unit 336 can be omitted.

[0149] [Imaging apparatus (scanning method) according to the second configuration example]

[0150] The imaging apparatus 20 according to the first configuration example described above is an asynchronous imaging apparatus that reads events using an asynchronous readout method. However, the event readout method is not limited to asynchronous readout methods, but can be a synchronous readout method. An imaging apparatus that applies a synchronous readout method is an imaging apparatus that uses the same scanning method as a normal imaging apparatus that performs imaging at a predetermined frame rate.

[0151] Figure 9This is a block diagram illustrating an example configuration of an imaging device according to a second configuration example (i.e., a scanning imaging device used as an imaging device 20 in an imaging system 10 applying the technology according to this disclosure).

[0152] like Figure 9 As shown, the imaging apparatus 20, which is a second configuration example of the imaging apparatus of this disclosure, includes a pixel array unit 21, a driving unit 22, a signal processing unit 25, a readout area selection unit 27, and a signal generation unit 28.

[0153] Pixel array unit 21 includes a plurality of pixels 30. Each of the plurality of pixels 30 outputs an output signal in response to a selection signal from readout region selection unit 27. For example, each of the plurality of pixels 30 may include, for example, a... Figure 7 The quantizer in the pixel shown. Each of the multiple pixels 30 outputs an output signal associated with the amount of change in light intensity. (Example...) Figure 9 As shown, multiple pixels 30 can be arranged in a two-dimensional matrix.

[0154] The driving unit 22 drives each of the plurality of pixels 30 to output the pixel signal generated in each pixel 30 to the signal processing unit 25. Note that the driving unit 22 and the signal processing unit 25 are circuit units used to acquire grayscale information. Therefore, if only event information is acquired, the driving unit 22 and the signal processing unit 25 may not be required.

[0155] The reading region selection unit 27 selects some of the plurality of pixels 30 included in the pixel array unit 21. For example, the reading region selection unit 27 selects any one or more rows included in the rows of the structure corresponding to the two-dimensional matrix of the pixel array unit 21. The reading region selection unit 27 selects one or more rows sequentially according to a preset period. Furthermore, the reading region selection unit 27 can determine the selected region in response to a request from each pixel 30 of the pixel array unit 21.

[0156] Based on the output signal of the pixel selected by the reading region selection unit 27, the signal generation unit 28 generates an event signal associated with an active pixel in the selected pixel where an event has been detected. The event is a change in light intensity. An active pixel is a pixel whose change in light intensity associated with the output signal exceeds or falls below a preset threshold. For example, the signal generation unit 28 compares the pixel's output signal with a reference signal, detects an active pixel that outputs the signal if the output signal is greater than or less than the reference signal, and generates an event signal associated with the active pixel.

[0157] The signal generation unit 28 may, for example, include a column selection circuit that arbitrates signals entering the signal generation unit 28. Furthermore, the signal generation unit 28 may be configured to output not only information about active pixels where events have been detected, but also information about passive pixels where events have not been detected.

[0158] The signal generation unit 28 outputs the address information and timestamp information (e.g., (X, Y, T)) of the active pixel whose event has been detected via output line 15. However, the data output from the signal generation unit 28 can be not only address information and timestamp information, but also information in frame format (e.g., (0, 0, 1, 0, ...)).

[0159] [Chip Architecture Configuration Example]

[0160] The chip (semiconductor integrated circuit) structure of the imaging device 20 according to the first configuration example or the second configuration example described above may, for example, have a stacked chip structure. Figure 10 This is an exploded perspective view schematically showing the stacked chip structure of the imaging device 20.

[0161] like Figure 10 As shown, the stacked chip structure, that is, the stacked structure has at least two chips, namely, an optical receiving chip 201 as a first chip and a detection chip 202 as a second chip, stacked together. Then, in Figure 4 In the circuit configuration of the pixel 30 shown, each light receiving element 311 is arranged on the light receiving chip 201, and all components other than the light receiving elements 311, as well as components of other circuit parts of the pixel 30, are arranged on the detection chip 202. The light receiving chip 201 and the detection chip 202 are electrically connected via connection portions such as vias (VIA), Cu-Cu bonds, or bumps.

[0162] Note that here, an example of a configuration has been shown in which the light receiving element 311 is arranged on the light receiving chip 201 and other components, such as the components of other circuit parts of the pixel 30, are arranged on the detection chip 202. However, the present invention is not limited to this configuration example.

[0163] For example, in Figure 4In the circuit configuration of pixel 30 shown, the corresponding components of the light receiving unit 31 can be arranged on the light receiving chip 201, and components other than the light receiving unit 31, components of other circuit parts of pixel 30, etc., can be arranged on the detection chip 202. Furthermore, the corresponding components of the light receiving unit 31, as well as the reset transistor 321 and floating diffusion layer 324 of the pixel signal generation unit 32, can be arranged on the light receiving chip 201, and other components can be arranged on the detection chip 202. Additionally, some components that together with the corresponding components of the light receiving unit 31, etc., constitute the address event detector 33 can be arranged on the light receiving chip 201.

[0164] [Configuration example of column processing unit]

[0165] Figure 11 This is a block diagram illustrating an example configuration of the column processing unit 24 of the imaging apparatus 20 according to a first configuration example. Figure 11 As shown, the column processing unit 24 according to this example includes a plurality of analog-to-digital converters (ADCs) 241 arranged for each pixel column of the pixel array unit 21.

[0166] Note that this example illustrates a configuration where the analog-to-digital converter 241 is arranged in a one-to-one correspondence with the pixel columns of the pixel array unit 21; however, the invention is not limited to this configuration example. For instance, the analog-to-digital converter 241 can be arranged in units of multiple pixel columns, and the analog-to-digital converter 241 can be used in a time-division manner among multiple pixel columns.

[0167] The analog-to-digital converter 241 converts the analog pixel signal SIG provided via the vertical signal line VSL into a digital signal with a larger number of bits than the detection signal of the address event described above. For example, when the detection signal of the address event is 2 bits, the pixel signal is converted into a digital signal of 3 bits or more (16 bits, etc.). The analog-to-digital converter 241 provides the digital signal generated by the analog-to-digital conversion to the signal processing unit 25.

[0168] (First Embodiment)

[0169] Figure 12 This is a block diagram showing the internal configuration of the signal processing unit 25 according to the first embodiment. Figure 12 The signal processing unit 25 includes an event counter 51 and a threshold adjustment unit 52. The event counter 51 counts the number of events. As will be described later, an event is detected when the brightness changes rapidly. It is conceivable that events would be counted separately when the brightness increases rapidly, and events would be counted separately when the brightness decreases rapidly. However, in this embodiment, events are counted without distinguishing the direction of the brightness change.

[0170] The threshold adjustment unit 52 adjusts the threshold based on the event detection status. The threshold is determined when... Figure 4 The threshold for when the address event detector 33 detects the presence or absence of an address event. The event detection status includes one of the following: the number of event signals detected in a predetermined area within a predetermined period, the ratio of the number of address event detectors 33 that output event signals to the total number of address event detectors 33 in the predetermined area, and the signal level of the event signal.

[0171] Figure 13 This is a flowchart illustrating the processing operations of the pixel array unit 21 and the signal processing unit 25 according to the first embodiment. First, the period and range of the detected events are initially set (step S1). Then, event detection of the pixel array unit 21 begins within the predetermined range set initially (step S2).

[0172] Next, it is determined whether an event has occurred (step S3). Here, if an event detection signal is output from pixel array unit 21, it is determined that an event has occurred. If an event has occurred, the count value (number of events) of event counter 51 is counted (step S4).

[0173] Next, it is determined whether the predetermined period set in step S1 has elapsed (step S5). If the predetermined period has not elapsed, the processing after step S2 is repeated, and if it is determined that the predetermined period has elapsed, it is determined whether the number of events counted by the event counter 51 is greater than or equal to a first reference value (step S6). When it is greater than or equal to the first reference value, the event detection threshold in the address event detector 33 is increased to reduce the event detection sensitivity of the address event detector 33 (step S7). The frequency of events detected by the address event detector 33 decreases as the threshold increases.

[0174] When the number of events is determined to be less than the first reference value in step S6, it is then determined whether the number of events is less than the second reference value (step S8). The second reference value is the same as or less than the first reference value. When the number of events is less than the second reference value, the event detection threshold is lowered to improve the event detection sensitivity of the address event detector 33 (step S9). As the threshold decreases, the frequency of events detected by the address event detector 33 increases. The updated threshold is then communicated to the address event detector 33 in the pixel array unit 21.

[0175] exist Figure 13 In the flowchart, when the number of events is less than the first reference value but equal to or greater than the second reference value, the threshold of the address event detector 33 remains unchanged. Note that, as mentioned above, the first reference value can be made equal to the second reference value, thereby reducing... Figure 13The flowchart determines the number of processes and allows for rapid adjustment of the threshold.

[0176] exist Figure 13 In step S1, the initial setting is to adjust the threshold range for the execution event, but as... Figure 14 As shown, it can be performed individually for each of the multiple pixel regions in the imaging scene captured from the viewpoint of the imaging device 20. Figure 13 The processing can be performed, and the threshold of the address event detector 33 can be set independently. For example, if the detection status of an event is significantly different between pixel regions r1 and r2, the threshold of pixel region r1 and the threshold of pixel region r2 can also be set as the threshold of the address event detection unit 33.

[0177] exist Figure 13 In this process, the threshold of the address event detector 33 is adjusted by the number of events, but the threshold can also be adjusted by the event excitation rate or the signal level of the event signal. The event excitation rate is the ratio of the number of pixels that detect events to the number of pixels within the predetermined range initially set in step S1.

[0178] exist Figure 13 In step S4, the method of counting the number of events by event counter 51 varies depending on whether the imaging device is in an asynchronous system or a synchronous system (scanning system).

[0179] Figure 15 It is shown Figure 9 The flowchart illustrates the event counting process in the imaging apparatus of the scanning method described above. First, the imaging process waits until one frame of the pixel array unit is completed (step S401), and the output signal of each pixel is read sequentially. When an event signal is detected (step S402), the event signal is acquired (step S403) and stored (step S404). Next, the event counter 51 counts all stored event signals (step S405).

[0180] Figure 16 It is shown Figure 9 A flowchart illustrating the modification of the event counting processing procedure in the imaging device of the scanning method. Figure 16 Flowcharts and Figure 15 The difference in processing is that, after waiting until the imaging process is completed (step S411), whenever an event signal is detected and acquired (steps S412 and S413), the number of events is counted by the event counter 51 (step S414).

[0181] Figure 17 It is shown Figure 2 A flowchart of the event counting process in the asynchronous imaging device. Figure 17An example of asynchronous event detection is shown, performed on a row or pixel-by-pixel basis in the pixel array. First, the process waits until an event occurs (step S421). When an event is detected (step S422), the event is acquired (step S423), and the event counter 51 counts the events (step S424).

[0182] Figure 18 This is a diagram illustrating an example of a first captured image in which the threshold of the address event detector 33 is adjusted while the entire imaging scene captured by the pixel array unit 21 is initially set to a predetermined range. Figure 18 The example shows a captured image of a human hand captured by imaging device 20 in the dark. Figure 18 Captured image IM1 shows an example of excessively low event detection sensitivity of the address event detector 33, while captured image IM2 shows an example of excessively high event detection sensitivity. Because captured image IM1 has excessively low event detection sensitivity, almost no event is detected, and captured image IM1 is almost completely dark. Because captured image IM2 has excessively high event detection sensitivity, events including noise are detected beyond what is necessary, and the outline of the hand is unclear. Captured image IM3 is during the execution... Figure 13 or Figure 14 The captured image after processing. Even in the dark, the outline of the hand is clear, and it can be seen that the appropriate event detection sensitivity has been set.

[0183] Figure 19 This is an illustration of a second captured image example showing the adjustment of a first reference value and a second reference value for a portion of a pixel area in an imaging scene. Capture image IM4 is an example of a captured image under conditions of excessively high event detection sensitivity. For example, in a portion of pixel area IR1 in captured image IM4, there is a flickering light source, and a large number of events occur. A human hand appears near pixel area IR1. The tip of the hand overlaps with the events caused by the flickering, and the tip of the hand is particularly blurry.

[0184] Image capture IM5 is performed in pixel region IR1. Figure 13 or Figure 14 The captured image is processed as follows. By adjusting the first and second reference values, the event detection sensitivity in pixel region IR1 is made so low that events in pixel region IR1 can hardly be detected. Therefore, even in dark conditions, a captured image in which the outline of a human hand is clearly visible is obtained.

[0185] As described above, in the first embodiment, since the threshold of the address event detector 33 is adjusted based on at least one of the number of events detected by the pixel array unit 21, the excitation rate, and the detection signal level, an optimal threshold setting can be performed according to the event detection conditions in the imaging scene. For example, in the case of a pixel region where a large number of events are detected in the imaging scene, by increasing the threshold in that pixel region to reduce the event detection sensitivity, important events can be reliably detected while eliminating events caused by noise such as flicker.

[0186] (Second Embodiment)

[0187] When the address event detector 33 detects an event, it can detect the Pos event and the Neg event separately. A Pos event occurs when the absolute value of the change in brightness exceeds a first threshold as brightness changes in a direction increasing. A Neg event occurs when the change in brightness exceeds a second threshold as brightness changes in a direction decreasing. For example, in the case of an event caused by flicker, since a large number of Pos and Neg events are detected alternately, it is possible to determine whether the event is caused by flicker or by other factors based on the detection order and number of Pos and Neg events.

[0188] The signal processing unit 25 according to the second embodiment described below has the same... Figure 12 The configuration is similar to the block configuration in this embodiment, but the processing operations of the event counter 51 and the threshold adjustment unit 52 are different from those in the first embodiment. The event counter 51 counts the number of Pos events and the number of Neg events, respectively. The threshold adjustment unit 52 adjusts the threshold for Pos event detection based on the detection status of Pos events, and adjusts the threshold for Neg event detection based on the detection status of Neg events.

[0189] As described above, the address event detector 33 according to this embodiment includes a first detector and a second detector. The first detector detects a first detection signal when the absolute value of the change in the electrical signal after photoelectric conversion along the increasing direction exceeds a first threshold. The second detector detects a second detection signal when the absolute value of the change in the electrical signal after photoelectric conversion along the decreasing direction exceeds a second threshold. The threshold adjustment unit 52 adjusts the first threshold and the second threshold based on the detection status of the first detector and the second detector.

[0190] Furthermore, the signal processing unit 25 according to this embodiment includes a first determining unit and a second determining unit. The first determining unit determines whether the detection status in the first detector is within a first allowable range. The second determining unit determines whether the detection status in the second detector is within a second allowable range. If the first determining unit determines that the detection status is not within the first allowable range, the threshold adjustment unit adjusts the first threshold, and if the second determining unit determines that the detection status is not within the second allowable range, the threshold adjustment unit adjusts the second threshold.

[0191] Figure 20 This is a flowchart illustrating the processing operation of the signal processing unit 25 according to the second embodiment. First, the period and range of the detected events are initially set (step S11). Then, the address event detector 33 begins event detection in the pixel array unit 21 within the predetermined range set initially (step S12). The address event detector 33 detects and outputs Pos events and Neg events separately.

[0192] Event counter 51 counts Pos and Neg events separately (step S13). When a Pos or Neg event is detected, depending on whether the imaging device 20 is synchronous (scanning) or asynchronous, the following needs to be changed: Figures 15 to 17 The process shown is for counting the number of events.

[0193] Event detection continues until the predetermined period initially set in step S11 has elapsed. When the predetermined period has elapsed, the threshold adjustment unit 52 detects the excitation rate of the Pos event (step S14) and the excitation rate of the Neg event (step S15). The excitation rate is the ratio of the number of pixels that detect the Pos event (Neg event) to the number of pixels within the predetermined range initially set in step S11.

[0194] Next, determine whether the firing rate of the Pos event is within the allowable range (step S16), and determine whether the firing rate of the Neg event is within the allowable range (step S17). Figure 21 As shown, the permissible range is the range between the upper and lower permissible limits of the excitation rate near the expected value of the excitation rate. The permissible range of the excitation rate for the Pos event and the permissible range of the excitation rate for the Neg event can be the same or different.

[0195] When it is determined in step S16 that the firing rate of the Pos event is not within the allowable range, the threshold of the Pos event is adjusted so that the firing rate of the Pos event falls within the allowable range (step S18). This threshold is the threshold of the Pos event in the address event detector 33. Similarly, the threshold of the Neg event is adjusted so that the firing rate of the Neg event falls within the allowable range (step S19).

[0196] exist Figure 20 In the flowchart, it is determined whether the excitation rate of the Pos event (Neg event) is within the allowable range. However, instead of the excitation rate, it can be determined whether the number of Pos events (Neg events) or the detection signal level is within the allowable range. Alternatively, it can be determined whether the event detection result in the CPU or the like arranged after the signal processing unit 25 is within the allowable range.

[0197] As described above, in the second embodiment, the excitation rate of Pos and Neg events is detected separately, and the event detection threshold is adjusted when the excitation rate is outside the allowable range. Therefore, the thresholds for Pos and Neg events can be optimized. This embodiment is effective when events changing along the direction of increasing brightness and events changing along the direction of decreasing brightness exist individually. By processing Pos and Neg events separately and performing threshold adjustment, the optimal number of both Pos and Neg events can be detected.

[0198] (Third Embodiment)

[0199] In the third embodiment, the threshold is adjusted based on the detection frequency of the event. Figure 22 This is a block diagram illustrating the internal configuration of the signal processing unit 25 according to the third embodiment. Besides... Figure 12 In addition to the configuration of the signal processing unit 25, Figure 22 The signal processing unit 25 includes a frequency determination unit 53 and an event output processing unit 54. The frequency determination unit 53 calculates the event detection frequency based on the number of events counted by the event counter 51, and compares the calculated detection frequency with a first reference value and a second reference value, such as... Figure 13 and Figure 14 As shown.

[0200] The event output processing unit 54 performs noise removal processing and signal level adjustment processing on the event signal output from the pixel array unit 21 to output the event signal. The output event signal is, for example, input to... Figure 1 The central processing unit (CPU) shown is an example.

[0201] Figure 23 This is a flowchart of the processing operation of the signal processing unit 25 according to the third embodiment. The processing in steps S21 to S25 is... Figure 13 The processing in steps S1 to S5 is similar. When it is determined that a predetermined period has elapsed, the detection frequency of events within the predetermined range initially set in step S2 is calculated (step S26). The detection frequency can be calculated based on the value obtained by dividing the number of events counted by event counter 51 by the predetermined range or predetermined period.

[0202] Determine whether the event detection frequency calculated in step S26 is greater than or equal to the first reference value (step S27). If it is greater than or equal to the first reference value, increase the event detection threshold in the address event detector 33 to reduce the event detection sensitivity of the address event detector 33 (step S28).

[0203] When it is determined in step S27 that the detected value is less than the first reference value, it is determined whether the event detection frequency is less than the second reference value (step S29). The second reference value is the same as or less than the first reference value. When the event detection frequency is less than the second reference value, the threshold is lowered to improve the event detection sensitivity of the address event detector 33 (step S30). The updated threshold is notified to the address event detector 33 in the pixel array unit 21.

[0204] As described above, in the third embodiment, since the threshold for event detection in the address event detector 33 is adjusted based on the event detection frequency, the threshold can be set to make the event detection frequency optimal.

[0205] (Fourth Embodiment)

[0206] In the fourth embodiment, object identification is performed by a CPU connected to the signal processing unit 25.

[0207] Figure 24 This is a block diagram illustrating the connection relationship between the signal processing unit 25 and the CPU 55 according to the fourth embodiment. Figure 1 As shown, the CPU 55 can be disposed separately from the imaging device 20, or it can be disposed inside the imaging device 20. The CPU 55 is a concept that includes application processors (APs), digital signal processors (DSPs), image signal processors (ISPs), etc., installed in smartphones, etc.

[0208] The CPU 55 serves as a region extraction unit, configured to extract a portion of the imaging view of the pixel array unit 21 based on the detection status of event signals from the address event detector 33. This portion of the image is a portion of the pixel area in the imaging scene and is referred to as the region of interest (ROI). The ROI is, for example, the area where an event is detected. Furthermore, the CPU 55 also serves as an object recognition unit, configured to identify objects included in the imaging scene based on the detection status of event signals. The aforementioned ROI is set to include the region of the identified object. Additionally, the CPU 55 also serves as a notification unit, configured to notify the signal processing unit 25 or the pixel array unit 21 of information regarding the coordinate position of the ROI.

[0209] The signal processing unit 25 sends the event signal output from the pixel array unit 21 to the CPU 55. The CPU 55 identifies objects present in the imaging scene imaged by the pixel array unit 21 based on the event signal. Then, the pixel region including the object is set as the Region of Interest (ROI). The CPU 55 sends information about the coordinate position of the ROI in the imaging scene to the signal processing unit 25. The signal processing unit 25 notifies the pixel array unit 21 of the coordinate position information of the ROI. The pixel array unit 21 performs photoelectric conversion only on pixels within the ROI, and only the corresponding address event detector 33 outputs the event signal. As described above, because the pixel array unit 21 can limit the pixel region for event detection, the power consumption of the pixel array unit 21 can be reduced.

[0210] Furthermore, the CPU 55 can send the number of desired events detected by the address event detector 33, along with information about the coordinate position of the ROI, to the pixel array unit 21. In this case, the address event detector 33 not only sets the range of events to be detected based on the information about the coordinate position of the ROI sent from the CPU 55, but also adjusts the threshold of the address event detector 33 so that it can output an event signal of the number of events sent from the CPU 55.

[0211] Figure 25 It is shown Figure 24 The flowchart illustrates the processing operations of the signal processing unit 25 and the CPU 55. The signal processing unit 25 sends the event signal received from the pixel array unit 21 to the CPU 55 (step S31). The CPU 55 identifies objects in the imaging scene based on the event signal (step S32). Since the outline shape of an object can be determined through the event signal, the object can be identified using known pattern matching processes. When multiple objects exist in the imaging scene, the CPU 55 can identify multiple objects.

[0212] Next, CPU 55 sets the pixel region containing the objects in the imaging scene as the ROI (step S33). The ROI is, for example, a rectangular pixel region. In the case of recognizing multiple objects in the imaging scene, multiple ROIs can be set.

[0213] Next, the CPU 55 sends information about the coordinates of the ROI in the imaging scene to the imaging device 20 (more specifically, the signal processing unit 25) (step S34). The signal processing unit 25 notifies the pixel array unit 21 of the coordinates of the ROI. Note that the coordinates of the ROI can be sent directly from the CPU 55 to the pixel array unit 21 without going through the signal processing unit 25.

[0214] The address event detector 33 in the pixel array unit 21 detects events within the range of the ROI based on information about the coordinate position of the ROI (step S35). Therefore, the range in which the pixel array unit 21 performs event detection can be limited, the event detection processing of the pixel array unit 21 can be accelerated, and the power consumption of the pixel array unit 21 can be reduced.

[0215] Furthermore, when information about the number of events is sent from the CPU 55, the pixel array unit 21 adjusts the threshold of the address event detector 33 so that the number of detected events is as indicated by the CPU 55.

[0216] Figure 26 This is a diagram showing examples of objects included in an imaging scene. Figure 26 An image generated based on events detected in the dark is shown. Figure 26 The captured image as a whole is dark, but the outline of the object is clear due to the difference in brightness. The CPU 55 can identify the object based on the shape of the object's outline in the captured image. Then, the CPU 55 sets a rectangular pixel region based on the object's outline and sets that pixel region as the ROI.

[0217] As described above, in the fourth embodiment, the CPU 55 performs object recognition and sets the Region of Interest (ROI) based on the event signals output from the pixel array unit 21. Thereafter, since the pixel array unit 21 detects events within the ROI set by the CPU 55, events can be detected at high speed and low power consumption. Furthermore, the CPU 55 can specify the number of events to be output from the pixel array unit 21, and the CPU 55 can control the threshold of the address event detector 33.

[0218] (Fifth Embodiment)

[0219] In the fifth embodiment, the CPU 55 commands the pixel array unit 21 to perform a predetermined operation based on the occurrence of the event.

[0220] Figure 27 This is a block diagram illustrating the connection relationship between the signal processing unit 25 and the CPU 55 according to the fifth embodiment. Figure 24 As shown, the CPU 55 can be disposed separately from the imaging device 20, or it can be disposed inside the imaging device 20. For example, in Figure 24 In this context, CPU 55 encompasses concepts such as AP, DSP, and ISP. CPU 55 transmits information about the operating conditions of the pixel array unit 21 in the imaging device 20 based on event signals sent from the signal processing unit 25.

[0221] Figure 27The CPU 55 in the CPU is used as an information processing unit, which is configured to generate operating condition information based on the event detection status, including at least one of the frame rate (photoelectric conversion speed) of the pixel array unit 21, the sparsity of the pixels performing photoelectric conversion, and the activation frequency of the pixel array unit 21 performing photoelectric conversion.

[0222] Figure 28 It is shown Figure 27 The flowchart illustrates the processing operations of the signal processing unit 25 and the CPU 55. The signal processing unit 25 sends a signal indicating the occurrence of an event from the pixel array unit 21 to the CPU 55 (step S41). The signal indicating the occurrence of an event can be any one of the following: the number of events, the event detection frequency, the aforementioned excitation rate, and the event signal level.

[0223] The CPU 55 determines the operating conditions of the pixel array unit 21 based on signals indicating the occurrence of events (step S42). For example, when the frequency of event occurrence decreases, the period (frame rate) for performing event detection in the pixel array unit 21 can be extended, or the pixels performing event detection in the pixel array unit 21 can be made sparser. Alternatively, as the frequency of event occurrence decreases, the frequency of activating the pixel array unit 21 can be further reduced. Conversely, as the frequency of event occurrence increases, the frame rate of the pixel array unit 21 can be increased, the number of pixels used to perform event detection can be increased, or the activation speed of the pixel array unit 21 can be increased.

[0224] Next, the CPU 55 sends the determined operating conditions of the pixel array unit 21 to the imaging device 20 (e.g., the signal processing unit 25 or the pixel array unit 21) (step S43). The pixel array unit 21 changes its settings based on the operating conditions determined by the CPU 55 (step S44). Specifically, the frame rate of the pixel array unit 21 is changed, pixel sparsity processing for event detection is performed, and the activation speed of the pixel array unit 21 is changed.

[0225] As described above, in the fifth embodiment, since the CPU 55 sets the operating conditions of the pixel array unit 21 according to the event occurrence, for example, when the event occurrence frequency is low, the power consumption of the pixel array unit 21 can be reduced by decreasing the frame rate of the pixel array unit 21 or sparsifying the pixels that detect events. Conversely, when the event occurrence frequency is high, for example, by increasing the frame rate of the pixel array unit 21 or increasing the number of pixels that can detect events, events can be detected quickly and accurately. Furthermore, according to this embodiment, since the CPU 55, which is separate from the signal processing unit 25, can change the setting of the operating conditions of the pixel array unit 21, the operation of the pixel array unit 21 can be finely controlled according to the situation.

[0226] (Sixth Embodiment)

[0227] In the sixth embodiment, the ROI is set by the signal processing unit 25 instead of the CPU 55.

[0228] Figure 29 This is a block diagram showing the connection relationship between the signal processing unit 25 and the CPU 55 according to the sixth embodiment. Figure 29 The signal processing unit 25 extracts the pixel region in the pixel array unit 21 from the event signal output from the pixel array unit 21, and sets the pixel region as the Region of Interest (ROI). The signal processing unit 25 sends the event information detected within the ROI to the CPU 55. The event information sent to the CPU 55 includes information such as the pixel position where the event occurred and the number of events. Therefore, the amount of event information sent to the CPU 55 can be reduced.

[0229] Figure 30 It is shown Figure 29 The flowchart illustrates the processing operations of the signal processing unit 25 and the CPU 55. First, the signal processing unit 25 acquires the event signal output from the pixel array unit 21 (step S51). Next, the signal processing unit 25 extracts the pixel region where the event occurred based on the event signal and sets the extracted pixel region as the Region of Interest (ROI) (step S52). Next, the signal processing unit 25 sends the event information from the ROI to the CPU 55 (step S53). The CPU 55 performs various types of signal processing based on the event information sent from the signal processing unit 25.

[0230] As described above, in the sixth embodiment, since the signal processing unit 25 sets the ROI based on the event signal output from the pixel array unit 21 and sends the event information in the ROI to the CPU 55, the amount of data sent from the signal processing unit 25 to the CPU 55 can be reduced, and the power consumption in the entire imaging system 10 can be reduced.

[0231] (Seventh Embodiment)

[0232] In the seventh embodiment, the ROI is set based on the event signal output from the pixel array unit 21, and the set ROI is sent to another imaging device (imaging unit) 20.

[0233] Figure 31 This is a block diagram illustrating a schematic configuration of the imaging system 10 according to the seventh embodiment. Besides... Figure 1 In addition to the configuration of the imaging system 10, Figure 31The imaging system 10 includes a new imaging device 20a. For example, the newly added imaging device 20a may be a conventional CMOS image sensor (CIS) that outputs a brightness signal. In the following description, the newly added imaging device 20a outputs the brightness signals of all pixels within a specified pixel area.

[0234] Figure 32 This is a block diagram showing the connection relationship between the signal processing unit 25 and the CPU 55 according to the seventh embodiment. Figure 32 The signal processing unit 25 sets a pixel region including the location of the event based on the event signal output from the pixel array unit 21, and sets this pixel region as the ROI. The signal processing unit 25 sends the event information in the ROI and the coordinate position information of the ROI to the CPU 55.

[0235] The CPU 55 sends information about the coordinates of the received ROI to the imaging device 20a. The imaging device 20a then sends image data, including brightness signals from the ROI, to the CPU 55 based on the coordinates of the received ROI. For example, the CPU 55 performs image processing based on event information from the ROI sent from the signal processing unit 25 and the image data from the ROI sent from the imaging device 20a.

[0236] For example, the CPU 55 can perform a correction process on the brightness information of underexposed pixel areas in the image data output from the imaging device 20a based on the event information sent from the signal processing unit 25. As described above, the CPU 55 serves as a correction processing unit for correcting image data from the imaging device 20a based on event information.

[0237] Figure 33 It is shown Figure 32 The flowchart shows the processing operations of the signal processing unit 25 and the CPU 55. Figure 33 Steps S61 to S63 in the middle Figure 30 Steps S41 to S43 are similar. The CPU 55 sends the coordinate position information of the ROI received from the signal processing unit 25 to the imaging device 20a (step S64). Based on the left table position information of the ROI received from the CPU 55, the imaging device 20a images the pixel region of the ROI and sends the image data to the CPU 55 (step S65). For example, the CPU 55 can perform correction processing on the image data sent from the imaging device 20a based on event information sent from the signal processing unit 25 and generate new image data. Note that the processing content of the CPU 55 is not limited.

[0238] As described above, in the seventh embodiment, a Region of Interest (ROI) is set based on an event signal, and imaging is performed within the set ROI by another imaging device 20a. Therefore, for example, image data of the region where the event occurred can be acquired, and this event information can be used for image data correction processing. Furthermore, according to this embodiment, since the newly set imaging device 20a does not generate image data for pixel regions where no event occurred, the overall imaging system 10 can reduce the amount of image data communication and lower power consumption.

[0239] <Examples of the application of the technology according to this disclosure>

[0240] The technology disclosed herein can be applied to a variety of products. More specific application examples will be described below. For example, the technology disclosed herein can be implemented as a distance measuring device mounted on any type of mobile body, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, aircraft, drones, ships, robots, construction machinery, and agricultural machinery (tractors)).

[0241] [Moving Object]

[0242] Figure 34 This is a block diagram illustrating a schematic configuration example of a vehicle control system 7000, which is an example of a mobile body control system to which the technology according to this disclosure can be applied. The vehicle control system 7000 includes multiple electronic control units connected via a communication network 7010. Figure 34 In the example shown, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting the multiple control units can be, for example, an in-vehicle communication network conforming to any standard such as Controller Area Network (CAN), Local Area Network (LIN), Local Area Network (LAN), or FlexRay (registered trademark).

[0243] Each control unit includes a microcomputer that performs calculations according to various programs, a storage unit that stores the programs executed by the microcomputer, parameters used for various calculations, etc., and drive circuits that drive various devices to be controlled. Each control unit includes a network I / F for communicating with other control units via the communication network 7010, and a communication I / F for communicating with devices, sensors, etc., inside and outside the vehicle via wired or wireless communication. Figure 34In the diagram, the integrated control unit 7600 is configured with the following functional components: a microcomputer 7610, a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, a sound / image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Other control units similarly include microcomputers, communication I / Fs, and storage units.

[0244] The drive system control unit 7100 controls the operation of devices related to the vehicle's drive system according to various programs. For example, the drive system control unit 7100 functions as a drive force generating device (such as an internal combustion engine or drive motor) that generates the vehicle's driving force, a drive force transmission mechanism that transmits the driving force to the wheels, a steering mechanism for adjusting the vehicle's steering angle, and a control device such as a braking device that generates the vehicle's braking force. The drive system control unit 7100 may also function as a control device for systems such as anti-lock braking systems (ABS) or electronic stability control (ESC).

[0245] The vehicle condition detector 7110 is connected to the drive system control unit 7100. The vehicle condition detector 7110 includes, for example, at least one of the following sensors: a gyroscope sensor for detecting the angular velocity of the vehicle's axial rotational motion, an acceleration sensor for detecting the vehicle's acceleration, or a sensor for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, engine speed, wheel speed, etc. The drive system control unit 7100 processes the signals input from the vehicle condition detector 7110 and controls the internal combustion engine, drive motor, electric power steering, braking system, etc.

[0246] The vehicle body system control unit 7200 controls the operation of various devices installed on the vehicle body according to various programs. For example, the vehicle body system control unit 7200 is used as a control device for keyless entry systems, smart key systems, power windows, or various lights such as headlights, reversing lights, brake lights, hazard lights, or fog lights. In this case, radio waves or signals from various switches transmitted from a portable device that replaces the key can be input to the vehicle body system control unit 7200. The vehicle body system control unit 7200 receives these radio wave or signal inputs and controls the vehicle's door locking devices, power windows, lights, etc.

[0247] The battery control unit 7300 controls the secondary battery 7310, which serves as a power source for the drive motor, according to various programs. For example, information such as battery temperature, battery output voltage, or remaining battery capacity is input from the battery device including the secondary battery 7310 to the battery control unit 7300. The battery control unit 7300 uses these signals to perform arithmetic processing and performs temperature regulation control of the secondary battery 7310 or control of cooling devices included in the battery device.

[0248] The exterior information detection unit 7400 detects information about the exterior of the vehicle on which the vehicle control system 7000 is installed. For example, at least one of the imaging unit 7410 and the exterior information detector 7420 is connected to the exterior information detection unit 7400. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The exterior information detector 7420 includes, for example, at least one of an environmental sensor that detects current atmospheric or weather conditions and a surrounding information detection sensor that detects other vehicles, obstacles, pedestrians, etc., around the vehicle on which the vehicle control system 7000 is installed.

[0249] The environmental sensor can be at least one of, for example, a rain sensor for detecting rain, a fog sensor for detecting fog, a sunlight sensor for detecting sunlight intensity, and a snow sensor for detecting snowfall. The ambient information detection sensor can be at least one of an ultrasonic sensor, a radar device, and a light detection and ranging, laser imaging detection and ranging (LIDAR) device. The imaging unit 7410 and the exterior information detector 7420 can be configured as independent sensors or devices, or they can be configured as a device integrating multiple sensors or devices.

[0250] here, Figure 35 An example of the mounting positions of imaging unit 7410 and exterior information detector 7420 is shown. Imaging units 7910, 7912, 7914, 7916, and 7918 are, for example, disposed at at least one of the following locations: the front nose, side mirrors, rear bumper, rear door, and the upper part of the windshield inside the vehicle 7900. Imaging unit 7910 disposed at the front nose and imaging unit 7918 disposed at the upper part of the windshield inside the vehicle primarily acquire images of the front of the vehicle 7900. Imaging units 7912 and 7914 disposed at the side mirrors primarily acquire images of the sides of the vehicle 7900. Imaging unit 7916 disposed at the rear bumper or rear door primarily acquires images of the rear of the vehicle 7900. Imaging unit 718 disposed at the upper part of the windshield inside the vehicle is primarily used to detect vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0251] Notice, Figure 35Examples of the imaging ranges of the corresponding imaging units 7910, 7912, 7914, and 7916 are shown. Imaging range a represents the imaging range of imaging unit 7910 located in the front nose; imaging ranges b and c represent the imaging ranges of imaging units 7912 and 7914 located in the side mirrors, respectively; and imaging range d represents the imaging range of imaging unit 7916 located in the rear bumper or rear door. For example, by superimposing the image data captured by imaging units 7910, 7912, 7914, and 7916, a top-down image of the vehicle 7900 viewed from above can be obtained.

[0252] The exterior information detectors 7920, 7922, 7924, 7926, 7928, and 7930, located at the front, rear, sides, corners, and above the windshield inside the vehicle 7900, can be, for example, ultrasonic sensors or radar devices. The exterior information detectors 7920, 7926, and 7930, located at the front nose, rear bumper, rear door, and above the windshield inside the vehicle 7900, can be, for example, LIDAR devices. These exterior information detectors 7920 to 7930 are primarily used to detect vehicles, pedestrians, obstacles, etc., ahead.

[0253] return Figure 34 The description will continue. The exterior information detection unit 7400 enables the imaging unit 7410 to capture images of the exterior of the vehicle and receives the captured image data. Additionally, the exterior information detection unit 7400 receives detection information from the connected exterior information detector 7420. If the exterior information detector 7420 is an ultrasonic sensor, radar device, or LIDAR device, the exterior information detection unit 7400 transmits ultrasonic waves, electromagnetic waves, etc., and receives information about the received reflected waves. The exterior information detection unit 7400 can perform processing based on the received information to detect objects such as people, vehicles, obstacles, signs, or characters on the road surface, or to detect their distance. The exterior information detection unit 7400 can perform environmental recognition processing based on the received information, such as identifying rain, fog, and road conditions. The exterior information detection unit 7400 can calculate the distance to objects outside the vehicle based on the received information.

[0254] Furthermore, the exterior information detection unit 7400 can perform image recognition processing or distance detection processing based on the received image data to identify people, vehicles, obstacles, signs, characters on the road surface, etc. The exterior information detection unit 7400 can perform processing such as distortion correction or alignment on the received image data and synthesize image data captured by different imaging units 7410 to generate a top-down or panoramic image. The exterior information detection unit 7400 can use image data captured by different imaging units 7410 to perform viewpoint switching processing.

[0255] The vehicle information detection unit 7500 detects information inside the vehicle. For example, a driver state detector 7510, which detects the driver's state, is connected to the vehicle information detection unit 7500. The driver state detector 7510 may include a camera that images the driver, a biosensor that detects biological information about the driver, a microphone that collects sounds inside the vehicle, etc. The biosensor is disposed, for example, on the seat surface, steering wheel, etc., and detects biological information about occupants sitting in the seat or drivers holding the steering wheel. The vehicle information detection unit 7500 can calculate the driver's fatigue level or concentration level based on the detection information input from the driver state detector 7510, or it can determine whether the driver is drowsy. The vehicle information detection unit 7500 can perform processing such as noise removal processing on the collected sound signals.

[0256] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. An input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 is implemented, for example, through a device that allows occupants to input operations, such as a touch panel, button, microphone, switch, or joystick. Data obtained by performing voice recognition on voice input via a microphone can be input to the integrated control unit 7600. The input unit 7800 can be, for example, a remote control device using infrared or other radio waves, or an external connection device associated with the operation of the vehicle control system 7000, such as a mobile phone or personal digital assistant (PDA). The input unit 7800 can be, for example, a camera, in which case the occupant can input information via gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the occupant can be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by the occupant or others using the input unit 7800, and outputs the input signal to the integrated control unit 7600. Through the operation input unit 7800, occupants and others can input various data into the vehicle control system 7000 or instruct processing operations.

[0257] The storage unit 7690 may include a read-only memory (ROM) for storing various programs to be executed by a microcomputer, and a random access memory (RAM) for storing various parameters, calculation results, sensor values, etc. Furthermore, the storage unit 7690 may be implemented using magnetic storage devices such as hard disk drives (HDDs), semiconductor storage devices, optical storage devices, magneto-optical storage devices, etc.

[0258] The Universal Communication I / F 7620 is a universal communication I / F that mediates communication with various devices existing in the external environment 7750. The Universal Communication I / F 7620 can implement cellular communication protocols such as Global System for Mobile Communications (GSM) (registered trademark), WiMAx, LTE, or LTE-A Advanced, or another wireless communication protocol such as Wireless LAN (also known as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). The Universal Communication I / F 7620 can, for example, connect to devices (e.g., application servers or control servers) existing on external networks (e.g., the Internet, cloud networks, or company-specific networks) via base stations or access points. Furthermore, the Universal Communication I / F 7620 can, for example, connect to terminals existing near the vehicle using peer-to-peer (P2P) technology (e.g., terminals of drivers, pedestrians, or shops, or machine-type communication (MTC) terminals).

[0259] The Dedicated Communications I / F 7630 is a communications I / F that supports communication protocols developed for use in vehicles. For example, the Dedicated Communications I / F 7630 can implement standard protocols such as Wireless Access in a Vehicle Environment (WAVE), Dedicated Short Range Communication (DSRC), or cellular communication protocols, where WAVE is a combination of the underlying IEEE 802.11p and the upper-layer IEEE 1609. The Dedicated Communications I / F 7630 typically performs V2X communication, which includes one or more of the following concepts: vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0260] The positioning unit 7640 receives, for example, GNSS signals from Global Navigation Satellite System (GNSS) satellites (e.g., Global Positioning System (GPS) signals from GPS satellites), performs positioning, and generates location information including the vehicle's latitude, longitude, and altitude. Note that the positioning unit 7640 can identify its current location by exchanging signals with a wireless access point, or it can obtain location information from a terminal such as a mobile phone, PHS, or a smartphone with positioning capabilities.

[0261] The beacon receiving unit 7650 receives radio waves or electromagnetic waves, for example, from a radio station installed on the road, and obtains information such as current location, traffic congestion, road closures, or estimated time. Note that the functionality of the beacon receiving unit 7650 can be included in the dedicated communication I / F 7630 described above.

[0262] The vehicle-mounted device I / F 7660 is a communication interface that mediates the connection between the microcomputer 7610 and various vehicle-mounted devices 7760 present in the vehicle. The vehicle-mounted device I / F 7660 can establish a wireless connection using wireless communication protocols such as Wireless LAN, Bluetooth (registered trademark), Near Field Communication (NFC), or Wireless USB (WUSB). Furthermore, the vehicle-mounted device I / F 7660 can establish a wired connection such as Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI) (registered trademark), or Mobile High Definition Link (MHL) via connection terminals not shown (and, if necessary, cables). The vehicle-mounted device 7760 may include, for example, at least one of occupant-owned mobile or wearable devices, and information devices carried in or attached to the vehicle. Additionally, the vehicle-mounted device 7760 may include a navigation device that searches for routes to any destination. The vehicle-mounted device I / F 7660 exchanges control signals or data signals with these vehicle-mounted devices 7760.

[0263] The vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The vehicle network I / F 7680 sends and receives signals according to predetermined protocols supported by the communication network 7010.

[0264] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information acquired via at least one of the following: general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiving unit 7650, vehicle-mounted device I / F 7660, and vehicle network I / F 7680. For example, the microcomputer 7610 can calculate control target values ​​for the drive force generation device, steering mechanism, or braking device based on the acquired information about the vehicle's interior and exterior, and output control commands to the drive system control unit 7100. For example, the microcomputer 7610 can perform cooperative control to realize the functions of an advanced driver assistance system (ADAS), including collision avoidance or impact mitigation, following based on inter-vehicle distance, vehicle speed maintenance, vehicle collision warning, and vehicle lane departure warning. In addition, the microcomputer 7610 can also control the drive force generation device, steering mechanism, and braking device based on the acquired information about the vehicle's surroundings, thereby performing cooperative control such as autonomous driving that does not rely on driver operation.

[0265] The microcomputer 7610 can generate three-dimensional distance information between the vehicle and objects such as surrounding structures or people based on information acquired via at least one of a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, and an in-vehicle network I / F 7680, and create local map information including information about the vehicle's current location and its surroundings. Furthermore, the microcomputer 7610 can predict hazards, such as vehicle collisions, pedestrian approach, or entry into closed roads, based on the acquired information and generate warning signals. These warning signals may, for example, be signals used to generate warning sounds or activate warning lights.

[0266] The sound / image output unit 7670 sends an output signal of at least one of sound and image to an output device capable of visually or audibly notifying vehicle occupants or the outside of the vehicle. Figure 34 In the example, an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are shown as output devices. The display unit 7720 may include, for example, at least one of an onboard display and a head-up display. The display unit 7720 may have augmented reality (AR) display functionality. The output device may be another device different from these, such as headphones, a wearable device (such as glasses-type displays worn by occupants), a projector, or a lamp. When the output device is a display device, the display device visually displays, in various formats (such as text, images, tables, and graphics), the results obtained through various processes performed by the microcomputer 7610 or information received from another control unit. Furthermore, when the output device is a sound output device, the sound output device converts audio signals, including replayed sound data, acoustic data, etc., into analog signals and audibly outputs the analog signals.

[0267] Note that in Figure 34 In the example shown, at least two control units connected via communication network 7010 can be integrated into one control unit. Alternatively, each control unit may include multiple control units. Furthermore, the vehicle control system 7000 may include another control unit (not shown). Additionally, some or all of the functions performed by any control unit as described above can be provided to another control unit. That is, any control unit can perform predetermined computational processing as long as information is sent and received via communication network 7010. Similarly, sensors or devices connected to any control unit can be connected to another control unit, and multiple control units can send and receive detection information to each other via communication network 7010.

[0268] The foregoing describes examples of vehicle control systems to which the technology according to this disclosure can be applied. The technology according to this disclosure can be applied, for example, to imaging units 7910, 7912, 7914, 7916 and 7918, external information detectors 7920, 7922, 7924, 7926, 7928 and 7930, driver status detector 7510, etc., as described above. Specifically, imaging devices including the technology of this disclosure... Figure 1 The imaging system 10 can be applied to these imaging units and detectors. Then, by applying the techniques according to this disclosure, the effects of noise events such as sensor noise can be mitigated, and the occurrence of real events can be reliably and quickly sensed, thereby enabling safe vehicle operation.

[0269] Note that this technology can have the following configurations.

[0270] (1) An imaging device, comprising:

[0271] Multiple photoelectric conversion elements, each of which is configured to photoelectrically convert incident light to generate an electrical signal;

[0272] Multiple detectors, each configured to output a detection signal if the absolute value of the change in the electrical signal generated by each of the multiple photoelectric conversion elements exceeds a predetermined threshold; and

[0273] The threshold adjustment unit is configured to adjust the threshold based on the detection status of the corresponding detection signals from multiple detectors.

[0274] (2) The imaging apparatus according to item (1), wherein the detection status of the plurality of detectors includes at least one of the following: the number of detection signals detected in a predetermined area within a predetermined period, the ratio of the number of detectors that output detection signals to the total number of detectors in the predetermined area, and the signal level of the detection signals.

[0275] (3) The imaging apparatus according to item (2), wherein the threshold adjustment unit adjusts the threshold in at least one of the following cases: the number of detection signals in the predetermined area within a predetermined period is greater than or equal to a first reference value and the number of detection signals in the predetermined area within a predetermined period is less than a second reference value.

[0276] (4) The imaging apparatus according to item (3), wherein when the number of detection signals in a predetermined area within a predetermined period is equal to or greater than a first reference value, the threshold adjustment unit adjusts the threshold so that the detection sensitivity of the detector is low.

[0277] (5) The imaging apparatus according to item (3), wherein when the number of detection signals in a predetermined area within a predetermined period is less than a second reference value, the threshold adjustment unit adjusts the threshold so that the detector has high detection sensitivity.

[0278] (6) An imaging apparatus according to any one of items (1) to (5), wherein,

[0279] The thresholds include a first threshold and a second threshold.

[0280] The detectors include:

[0281] The first detector is configured to detect the first detection signal when the absolute value of the change in the electrical signal along the increasing direction exceeds a first threshold.

[0282] The second detector is configured to detect a second detection signal when the absolute value of the change in the electrical signal along a decreasing direction exceeds a second threshold; and

[0283] The threshold adjustment unit adjusts the first threshold and the second threshold based on the detection status in the first detector and the second detector.

[0284] (7) The imaging apparatus according to item (6) further includes:

[0285] The first determining unit is configured to determine whether the detection status in the first detector is within a first permissible range, and

[0286] The second determining unit is configured to determine whether the detection status in the second detector is within a second permissible range, wherein...

[0287] When the first determining unit determines that the detection condition is not within the first allowable range, the threshold adjustment unit adjusts the first threshold, and when the second determining unit determines that the detection condition is not within the second allowable range, the threshold adjustment unit adjusts the second threshold.

[0288] (8) An imaging device, comprising:

[0289] A plurality of first photoelectric conversion elements, each of which is configured to photoelectrically convert incident light to generate an electrical signal;

[0290] A plurality of detectors, each configured to output a detection signal if the absolute value of the change in the electrical signal generated by each of a plurality of first photoelectric conversion elements exceeds a predetermined threshold; and

[0291] The region extraction unit is configured to extract a portion of the viewing area within which multiple first photoelectric conversion elements perform photoelectric conversion based on the detection status of detection signals from multiple detectors.

[0292] (9) The imaging apparatus according to item (8), wherein the region extraction unit extracts a portion of the region based on the position of the output detection signal within the viewing angle in which the photoelectric conversion is performed by a plurality of first photoelectric conversion elements.

[0293] (10) The imaging apparatus according to item (8) or (9) further includes:

[0294] The object recognition unit is configured to identify objects existing within the viewing angle where photoelectric conversion is performed by multiple first photoelectric conversion elements based on the detection status of detection signals from multiple detectors.

[0295] The region extraction unit extracts a portion of the object identified by the object recognition unit.

[0296] (11) The imaging apparatus according to any one of items (8) to (10) further includes: a notification unit configured to notify at least one of a plurality of first photoelectric conversion elements and a plurality of detectors of information about a portion of the region.

[0297] (12) The imaging apparatus according to item (11), wherein the notification unit notifies the detector of information about the appropriate number of detection signals to be detected by multiple detectors and information about a portion of the region.

[0298] (13) The imaging apparatus according to item (12), wherein multiple detectors adjust thresholds based on information about an appropriate number of detection signals notified by the notification unit.

[0299] (14) The imaging apparatus according to any one of items (11) to (13) further includes:

[0300] The information processing unit is configured to generate operating condition information based on the detection status of detection signals from multiple detectors. This operating condition information includes at least one of the following: the photoelectric conversion speed of the multiple first photoelectric conversion elements, the ratio of the first photoelectric conversion element performing the photoelectric conversion to the multiple first photoelectric conversion elements, and the activation frequency of the multiple first photoelectric conversion elements.

[0301] The notification unit notifies at least one of the plurality of first photoelectric conversion elements and the plurality of detectors of operating condition information.

[0302] (15) The imaging apparatus according to item (8) or (9), wherein the region extraction unit outputs event information based on a detection signal output from a detector associated with a first photoelectric conversion element located in a portion of the region.

[0303] (16) The imaging device according to item (15), wherein the region extraction unit outputs information about the coordinate position of a portion of the region and event information.

[0304] (17) The imaging apparatus according to any one of items (8) to (16) further includes: an imaging unit comprising a plurality of second photoelectric conversion elements, each of the plurality of second photoelectric conversion elements photoelectrically converting incident light to generate an electrical signal, and the imaging unit being configured to output image data in a portion region based on the electrical signal.

[0305] (18) The imaging apparatus according to item (17) further includes: an image correction unit configured to correct image data based on event information based on a detection signal output from a detector associated with a first photoelectric conversion element located in a portion of the region.

[0306] (19) An imaging method comprising: outputting a detection signal when the absolute value of the change in an electrical signal generated by a plurality of photoelectric conversion elements exceeds a predetermined threshold, each of the plurality of photoelectric conversion elements photoelectrically converting incident light to generate an electrical signal; and adjusting the threshold based on a detection condition of the detection signal.

[0307] (20) According to the imaging method of item (19), the method further includes: extracting a portion of the field of view in which multiple photoelectric conversion elements perform photoelectric conversion based on the detection status of the detection signal.

[0308] This disclosure is not limited to the embodiments described above, but includes various modifications that may be conceived by those skilled in the art, and the effects of this disclosure are not limited to the foregoing. That is, various additions, modifications, and partial deletions may be made without departing from the conceptual idea and spirit of this disclosure as defined in the claims and their equivalents.

[0309] Reference tag list

[0310] 10 Imaging System

[0311] 11 Imaging Lens

[0312] 12 Recording Units

[0313] 13 Controllers

[0314] 20 Imaging devices

[0315] 21-pixel array unit

[0316] 22 drive units

[0317] 23 Arbitration Units

[0318] 24-column processing unit

[0319] 25 Signal Processing Units

[0320] 27 Reading Region Selection Unit

[0321] 28 Signal Generation Units

[0322] 30 pixels

[0323] 31 Optical receiving unit

[0324] 32-pixel signal generation unit

[0325] 33 Address Event Detector

[0326] 51 Event Counter

[0327] 52 Threshold Adjustment Unit

[0328] 53 Frequency Determination Unit

[0329] 54 Event Output Processing Unit

[0330] 55 CPUs.

Claims

1. An imaging device, comprising: A plurality of photoelectric conversion elements, each of which is configured to photoelectrically convert incident light to generate an electrical signal; A plurality of detectors, each of which is configured to output a detection signal if the absolute value of the change in the electrical signal generated by each of the plurality of photoelectric conversion elements exceeds a predetermined threshold. as well as The threshold adjustment unit is configured to adjust the threshold based on the detection status of the detection signals from the plurality of detectors. The imaging device further includes an information processing unit configured to generate operating condition information based on the detection status of the detection signals from the plurality of detectors. The operating condition information includes at least one of the following: the photoelectric conversion speed of the plurality of photoelectric conversion elements, the ratio of the photoelectric conversion element performing the photoelectric conversion to the plurality of photoelectric conversion elements, and the activation frequency of the plurality of photoelectric conversion elements.

2. The imaging device according to claim 1, wherein, The detection status of the plurality of detectors includes at least one of the following: the number of detection signals detected in a predetermined area within a predetermined period, the ratio of the number of detectors outputting the detection signals to the total number of detectors in the predetermined area, and the signal level of the detection signals.

3. The imaging device according to claim 2, wherein, The threshold adjustment unit adjusts the threshold in at least one of the following situations: the number of detected signals in the predetermined area within the predetermined period is greater than or equal to a first reference value, and the number of detected signals in the predetermined area within the predetermined period is less than a second reference value.

4. The imaging device according to claim 3, wherein, If the number of detected signals in the predetermined area within the predetermined period is equal to or greater than the first reference value, the threshold adjustment unit adjusts the threshold to make the detector's detection sensitivity low.

5. The imaging device according to claim 3, wherein, If the number of detected signals in the predetermined area within the predetermined period is less than the second reference value, the threshold adjustment unit adjusts the threshold to make the detector have high detection sensitivity.

6. The imaging apparatus according to claim 1, wherein, The threshold includes a first threshold and a second threshold. The plurality of detectors includes: A first detector is configured to detect a first detection signal when the absolute value of the change in the electrical signal along the increasing direction exceeds the first threshold. The second detector is configured to detect a second detection signal if the absolute value of the change in the electrical signal along the decreasing direction exceeds the second threshold; and The threshold adjustment unit adjusts the first threshold and the second threshold based on the detection status of the first detector and the second detector.

7. The imaging apparatus according to claim 6, further comprising: The first determining unit is configured to determine whether the detection status in the first detector is within a first allowable range; as well as The second determining unit is configured to determine whether the detection status in the second detector is within a second allowable range, wherein... When the first determining unit determines that the detection condition is not within the first allowable range, the threshold adjustment unit adjusts the first threshold, and when the second determining unit determines that the detection condition is not within the second allowable range, the threshold adjustment unit adjusts the second threshold.

8. An imaging device, comprising: A plurality of first photoelectric conversion elements, each of which is configured to photoelectrically convert incident light to generate an electrical signal; A plurality of detectors, each of which is configured to output a detection signal if the absolute value of the change in the electrical signal generated by each of the plurality of first photoelectric conversion elements exceeds a predetermined threshold. as well as The region extraction unit is configured to extract a portion of the viewing angle within which the plurality of first photoelectric conversion elements perform photoelectric conversion, based on the detection status of the detection signals from the plurality of detectors. The imaging device further includes an information processing unit configured to generate operating condition information based on the detection status of the detection signals from the plurality of detectors. The operating condition information includes at least one of the following: the photoelectric conversion speed of the plurality of first photoelectric conversion elements, the ratio of the first photoelectric conversion element performing the photoelectric conversion to the plurality of first photoelectric conversion elements, and the activation frequency of the plurality of first photoelectric conversion elements.

9. The imaging apparatus according to claim 8, wherein, The region extraction unit extracts the partial region based on the position of the output detection signal within the viewing angle of the plurality of first photoelectric conversion elements performing photoelectric conversion.

10. The imaging apparatus according to claim 8, further comprising: An object recognition unit is configured to identify objects existing within the viewing angle where photoelectric conversion is performed by the plurality of first photoelectric conversion elements, based on the detection status of the detection signals from the plurality of detectors. The region extraction unit extracts the partial regions of the object identified by the object recognition unit.

11. The imaging apparatus according to claim 8, further comprising: The notification unit is configured to notify at least one of the plurality of first photoelectric conversion elements and the plurality of detectors of information about the partial region.

12. The imaging apparatus according to claim 11, wherein, The notification unit notifies the plurality of detectors of the appropriate number of detection signals to be detected by the plurality of detectors, as well as information about the partial region.

13. The imaging apparatus according to claim 12, wherein, The plurality of detectors adjust the threshold based on information about the appropriate number of detection signals notified by the notification unit.

14. The imaging apparatus according to claim 11, wherein, The notification unit notifies at least one of the plurality of first photoelectric conversion elements and the plurality of detectors of the operating condition information.

15. The imaging apparatus according to claim 8, wherein, The region extraction unit outputs event information based on the detection signals output from the plurality of detectors associated with the plurality of first photoelectric conversion elements located in the partial region.

16. The imaging apparatus according to claim 15, wherein, The region extraction unit outputs information about the coordinates of the partial region and the event information.

17. The imaging apparatus according to claim 8, further comprising: An imaging unit includes a plurality of second photoelectric conversion elements, each of which photoelectrically converts incident light to generate an electrical signal, and the imaging unit is configured to output image data of the partial region based on the electrical signal.

18. The imaging apparatus according to claim 17, further comprising: An image correction unit is configured to correct the image data based on event information, the event information being based on detection signals output from a plurality of detectors associated with the plurality of first photoelectric conversion elements located in the partial region.

19. An imaging method, comprising: A detection signal is output when the absolute value of the change in the electrical signal generated by multiple photoelectric conversion elements exceeds a predetermined threshold, wherein each of the multiple photoelectric conversion elements photoelectrically converts the incident light to generate the electrical signal; as well as The threshold is adjusted based on the detection status of the detection signal. The operation condition information is generated based on the detection status of the detection signal. The operation condition information includes at least one of the following: the photoelectric conversion speed of the plurality of photoelectric conversion elements, the ratio of the photoelectric conversion element performing the photoelectric conversion to the plurality of photoelectric conversion elements, and the activation frequency of the plurality of photoelectric conversion elements.

20. The imaging method according to claim 19, further comprising: Based on the detection status of the detection signal, a portion of the viewing area within which the multiple photoelectric conversion elements perform photoelectric conversion is extracted.

Images