Sensor devices and reading methods

Abstract

The sensor device according to the present technology includes: a pixel array unit in which a plurality of pixel units are arranged in two dimensions, each pixel unit having one or more pixels and capable of generating a grayscale signal representing the intensity of light reception and an event signal representing a change in light reception; and a line control unit capable of sequentially selecting pixels for reading the event signal and selecting pixels for reading the grayscale signal at different times.

Description

Technical Field

[0001] This technology relates to a sensor device and a reading method therein, the sensor device including a pixel array unit in which a plurality of pixel units are arranged in two dimensions, each pixel unit having a pixel and capable of generating a grayscale signal representing the intensity of light received and an event signal representing a change in the amount of light received. Background Technology

[0002] For example, as in known dynamic vision sensors (DVS), there exist sensor devices that detect event signals representing changes in the amount of light received by each pixel. In this type of sensor device, there exists a sensor device in which pixels for event signal detection and pixels for grayscale signal detection are mixed, so that the grayscale signal (a signal representing the intensity of the amount of light received) of the pixel in which the event has been detected can be acquired (for example, see Patent Document 1 below).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: US 2014-9648 A Summary of the Invention

[0006] The problem to be solved by the present invention

[0007] Here, Patent Document 1 discloses a configuration in which an event signal and a grayscale signal are read using an arbitrator method for a pixel array unit, wherein pixels for event signal (motion) detection and pixels for grayscale signal (color) detection are mixed. However, since the arbitrator method is assumed, the reading times of the event signal and the grayscale signal are based on the event detection time of the corresponding pixel. Furthermore, Patent Document 1 only discloses selecting and outputting one of the event signal and the grayscale signal, and for a pixel in which an event has been detected, reading of the grayscale signal and reading of the event signal cannot be performed simultaneously.

[0008] In view of the above, this technology is proposed, and its purpose is to provide a sensor device capable of simultaneously reading event signals and grayscale signals.

[0009] Solution to the problem

[0010] The sensor device according to the present technology includes: a pixel array unit, wherein a plurality of pixel units are arranged in two dimensions, each pixel unit having one or more pixels and capable of generating a grayscale signal representing the intensity of light reception and an event signal representing a change in light reception; and a row control unit capable of sequentially performing row selection for pixels whose event signals are to be read and selection for pixels whose grayscale signals are to be read at different times.

[0011] Based on the above configuration, both the event signal and grayscale signal of each target pixel unit can be read within a frame period.

[0012] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which a pixel unit includes a pixel, and grayscale signal generation and event signal generation are performed alternately based on the charge generated in the light-receiving element included in the pixel.

[0013] Therefore, event signals and grayscale signals can be read in pixels.

[0014] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the row control unit performs row selection for reading event signals and row selection for reading grayscale signals for all pixel rows.

[0015] Therefore, it is possible to simultaneously read event signals and grayscale signals for all pixel units in frame periods.

[0016] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the line control unit performs line selection for reading grayscale signals based on the determination result of the presence or absence of an event in the event signal.

[0017] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the row control unit performs row selection for reading event signals for all pixel units, and performs row selection for reading grayscale signals based on the result of determining whether an event has occurred or not based on the event signals read through the row selection.

[0018] Therefore, based on the determination of whether an event has occurred in all pixel units, grayscale signals can be read only for the pixel units where the event has been identified.

[0019] In the sensor device according to the present technology described above, it is conceivable to include a grayscale output unit that outputs grayscale signals read from pixel units, and the grayscale output unit selectively outputs grayscale signals of pixel units whose events have been determined from the grayscale signals read in rows by the row selection of the row control unit.

[0020] Therefore, it is not necessary to perform circuitry operations for outputting grayscale signals at column positions where no event has been identified.

[0021] In the sensor device according to the present technology described above, it is conceivable to include a first reading determination unit, which counts the number of times an event occurs based on an event signal of a unit pixel region including pixel units in a predetermined row, and determines whether the grayscale signal of the unit pixel region can be read based on the counted number of events.

[0022] A unit pixel region refers to a region comprising n (n≥1) rows of pixel units. Typically, events occur at a certain number of pixels. Therefore, when the number of events is very small (e.g., once), it can be estimated that the event has been incorrectly detected due to noise. Thus, by determining whether a grayscale signal can be read based on the number of events as described above, grayscale signal readings can be prevented for unit pixel regions where events have been incorrectly detected due to noise.

[0023] In the sensor device according to the present technology described above, it is conceivable to include a second reading determination unit, which performs object recognition processing based on an event signal of a unit pixel region comprising pixel units in a plurality of predetermined rows, and determines whether the grayscale signal of the unit pixel region can be read based on the presence or absence of object recognition through the object recognition processing.

[0024] Therefore, for a unit pixel region comprising multiple predetermined rows, the grayscale signal can be read if the object is identified, and not read if the object is not identified.

[0025] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which a first reading determination unit performs noise removal processing on a unit pixel region, the noise removal processing being a process of removing events estimated to be erroneously detected due to noise, and determines whether the grayscale signal of the unit pixel region can be read based on the number of occurrences of the events after the noise removal processing.

[0026] Therefore, it is possible to prevent reading grayscale signals for unit pixel areas where events are erroneously detected due to noise.

[0027] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the row control unit skips row selection for reading grayscale signals for rows that are determined not to read grayscale signals.

[0028] In row-order selection, by skipping row selection, the interval between the selection time of the row immediately before skipping and the selection time of the row immediately after skipping can be shortened to a single row interval.

[0029] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the line control unit resets the charge of all line light receiving elements, regardless of whether a grayscale signal is read.

[0030] Therefore, the charge of the optical receiving element can be periodically reset during the frame period.

[0031] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the pixel unit can generate a first polarity event signal indicating a change on the side of increasing light reception and a second polarity event signal indicating a change on the side of decreasing light reception as event signals.

[0032] Therefore, it is possible to identify whether the generated event is an event on the side of increasing or decreasing light reception.

[0033] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the line control unit determines the charge accumulation time related to the generation of the grayscale signal based on the change in the polarity of the event signal.

[0034] Since the first polarity event occurs on the side of increased light reception, a short charge accumulation time for the grayscale signal is sufficient. Conversely, since the second polarity event occurs on the side of decreased light reception, it is desirable to increase the charge accumulation time for the grayscale signal.

[0035] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the row control unit performs row selection for reading event signals before the charge accumulation associated with the generation of grayscale signals begins.

[0036] By performing row selection for reading event signals before charge accumulation begins, it is possible to determine whether a row's grayscale signal can be read based on the event signals prior to the start of charge accumulation.

[0037] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which, during the execution cycle of charge accumulation associated with the generation of grayscale signals, the line control unit performs line selection for reading event signals.

[0038] Therefore, compared to reading the event signal before charge accumulation begins, the event signal can be read at a time closer to the start of reading the grayscale signal. For example, the event signal can be read immediately before the grayscale signal is read.

[0039] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the row control unit simultaneously begins charge accumulation related to the generation of grayscale signals for all pixel units.

[0040] Therefore, for the charge accumulation of grayscale signals, it is not necessary to perform independent timing control on a line-by-line basis.

[0041] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which each pixel unit outputs a generated event signal to a line control unit, and the line control unit determines the line from which grayscale signals will be read based on the event signal input from each pixel unit.

[0042] Therefore, before reading the event signals of all pixel units in row order, it is possible to determine which row from which the grayscale signal should be read.

[0043] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the row control unit performs row selection for reading grayscale signals sequentially for the rows determined to be read.

[0044] Therefore, even when events occur in multiple pixel units, the grayscale signal of each pixel unit is not read randomly at each event occurrence time, but is read sequentially row by row.

[0045] In the sensor device according to the present technology described above, it is conceivable to have a configuration in which the pixel unit includes a pixel that generates an event signal and a pixel that generates a grayscale signal.

[0046] In this case, pixel units generate event signals and grayscale signals in different pixels.

[0047] Furthermore, the reading method according to this technology is a reading method in a sensor device, the sensor device including a pixel array unit, wherein a plurality of pixel units are arranged in two dimensions, each pixel unit having a pixel and capable of generating a grayscale signal representing the intensity of light reception and an event signal representing a change in light reception, the reading method including: sequentially selecting rows of pixels to read event signals and pixels to read grayscale signals at different times.

[0048] Similarly, this reading method can achieve effects similar to those of the sensor device according to the present technology described above. Attached Figure Description

[0049] Figure 1 This is a block diagram illustrating an example of the internal configuration of a sensor device according to the present technology as a first embodiment.

[0050] Figure 2 This is an explanatory diagram illustrating an example of pixel arrangement in a pixel array unit.

[0051] Figure 3 It is an equivalent circuit diagram of the pixel according to the implementation method.

[0052] Figure 4 This is an explanatory diagram of a modified example of a pixel unit.

[0053] Figure 5 This is an equivalent circuit diagram of one of the two types of pixels included in the pixel unit (the pixel used to generate the grayscale signal), which is a variation.

[0054] Figure 6 This is an equivalent circuit diagram of another pixel (the pixel used to generate event signals) among the two types of pixels included in the pixel unit, as a variation.

[0055] Figure 7 This is an illustrative diagram illustrating an example of the internal configuration of an event detection circuit.

[0056] Figure 8 This is a timing diagram used to illustrate the reading method as a first embodiment.

[0057] Figure 9 This is a timing diagram of an example of generating and reading event signals during an execution cycle associated with the generation of grayscale signals, as described in the first embodiment.

[0058] Figure 10 This is a timing diagram illustrating an example of generating and reading event signals multiple times per frame in the first embodiment.

[0059] Figure 11 This is a timing diagram of an example of performing electronic shutter operation on all pixels simultaneously in the first embodiment.

[0060] Figure 12 This is an equivalent circuit diagram showing an example of pixel configuration corresponding to a global shutter system.

[0061] Figure 13 This is a block diagram illustrating an example of the internal configuration of a sensor device as a second embodiment.

[0062] Figure 14 This is a timing diagram used to explain the reading method as a second embodiment.

[0063] Figure 15 This is a flowchart illustrating an example of the processing procedure of an event processing / output circuit for implementing the reading method as a second embodiment.

[0064] Figure 16 This is a flowchart illustrating an example of the processing procedure of a row control circuit for implementing the reading method as a second embodiment.

[0065] Figure 17This is also a flowchart illustrating an example of the processing procedure of the row control circuit used to implement the reading method as a second embodiment.

[0066] Figure 18 This is a timing diagram of an example of performing electronic shutter operation on all rows in the second embodiment.

[0067] Figure 19 This is a timing diagram of an example of generating and reading event signals during an execution cycle related to the generation of grayscale signals, as described in the second embodiment.

[0068] Figure 20 This is a timing diagram illustrating an example of how charge accumulation time changes depending on the polarity of the event signal.

[0069] Figure 21 yes Figure 20 The timing diagram is for a variation of the example shown.

[0070] Figure 22 This is a timing diagram of an example of line selection that skips grayscale signal reading.

[0071] Figure 23 This is a timing diagram of another example of line selection that skips grayscale signal reading.

[0072] Figure 24 This is a block diagram illustrating an example of the internal configuration of a sensor device as a third embodiment.

[0073] Figure 25 This is a flowchart illustrating an example of the processing procedure of an event processing / output circuit for implementing the reading method as a third embodiment.

[0074] Figure 26 This is a flowchart illustrating an example of the processing procedure of a grayscale output circuit for implementing the reading method as a third embodiment.

[0075] Figure 27 This is a block diagram illustrating an example of the internal configuration of a sensor device as a fourth embodiment.

[0076] Figure 28 This is an explanatory diagram of the request signal line in the fourth embodiment.

[0077] Figure 29 This is a timing diagram used to illustrate the reading method as a fourth embodiment.

[0078] Figure 30 This is a block diagram showing an example of the internal configuration of a sensor device as a variation of the fourth embodiment.

[0079] Figure 31This is a flowchart illustrating an example of a processing procedure executed by an event processing / output circuit to implement a reading method as a fifth embodiment.

[0080] Figure 32 This is a flowchart illustrating an example of a processing procedure executed by an event processing / output circuit to implement the reading method as a sixth embodiment.

[0081] Figure 33 This is a circuit diagram showing a modified configuration as a logarithmic conversion unit.

[0082] Figure 34 This is a circuit diagram showing a configuration as a variant of the buffer.

[0083] Figure 35 This is a circuit diagram showing a configuration as a variant of the quantizer.

[0084] Figure 36 This is a circuit diagram showing a modified configuration of the output stage as an event signal in a pixel.

[0085] Figure 37 This is a circuit diagram showing a specific example of the connection points between chips.

[0086] Figure 38 This is another example of a circuit diagram showing the connection points between chips.

[0087] Figure 39 This is another example of a circuit diagram showing the connection points between chips.

[0088] Figure 40 This is another example of a circuit diagram showing the connection points between chips. Detailed Implementation

[0089] In the following description, embodiments according to the present technology will be described in the following order with reference to the accompanying drawings.

[0090] <1. First Implementation Method>

[0091] [1-1. Configuration of the sensor device]

[0092] [1-2. Configuration of pixel array units]

[0093] [1-3. Reading method as the first embodiment]

[0094] <2. Second Implementation Method>

[0095] <3. Third Implementation Method>

[0096] <4. Fourth Embodiment>

[0097] <5. Fifth Implementation Method>

[0098] <6. Sixth Implementation Method>

[0099] <7. Variations>

[0100] <8. Overview of Implementation Methods>

[0101] <9. This technology>

[0102] <1. First Implementation Method>

[0103] [1-1. Configuration of the sensor device]

[0104] Figure 1 This is a block diagram illustrating an example of the internal configuration of a sensor device 1 according to the present technology as a first embodiment.

[0105] As shown, the sensor device 1 includes a pixel array unit 2, a row control circuit 3, an event processing / output circuit 4, and a grayscale output circuit 5.

[0106] The pixel array unit 2 has a configuration in which a plurality of pixel units 20 are arranged in a matrix in two dimensions in the row and column directions. Here, the row direction refers to the pixel arrangement direction in the horizontal direction, and the column direction refers to the pixel arrangement direction in the vertical direction. In the figure, the row direction is the horizontal direction, and the column direction is the vertical direction.

[0107] Each pixel unit 20 includes one or more pixels, and each pixel unit 20 is capable of generating a grayscale signal representing the intensity of light received and an event signal representing a change in light received. In this example, the pixel unit 20 has a single pixel as pixel 21, as described later, and pixel 21 is configured to generate both the grayscale signal and the event signal. This will be discussed later.

[0108] In pixel array unit 2, relative to the matrix pixel arrangement, row control lines Lc are routed along the row direction of each pixel row, and event vertical signal lines Li and grayscale vertical signal lines Lt are routed along the column direction of each pixel column.

[0109] The row control line Lc transmits various signals used to execute the drive when reading signals from pixels in pixel unit 20. It should be noted that... Figure 1 For ease of explanation, each row control line Lc is shown as a single wiring, but as described later, each row control line Lc is configured with multiple wirings. One end of each row control line Lc is connected to the output terminal corresponding to each row of the row control circuit 3.

[0110] For example, the line control circuit 3 includes a time generator, a shift register, an address decoder, etc., which generate various time signals, and drives each pixel unit 20 in the pixel array unit 2 by outputting various signals via the line control line Lc, and controls the generation and reading of event signals and grayscale signals.

[0111] Note that the driving method of the pixel unit 20 in this embodiment will be described again later.

[0112] The event vertical signal line Li is a wiring used to transmit the event signal read from the pixel unit 20 to the event processing / output circuit 4, and one end of each event vertical signal line Li is connected to the output terminal corresponding to each column of the event processing / output circuit 4.

[0113] The event processing / output circuit 4 acquires the event signal read from each pixel unit 20 through the event vertical signal line Li, performs predetermined signal processing, and outputs the signal.

[0114] The grayscale vertical signal line Lt is a wiring used to transmit the grayscale signal read from the pixel unit 20 to the grayscale output circuit 5, and one end of each grayscale vertical signal line Lt is connected to the output terminal corresponding to each column of the grayscale output circuit 5.

[0115] The grayscale output circuit 5 receives the grayscale signal read from each pixel unit 20 through the grayscale vertical signal line Lt, performs predetermined signal processing, such as analog-to-digital (A / D) conversion processing, and outputs the signal.

[0116] [1-2. Configuration of pixel array units]

[0117] Figure 2 This is an explanatory diagram illustrating an example of pixel arrangement in pixel array unit 2.

[0118] As shown, in the pixel array unit 2 of this example, the pixel units 20 are arranged in a two-dimensional matrix, and each pixel unit 20 includes a single pixel 21.

[0119] Each pixel 21 includes a single photodiode PD, and each pixel 21 is configured to alternately generate grayscale signals and event signals using the charge obtained in the photodiode PD.

[0120] Figure 3 This is the equivalent circuit diagram for pixel 21.

[0121] As shown, pixel 21 includes a photodiode PD as a photoelectric conversion element. Then, pixel 21 includes a grayscale transfer transistor Qtt, a floating diffuser FD, a reset transistor Qr, an amplification transistor Qat, and a grayscale selection transistor Qst as configured for the generation and reading of grayscale signals.

[0122] In addition, pixel 21 includes an event transfer transistor Qti, a logarithmic transformation unit 22, a buffer 23, an event detection circuit 24, a transistor Qp, a first event selection transistor Qsip, a transistor Qm, and a second event selection transistor Qsim, as a configuration related to the generation and reading of event signals.

[0123] In this example, the various transistors included in pixel 21 include, for example, metal-oxide-semiconductor field-effect transistors (MOSFETs).

[0124] In addition, the line control line Lc1 for transmitting the grayscale transfer drive signal TG-T, the line control line Lc2 for transmitting the charge reset signal RST-T, the line control line Lc3 for transmitting the grayscale selection signal SLC-T, the line control line Lc4 for transmitting the event transfer drive signal TG-I, the line control line Lc5 for transmitting the reference level reset signal RST-I, and the line control line Lc6 for transmitting the event selection signal SLC-I are routed to pixel 21 as the aforementioned line control line Lc.

[0125] First, the configuration related to the generation and reading of grayscale signals will be described.

[0126] The grayscale transfer transistor Qtt has a gate connected to the horizontal control line Lc1. When the grayscale transfer drive signal TG-T provided from the horizontal control line Lc1 is turned on, it becomes on and transfers the signal charge accumulated in the photodiode PD to the floating diffuser FD.

[0127] Floating diffuser (FD) is a charge retention unit that temporarily retains the charge transferred from the photodiode (PD).

[0128] The reset transistor Qr has a gate connected to the row control line Lc2 and becomes on when the charge reset signal RST-T provided from the row control line Lc2 is turned on, and resets the potential of the floating diffuse FD to the reference potential VDD.

[0129] The source of the amplifying transistor Qat is connected to the grayscale vertical signal line Lt via the grayscale selection transistor Qst, and its drain is connected to the reference potential VDD (constant current source), thus forming a source follower circuit.

[0130] The grayscale selection transistor Qst is connected between the source of the amplifying transistor Qat and the grayscale vertical signal line Lt, and has a gate connected to the horizontal control line Lc3. When the grayscale selection signal SLC-T provided to the gate from the horizontal control line Lc3 is turned on, the grayscale selection transistor Qst becomes on, and the charge held in the floating diffusion FD is output to the grayscale vertical signal line Lt via the amplifying transistor Qat.

[0131] The operation of pixel 21 related to the generation and reading of grayscale signals will be briefly described.

[0132] Note that, as a prerequisite, the event transfer transistor Qti is in the off state from the start of generating the grayscale signal until its readout.

[0133] First, a charge reset operation (electronic shutter operation) is performed to reset the charge of pixel 21 before light reception begins. That is, the reset transistor Qr and the grayscale transfer transistor Qtt are turned on (conducted), and the accumulated charge in the photodiode PD and the floating diffuser FD is reset.

[0134] After resetting the accumulated charge, the reset transistor Qr and the grayscale transfer transistor Qtt are turned off to begin charge accumulation in the photodiode PD. Subsequently, when the charge signal accumulated in the photodiode PD is read, the grayscale transfer transistor Qtt is turned on, and the grayscale selection transistor Qst is turned on. Therefore, the charge signal is transferred from the photodiode PD to the floating diffuser FD, and the charge signal held in the floating diffuser FD is output to the grayscale vertical signal line Lt via the amplification transistor Qat.

[0135] This section describes the configuration related to the generation and reading of event signals.

[0136] The gate of the event transfer transistor Qti is connected to the row control line Lc4. When the event transfer drive signal TG-I provided from the row control line Lc4 is turned on, it becomes on and transfers the charge accumulated in the photodiode PD to the logarithmic conversion unit 22.

[0137] The logarithmic conversion unit 22 converts the photocurrent (corresponding to the current of light reception) obtained from the photodiode PD into its logarithmic voltage signal.

[0138] Buffer 23 corrects the voltage signal input from logarithmic conversion unit 22 and outputs the corrected voltage signal to event detection circuit 24.

[0139] As shown, the logarithmic transformation unit 22 includes transistors Q1, Q2, and Q3. In this example, transistors Q1 and Q3 are N-type transistors, and transistor Q2 is a P-type transistor.

[0140] The source of transistor Q1 is connected to the cathode of photodiode PD via event transfer transistor Qti, and its drain is connected to the power supply terminal (reference potential VDD).

[0141] Transistors Q2 and Q3 are connected in series between the power supply terminal and the ground terminal. Furthermore, the connection point between transistors Q2 and Q3 is connected to the gate of transistor Q1 and the input terminal of buffer 23 (the gate of transistor Q5, described later). Additionally, a predetermined bias voltage Vbias is applied to the gate of transistor Q2.

[0142] The drains of transistors Q1 and Q3 are connected to the power supply side (reference potential VDD), forming a source follower circuit. The photocurrent from the photodiode PD is converted into a logarithmic voltage signal by the two source followers connected in a loop. Additionally, transistor Q2 provides a constant current to transistor Q3.

[0143] The buffer 23 includes transistors Q4 and Q5 as P-type transistors, and these transistors Q4 and Q5 are connected in series between the power supply terminal and the ground terminal.

[0144] The connection point between transistors Q4 and Q5 is set as the output terminal of buffer 23, and the corrected voltage signal is output from the output terminal to the event detection circuit 24 as a light receiving signal.

[0145] The event detection circuit 24 detects changes in the amount of light received as an event by using the level of a past optical received signal as a reference level Lref to obtain the difference between the current level and the reference level Lref. Specifically, the event detection circuit 24 detects the presence or absence of an event based on whether the level (absolute value) of the difference signal, which represents the difference between the reference level Lref and the current level of the optical received signal, is equal to or greater than a predetermined threshold.

[0146] The event detection circuit 24 in this example is configured to detect events in which the light received amount changes to the increasing side, i.e., the event in which the difference from the reference level Lref becomes positive (hereinafter referred to as the "first polarity event") and events in which the light received amount changes to the decreasing side, i.e., the event in which the difference from the reference level Lref becomes negative (hereinafter referred to as the "second polarity event").

[0147] The event detection circuit 24 outputs a signal representing the detection result of the first polarity event as the first polarity event signal Vop, and outputs a signal representing the detection result of the second polarity event as the second polarity event signal Vom.

[0148] Here, the event detection circuit 24 resets the reference level Lref to the current level of the optical receiving signal based on the reference level reset signal RST-I input via the row control line Lc5.

[0149] By resetting the reference level Lref in this manner, new events can be detected based on changes in the optical received signal level starting from the point in time when the reset is performed. That is, resetting the reference level Lref serves as a process for controlling the event detection circuit 24 to a state where it can detect new events.

[0150] Note that an example of the internal circuit configuration of the event detection circuit 24 will be described again.

[0151] Transistor Qp and first event selection transistor Qsip are used as the selection output circuit for the first polarity event signal Vop, and transistor Qm and second event selection transistor Qsim are used as the selection output circuit for the second polarity event signal Vom.

[0152] Here, in this example, based on the relationship of detecting the first polarity event signal Vop and the second polarity event signal Vom as event signals, the first event vertical signal line Lip and the second event vertical signal line Lim are set as event vertical signal line Li.

[0153] As shown, transistor Qp and first event selection transistor Qsip are connected in series between the first event vertical signal line Lip and the ground terminal, and the first polarity event signal Vop is provided to the gate of transistor Qp.

[0154] In addition, transistor Qm and second event selection transistor Qsim are connected in series between the second event vertical signal line Lim and the ground terminal, and the second polarity event signal Vom is provided to the gate of transistor Qm.

[0155] The gates of the first event selection transistor Qsip and the second event selection transistor Qsim are connected to the row control line Lc6, respectively.

[0156] When the event selection signal SLC-I provided from the row control line Lc6 to the gate is turned on, the first event selection transistor Qsip turns on and outputs the first polarity event signal Vop to the first event vertical signal line Lip.

[0157] When the event selection signal SLC-I provided from the row control line Lc6 to the gate is turned on, the second event selection transistor Qsim is turned on and outputs the second polarity event signal Vom to the second event vertical signal line Lim.

[0158] In pixel 21, during different periods from the start of grayscale signal generation to the reading of the grayscale signal, the event transfer drive signal TG-I is turned on, and the light receiving signal corresponding to the accumulated charge of the photodiode PD is input to the event detection circuit 24, generating a first polarity event signal Vop and a second polarity event signal Vom. When the first polarity event signal Vop and the second polarity event signal Vom are read, the event selection signal SLC-I is turned on, and the first polarity event signal Vop and the second polarity event signal Vom are output to the first event vertical signal line Lip and the second event vertical signal line Lim, respectively.

[0159] It should be noted that regarding the "pixel unit," as a configuration capable of generating both grayscale signals and event signals, it can be adopted as follows: Figure 4 The configuration shown is a mixture of pixel 21-T, which generates grayscale signals, and pixel 21-I, which generates event signals, instead of using a single pixel as described above that can generate both grayscale signals and event signals.

[0160] In the following text, a pixel unit that can generate both grayscale signals and event signals by including pixels 21-T that generate grayscale signals and pixels 21-I that generate event signals in this manner will be referred to as "pixel unit 20A". Furthermore, a pixel array unit in which pixel units 20A are arranged in a two-dimensional matrix will be referred to as "pixel array unit 2A".

[0161] Figure 4 An example is shown where pixel unit 20A comprises 2×2=4 pixels, one pixel being pixel 21-I and the remaining three being pixels 21-T, with pixel 21-I positioned in the upper left corner of pixel unit 20A. However, the number of pixels 21-I and pixels 21-T constituting pixel unit 20A, as well as the arrangement pattern of pixels 21-I and pixels 21-T in pixel unit 20A, are not limited to this.

[0162] Figure 5 This is the equivalent circuit diagram of pixel 21-T, and Figure 6 This is the equivalent circuit diagram of pixel 21-I.

[0163] From the above and Figure 3 A comparison shows that the equivalent circuit configuration of pixel 21-T is similar to that described above. Figure 3 The equivalent circuit configuration related to the generation and reading of grayscale signals in the image is described as the equivalent circuit configuration related to the generation and reading of event signals.

[0164] It should be noted that, Figure 4In the case of the arrangement mode of pixels 21-T and pixels 21-I shown, the row control lines Lc1, Lc2 and Lc3 related to the generation and reading of grayscale signals are routed for each pixel row in pixel array unit 2A, and the row control lines Lc4, Lc5 and Lc6 related to the generation and reading of event signals only need to be routed every other row.

[0165] Here, pixel unit 20A may have a configuration in which pixel 21 is arranged instead of pixel 21-I. Therefore, grayscale signals can also be generated and read at the arrangement position of pixel 21-I.

[0166] Furthermore, the pixel unit 20A may have a configuration where pixels 21 are arranged at all pixel positions. In this case, by driving control of the pixels 21, a predetermined number of pixels 21 in the pixel unit 20A are used as pixels for generating and reading grayscale signals, and the remaining pixels 21 are used as pixels for generating and reading event signals.

[0167] Figure 7 This is an explanatory diagram of an example of the internal configuration of the event detection circuit 24, and shows an example of the internal circuit configuration of the photodiode PD, the logarithmic conversion unit 22 and the buffer 23, and the event detection circuit 24.

[0168] As shown, the event detection circuit 24 includes a subtractor 25 and a quantizer 26.

[0169] Subtractor 25 reduces the level of the optical received signal (voltage signal) from buffer 23 according to the reference level reset signal RST-I. Subtractor 25 outputs the reduced optical received signal to quantizer 26.

[0170] Quantizer 26 quantizes the optical received signal from subtractor 25 into a digital signal and outputs the digital signal as an event signal (in this example, a first polarity event signal Vop and a second polarity event signal Vom).

[0171] Subtractor 25 includes capacitors C1 and C2, transistors Q7 and Q8, and a reset switch SWr. Transistor Q7 is a P-type transistor, and transistor Q8 is an N-type transistor.

[0172] Transistors Q7 and Q8 are connected in series between the power supply terminal and the ground terminal, forming an inverter. Specifically, the source of transistor Q7 is connected to the power supply terminal, the drain of transistor Q7 is connected to the drain of transistor Q8, and the source of transistor Q8 is connected to the ground terminal. Note that a voltage Vbdif is applied to the gate of transistor Q8.

[0173] One end of capacitor C1 is connected to the output terminal of buffer 23 and the other end is connected to the gate of transistor Q7 (the input terminal of the inverter). One end of capacitor C2 is connected to the other end of capacitor C1 and the other end is connected to the connection point between transistors Q7 and Q8.

[0174] One end of the reset switch SWr is connected to the junction between capacitors C1 and C2, and the other end is connected to the junction between transistors Q7 and Q8 and to capacitor C2, and is also connected in parallel to capacitor C2. The reset switch SWr is a switch that is turned on / off according to the reference level reset signal RST-I.

[0175] The inverter, including transistors Q7 and Q8, inverts the light-receiving signal input via capacitor C1 and outputs the inverted light-receiving signal to quantizer 26.

[0176] Here, in subtractor 25, the potential generated on the buffer 23 side of capacitor C1 at a certain point in time is set to potential Vinit. Then, it is assumed that reset switch SWr is on at this time. With reset switch SWr on, the side of capacitor C1 opposite to buffer 23 is a virtual ground terminal. For convenience, the potential of this virtual ground terminal is set to zero. At this time, when the capacitance of capacitor C1 is Cp1, the charge CHinit accumulated in capacitor C1 is represented as follows.

[0177] CHinit = Cp1 × Vinit... [Expression 1]

[0178] In addition, when the reset switch SWr is turned on, the two ends of capacitor C2 are short-circuited, causing the accumulated charge to become zero.

[0179] Next, assume the reset switch SWr is open. If the amount of light received changes, the potential on the buffer 23 side of capacitor C1 changes from the aforementioned Vinit. When the potential after the change is Vafter, the charge CH accumulated in capacitor C1 is represented as follows.

[0180] CHafter = Cp1 × Vafter... [Expression 2]

[0181] On the other hand, when the capacitance of capacitor C2 is Cp2 and the output voltage of subtractor 25 is Vout, the charge CH2 accumulated in capacitor C2 is represented as follows.

[0182] CH2 = -Cp2 × Vout... [Expression 3]

[0183] At this point, since the total charge of capacitors C1 and C2 remains unchanged, the following statement holds true.

[0184] CHinit = CHafter + CH2... [Expression 4]

[0185] When [expression 1] through [expression 3] are substituted into [expression 4] for conversion, the following is obtained.

[0186] Vout = -(Cp1 / Cp2) × (Vafter - Vinit)...[Expression 5]

[0187] This represents a subtraction operation on a voltage signal, and the gain of the subtraction result is Cp1 / Cp2.

[0188] According to [Expression 5], the output of subtractor 25 is a signal representing the difference between the past level (Vinit) of the optical received signal and the current level (Vafter) of the optical received signal.

[0189] Here, the potential Vinit corresponds to the aforementioned reference level Lref. According to the above description, when the reset switch SWr is turned on, this potential Vinit, i.e., the reference level Lref, is reset to the current level of the optical receiving signal; in other words, the level of the optical receiving signal when the reset switch SWr is turned on.

[0190] Quantizer 26 includes transistors Q9, Q10, Q11 and Q12, and is configured as a 1.5-bit quantizer.

[0191] Transistors Q9 and Q11 are P-type transistors, while transistors Q10 and Q12 are N-type transistors.

[0192] As shown, transistors Q9 and Q10, and transistors Q11 and Q12 are connected in series between the power supply terminal and the ground terminal. The output voltage (Vout) of subtractor 25 is input to the gate of each of transistors Q9 and Q11. In addition, voltage Vhigh is applied to the gate of transistor Q10, and voltage Vlow is applied to the gate of transistor Q12.

[0193] The first polarity event signal Vop, representing the result of the first polarity event detection, is obtained at the connection point of transistors Q9 and Q10, and the second polarity event signal Vom, representing the result of the second polarity event detection, is obtained at the connection point of transistors Q11 and Q12.

[0194] Specifically, on the transistors Q9 and Q10 sides, when the output voltage (Vaafter-Vinit) of the subtractor 25 is equal to or higher than the positive threshold corresponding to the voltage Vhigh, a first polarity event signal Vop at level H is obtained at the connection point between transistors Q9 and Q10. Furthermore, when the output voltage of the subtractor 25 is lower than the positive threshold, a first polarity event signal Vop at level L is obtained. That is, at the connection point between transistors Q9 and Q10, a first polarity event signal Vop is obtained, indicating whether the light received amount has changed by a predetermined threshold in the rising direction, i.e., indicating the detection result of the first polarity event.

[0195] Furthermore, on the transistors Q11 and Q12 sides, when the output voltage level of the subtractor 25 based on voltage Vlow is equal to or lower than the negative threshold, a second polarity event signal Vom at level H is obtained at the connection point between transistors Q11 and Q12. Furthermore, when the output voltage level of the subtractor 25 is greater than the negative threshold, a second polarity event signal Vom at level L is obtained. As described above, at the connection point between transistors Q11 and Q12, a second polarity event signal Vom is obtained, indicating whether the light reception amount has changed by more than a predetermined threshold in the decreasing direction; that is, indicating the detection result of the second polarity event.

[0196] [1-3. Reading method as a first embodiment]

[0197] For all pixel units 20, the sensor device 1 of the first embodiment reads the event signal row by row and reads the grayscale signal row by row in sequence.

[0198] Here, "all pixel units" refers to all pixel units formed within the effective pixel area of ​​pixel array unit 2, excluding pixel units formed outside the effective pixel area (such as dummy pixel formation areas). It should be noted that this also applies to "pixel rows" and "pixel columns."

[0199] Figure 8 This is a timing diagram used to illustrate the reading method as a first embodiment.

[0200] It is important to note that, including Figure 8 In each timing diagram described below, the grayscale squares with the symbol "Read" schematically represent the timing of reading the grayscale signal. Furthermore, the vertical strip squares with the symbol "Shutter" schematically represent the timing of the electronic shutter operation (charge reset operation) based on the aforementioned charge reset signal RST-T. Additionally, the thick horizontal lines with the symbol "Accumulation" schematically represent the charge accumulation period used to obtain the grayscale signal.

[0201] In addition, the black square with the symbol "IVN" schematically represents the execution time from the generation of the event signal (in this example, the first polarity event signal Vop and the second polarity event signal Vom) to its reading.

[0202] First, as a prerequisite, this example uses the vertical sync signal XVS as a trigger to perform row-by-row reading of the grayscale signal. As part of the operation to generate the grayscale signal for each pixel row, the reset transistor Qr and the grayscale transfer transistor Qtt are first turned on after a pre-time interval from the start of reading the grayscale signal to perform the electronic shutter operation. Subsequently, the reset transistor Qr and the grayscale transfer transistor Qtt are turned off to begin charge accumulation in the photodiode PD.

[0203] At this time, since the charge accumulation time used to generate the grayscale signal is the same in each row, the electronic shutter operation is also performed in row sequence, similar to the reading of the grayscale signal.

[0204] Note that, for confirmation, the grayscale signal reading operation in each pixel row is performed by turning on the line control circuit 3, which controls the grayscale selection signal SLC-T. Furthermore, the electronic shutter operation in each pixel row is implemented by turning on the charge reset signal RST-T and the grayscale transfer drive signal TG-T through the line control circuit 3, and charge accumulation begins by turning off the charge reset signal RST-T and the grayscale transfer drive signal TG-T.

[0205] As shown, the event signal generation and reading operations in this case are performed before the electronic shutter operation begins. In other words, the line control circuit 3 selects the line to read the event signal before the charge accumulation associated with the generation of the grayscale signal begins.

[0206] It should be noted that in this article, the temporal sequence between the event signal generation and readout operations and the electronic shutter operation refers to the sequence within the period divided by the vertical synchronization signal XVS (i.e., the period of one frame), and does not include the sequence of electronic shutter operations in other frame periods.

[0207] Here, by turning on the event transfer drive signal TG-I in the state where the grayscale transfer transistor Qtt is turned off by the grayscale transfer drive signal TG-T, the generation operation of the event signal in each pixel row is started.

[0208] Then, the row control circuit 3 turns on the event selection signal SLC-I to turn on the first event selection transistor Qsip and the second event selection transistor Qsim, thereby performing the event signal reading operation.

[0209] In this example, the line control circuit 3 causes the aforementioned reset switch SWr (see...) Figure 7 The reference level Lref is reset at the moment the event signal has been read.

[0210] Therefore, the optical received signal level when the event signal of the previous frame is read is used as the reference level Lref to generate the event signal of the next frame.

[0211] For reference Figure 8 As can be seen, in this embodiment, the reading of event signals and the reading of grayscale signals are performed sequentially row by row at different times. In other words, the row control circuit 3 sequentially performs row selection for the pixel row to be read event signals and selection for the pixel row to be read grayscale signals at different times.

[0212] Therefore, both the event signal and the grayscale signal can be read from each pixel unit 20 (each pixel 21) within a frame period divided by the vertical synchronization signal XVS. That is, simultaneous reading of the event signal and the grayscale signal in frame periods is possible.

[0213] Note that simultaneous reading in frame periods does not necessarily mean reading both the event signal and the grayscale signal within a period divided by the vertical sync signal XVS. Rather, it means that for each pixel unit 20 from which the grayscale signal will be read, both the event signal and the grayscale signal will be read within one frame period. It should be noted that when using pixel units 20A, one frame period can be a period that allows for a one-line delay.

[0214] Here, in the above description, an example has been described of performing the generation and reading of event signals before the start of charge accumulation associated with the generation of grayscale signals. However, as... Figure 9 As shown, during the execution cycle of charge accumulation associated with the generation of grayscale signals, the generation and reading of event signals can also be performed. Specifically, in Figure 9 In the example, the generation and reading of the event signal are performed immediately before the end of the execution cycle related to charge accumulation for grayscale signal generation. In other words, they are executed immediately before the grayscale signal is read.

[0215] In this case, in sensor device 1, using Figure 4 The pixel array unit 2A shown replaces pixel array unit 2. That is, for each pixel unit 20A, during the period of charge accumulation related to the generation of grayscale signal in pixel 21-T, the generation and reading of event signal are performed in pixel 21-I.

[0216] It is important to note that, according to Figure 4 The pixel unit 20A shown is configured such that rows containing pixels 21-I and rows where pixels 21-I are not present are alternately formed in the pixel array unit 2A. For ease of explanation, Figure 9 The timing of grayscale signal readout, electronic shutter operation, and event signal generation and readout for each row of the row where pixel 21-I exists is shown.

[0217] In this case, the line control circuit 3 performs on / off control on the grayscale transfer drive signal TG-T, the charge reset signal RST-T, and the grayscale selection signal SLC-T, thereby similarly performing grayscale signal reading and electronic shutter operation on lines where pixel 21-I is absent. At this time, for all lines, grayscale signal reading and electronic shutter operation are performed sequentially line by line.

[0218] In addition, such as Figure 10 As shown, multiple event signals can be generated and read in each frame.

[0219] exist Figure 10 In the example, the generation and reading of the event signal are performed a total of two times: before the start of the electronic shutter operation and during the execution cycle of charge accumulation associated with the generation of the grayscale signal in each frame period.

[0220] It should be noted that when the generation and reading of event signals are performed during the execution cycle related to charge accumulation associated with the generation of grayscale signals, pixel array unit 2A is used in a manner similar to... Figure 9 The purpose of this situation.

[0221] It should be noted that, Figures 8 to 10 In the example, it is assumed that the electronic shutter operation is performed in row order, but as Figure 11 As shown, electronic shutter operation can be performed simultaneously in all rows, as in a so-called global shutter system.

[0222] In this case, for pixel 21 (or pixel 21-T) arranged in each row, as Figure 12 As shown in the equivalent circuit diagram, two grayscale transfer transistors Qtt1 and Qtt2 are set as grayscale transfer transistor Qtt, and a capacitor Cm (memory) is set between grayscale transfer transistors Qtt1 and Qtt2 to temporarily hold the accumulated charge of photodiode PD.

[0223] In this configuration, the horizontal control circuit 3 controls the on / off state of the grayscale transfer drive signals TG-I1 and TG-12 and the charge reset signal RST-T shown in the attached diagram, thereby performing electronic shutter operation. Specifically, the horizontal control circuit 3 first turns on all grayscale transfer drive signals TG-I1 and TG-12 and the charge reset signal RST-T to turn on all grayscale transfer transistors Qtt1 and Qtt2 and the reset transistor Qr, thereby resetting the charge in the photodiode PD and the capacitor Cm. Afterward, all grayscale transfer drive signals TG-I1 and TG-12 and the charge reset signal RST-T are turned off, and the grayscale transfer transistors Qtt1 and Qtt2 and the reset transistor Qr are turned off, thereby initiating charge accumulation in the photodiode PD.

[0224] Then, the line control circuit 3 transfers the accumulated charge in the photodiode PD to the capacitor Cm, and keeps the accumulated charge in the capacitor Cm by turning on and then off the grayscale transfer drive signal TG-I1 according to the elapsed time of the predetermined charge accumulation. In addition, by turning on the grayscale transfer drive signal TG-12 and the grayscale selection signal SLC-T according to the arrival of the subsequent reading time of the grayscale signal, the charge held in the capacitor Cm is output to the grayscale vertical signal line Lt via the floating diffusion FD.

[0225] <2. Second Implementation Method>

[0226] Next, the second embodiment will be described.

[0227] In the second embodiment, row selection for reading grayscale signals is performed based on the determination result of the presence or absence of an event according to the event signal.

[0228] In addition, in the following description, the same reference numerals are used for parts that are the same as those already described, and their descriptions are omitted.

[0229] Figure 13 This is a block diagram showing an example of the internal configuration of the sensor device 1B as a second embodiment.

[0230] and Figure 1 The difference in the sensor device 1 shown is that a line control circuit 3B, an event processing / output circuit 4B, and a grayscale output circuit 5B are provided, replacing the line control circuit 3, the event processing / output circuit 4, and the grayscale output circuit 5, respectively.

[0231] The event processing / output circuit 4B determines whether an event has occurred line by line based on the event signals read sequentially from the pixel array unit 2, and outputs the event presence / absence determination signals, which indicate the determination results, to the row control circuit 3B in sequence.

[0232] Based on the presence / absence of the event determination signal, the row control circuit 3B only performs row-by-row grayscale signal reading for the rows in which the event has occurred.

[0233] In addition, the line control circuit 3B controls the grayscale output circuit 5B so that no processing related to grayscale signal output is performed for lines where no event has occurred.

[0234] The grayscale output circuit 5B is configured to switch whether to perform processing related to grayscale signal output according to instructions from the line control circuit 3B. Therefore, the line control circuit 3B instructs the grayscale output circuit 5B to not perform processing related to the grayscale signal output of the line based on the output time of the grayscale signal of the line where no event has occurred.

[0235] As can be seen from the above description, the processing related to grayscale signal output includes A / D conversion processing, etc.

[0236] Figure 14 This is a timing diagram used to explain the reading method as a second embodiment.

[0237] Here, in the following description from Figure 14 In each initial timing diagram, square markers (with the symbol "IVN") representing the generation and reading times of event signals are written in black in rows where an event has occurred, and in white in rows where no event has occurred, thus distinguishing the presence or absence of an event on a row-by-row basis.

[0238] exist Figure 14 In the example, similar to Figure 9 For example, the generation and reading of event signals are performed before the electronic shutter operation begins.

[0239] In addition, as shown, grayscale signals are not read from rows where no event has occurred.

[0240] In addition, Figure 14 In the example, for rows where no event has occurred, neither grayscale signal reading nor electronic shutter operation is performed. That is, for rows where no event has occurred based on the aforementioned event presence / absence determination signal, even if the start time of the electronic shutter operation for that row arrives, the row control circuit 3B will keep the aforementioned grayscale transfer drive signal TG-T and charge reset signal RST-T in the off state, thereby preventing the electronic shutter operation from being performed.

[0241] Figures 15 to 17 This is a flowchart illustrating an example of a processing program for implementing the reading method as described in the second embodiment above.

[0242] Figure 15The processing flow of the event processing / output circuit 4B is illustrated. It is important to note that the event processing / output circuit 4B repeats this process whenever an event signal is read line by line. Figure 15 The processing shown.

[0243] exist Figure 15 In step S101, the event processing / output circuit 4B determines whether there is a pixel in a row where an event has occurred. In this example, the presence or absence of an event is determined based on whether a first polarity event signal Vop or a second polarity event signal Vom is obtained by indicating that the event has occurred.

[0244] If a pixel in the row has been the subject of an event, the event processing / output circuit 4B proceeds to step S102, outputting a confirmation signal indicating the presence of an event as an event presence / absence confirmation signal to the row control circuit 3B on a row-by-row basis, and then terminates. Figure 15 The series of processes shown in the figure.

[0245] On the other hand, if no pixel in the row has been found to have experienced the event, the event processing / output circuit 4B proceeds to step S103, outputting a determination signal indicating the absence of the event as an event presence / absence determination signal to the row control circuit 3B on a row-by-row basis, and then terminates. Figure 15 The series of processes shown in the figure.

[0246] Figure 16 and Figure 17 This is a flowchart showing the processing procedure of the outgoing control circuit 3B. Figure 16 This illustrates the processing on the event reading side, and Figure 17 This illustrates the processing on the grayscale reading side.

[0247] exist Figure 16 In step S201, the row control circuit 3B resets the row number Ni to 0. The row number Ni is the number information used to identify the row to be processed for the event signal.

[0248] In step S202, following step S201, the row control circuit 3B begins row sequence selection on the event reading side. That is, it begins processing for row selection for event signal reading by sequentially executing the row selection process in the rows via the event selection signal SLC-I.

[0249] In step S203, following step S202, the row control circuit 3B determines whether the determination signal of the Ni-th row is a determination signal indicating the existence of an event. That is, for the row order selection that started event reading in step S202, it determines whether there is a determination signal for the events obtained sequentially row by row from the event processing / output circuit 4B, and whether the determination signal of the Ni-th row is a determination signal indicating the existence of an event.

[0250] If the determination signal of row Ni is a determination signal indicating the existence of an event, then row control circuit 3B proceeds to step S204, opens the event flag of row Ni, and proceeds to step S206.

[0251] On the other hand, if the determination signal of the Nith row is not a determination signal indicating the existence of an event, the row control circuit 3B proceeds to step S205, turns off the event flag of the Nith row, and proceeds to step S206.

[0252] In step S206, the row control circuit 3B activates the reset signal for the Ni row. That is, the aforementioned reference level reset signal RST-I is turned on to reset the reference level Lref.

[0253] In step S207, following step S206, the row control circuit 3B determines whether the row number Ni is equal to or greater than the maximum value Nimax. The maximum value Nimax is the maximum value of the row number Ni and is set to the same value as the row number Ni of the last row in the pixel array unit 2.

[0254] If the row number Ni is not equal to or greater than the maximum value Nimax, the row control circuit 3B proceeds to step S208, increments the row number Ni by 1, and returns to step S203. Therefore, for each pixel row, the event flag is set based on the event presence / absence determination signal (S204 or S205) and the reference level Lref is reset (S206).

[0255] In step S207, if the number of rows Ni is equal to or greater than the maximum value Nimax, then the row control circuit 3B terminates. Figure 16 The series of processes shown in the figure.

[0256] will describe Figure 17 Processing of grayscale reading side in the middle.

[0257] exist Figure 17 In step S210, the row control circuit 3B resets the row number Nt to 0. The row number Nt is the number of rows used to identify the grayscale signal to be processed.

[0258] In step S211 following step S210, the line control circuit 3B determines whether the event flag for the Nt-th line is turned on. If the event flag for the Nt-th line is turned on, the line control circuit 3B proceeds to step S212, turns on the selection signal (grayscale selection signal SLC-T) for the Nt-th line, instructs the grayscale output circuit 5B to output the grayscale of the Nt-th line in step S213, and proceeds to step S215.

[0259] Here, for example, an instruction to output grayscale is executed by turning on an enable signal to the grayscale output circuit 5B. In response to the instruction to output grayscale, the grayscale output circuit 5B performs processing related to grayscale signal output for the grayscale reading line performed in step S212.

[0260] On the other hand, if it is determined in step S211 that the event flag of the Ntth row is not turned on, the row control circuit 3B enters step S214, instructs the grayscale output circuit 5B not to output the grayscale of the Ntth row, and proceeds to step S215.

[0261] Here, for example, by disconnecting the enable signal to the grayscale output circuit 5B, an instruction to not output grayscale is executed. In response to the instruction to not output grayscale, the grayscale output circuit 5B does not perform any processing related to grayscale signal output.

[0262] In step S215, the row control circuit 3B increments the row number Nt by 1, and in the subsequent step S216, it determines whether the row number is greater than the maximum value Ntmax. The maximum value Ntmax is the maximum value of the row number Nt, and is set to the same value as the row number Nt of the last row in the pixel array unit 2.

[0263] If the number of rows Nt is not greater than the maximum value Ntmax, the row control circuit 3B enters step S217 and waits until the grayscale reading time of the Ntth row arrives. If the grayscale reading time of the Ntth row arrives, it returns to step S211.

[0264] Therefore, for each pixel row, the execution / non-execution of grayscale signal reading and the execution / non-execution of processing related to grayscale output in grayscale output circuit 5B are controlled according to the value of the event flag.

[0265] It should be noted that Figure 14 This illustrates an example where electronic shutter operation is not performed on lines where no event is detected and no grayscale signal is read.

[0266] However, as Figure 18 As shown in the timing diagram, electronic shutter operations (charge reset of photodiode PD and floating diffusion FD) can be performed on all rows, regardless of whether grayscale signals are read.

[0267] In this case, even for lines that are determined to have no event by the event presence / absence determination signal, the line control circuit 3B turns on and then turns off the grayscale transfer drive signal TG-T and the charge reset signal RST-T, thereby performing electronic shutter operation.

[0268] By performing electronic shutter operation on all lines as described above, regardless of whether the grayscale signal is read, the charge of the photodiode PD can be periodically reset during the frame period.

[0269] Therefore, it can prevent the charge of the photodiode PD from not being reset, which would have an adverse effect on the generation of grayscale signals and event signals in another pixel.

[0270] In addition, Figure 14 An example of generating and reading an event signal before the electronic shutter operation begins has already been described. Similarly, in the second embodiment, as... Figure 9 As shown, event signals can be generated and read during the execution cycle of charge accumulation.

[0271] Figure 19 The timing diagram for this situation is shown.

[0272] Here, when using the pixel array unit 2A in the second embodiment, the determination of whether an event has occurred (regarding whether a grayscale signal has been read) is performed on a unit basis of pixel unit 20A. That is, since pixel unit 20A includes two rows in this example, the determination is performed on a unit basis of two rows. Specifically, if there is a pixel 21-I that has an event based on the event signal read on a row-by-row basis, in this case, the event processing / output circuit 4B outputs an event presence / absence determination signal indicating the presence of an event to the row control circuit 3B for the pixel row including pixel 21-I and the subsequent pixel row.

[0273] Here, the charge accumulation time associated with the generation of the grayscale signal can be changed according to the polarity of the event signal.

[0274] Figure 20 The timing diagram shows the case where the charge accumulation time changes according to the polarity of the event signal as described above.

[0275] It should be noted that in the following description Figure 20 and Figure 21 In the middle, the square marker (“IVN”) indicating the generation and reading time of the event signal is written with “P” for the row where the first polarity event has been detected and “M” for the row where the second polarity event has been detected.

[0276] As shown, for rows where a first polarity event has been detected, the charge accumulation time is shortened, and for rows where a second polarity event has been detected, the charge accumulation time is extended.

[0277] Since the first polarity event occurs on the side of increased light reception, a short charge accumulation time for the grayscale signal is sufficient. Conversely, since the second polarity event occurs on the side of decreased light reception, it is desirable to increase the charge accumulation time for the grayscale signal.

[0278] By changing the charge accumulation time according to the polarity of the event signal as described above, the charge accumulation time can be adjusted to an appropriate charge accumulation time based on the polarity of the event.

[0279] Here, in the same row, there is a situation where pixel 21 that detected the first polarity event is mixed with pixel 21 that detected the second polarity event. In that case, it is conceivable that the event with a larger number of detected pixels in the first polarity event and the second polarity event has already occurred as the target row.

[0280] It should be noted that Figure 20 An example is shown where an event signal is generated and read before the start of electronic shutter operation, assuming the charge accumulation time changes according to the polarity of the event. However, as... Figure 21 As shown, even when the charge accumulation time changes according to the polarity of the event, event signals can be generated and read during the execution period of charge accumulation.

[0281] In this case, the generation and reading times of the event signals are set according to the shorter of the charge accumulation time corresponding to the first polarity event and the charge accumulation time corresponding to the second polarity event. Specifically, in this case, the generation start time of the event signal for each row is set before the charge accumulation start time in that row has elapsed from the charge accumulation start time corresponding to the first polarity event.

[0282] Therefore, for both rows where the first polarity event has occurred and rows where the second polarity event has occurred, the event signals can be read before the time to read the grayscale signal. That is, it can be determined whether to read the grayscale signal based on the event signals before the time to read the grayscale signal arrives.

[0283] It should be noted that the method of changing the charge accumulation time according to the polarity of the event can also be applied to situations such as performing full-line reading of grayscale signals as in the first embodiment.

[0284] The above description has already illustrated an example of setting the frame period to a constant period by allocating the processing time of one row to rows that do not read grayscale signals.

[0285] However, as Figure 22 and Figure 23 As shown in the timing diagram, for rows where it is determined that grayscale signals will not be read, the row selection for reading grayscale signals is skipped, and the frame period can be made variable according to the number of rows for reading grayscale signals.

[0286] Figure 22 This illustrates an example of skipping the electronic shutter operation even for lines where the grayscale signal is determined not to be read, and Figure 23 This illustrates an example of performing electronic shutter operation even for lines where the grayscale signal is determined not to be read. Figure 23In the example, if it is determined that the lines for which grayscale signals are not read are consecutive, an electronic shutter operation is performed simultaneously for those lines.

[0287] In row-order selection, by skipping row selection, the interval between the selection time of the row immediately before skipping and the selection time of the row immediately after skipping can be shortened to a single row interval.

[0288] Therefore, the time required until the grayscale signal of the necessary row is fully read can be shortened.

[0289] <3. Third Implementation Method>

[0290] In the third embodiment, only the column where the event occurred outputs a grayscale signal.

[0291] Figure 24 This is a block diagram showing an example of the internal configuration of the sensor device 1C as a third embodiment.

[0292] With Figure 13 The difference in the sensor device 1B of the second embodiment shown is that the event processing / output circuit 4C and the grayscale output circuit 5C are respectively replaced by the event processing / output circuit 4B and the grayscale output circuit 5B.

[0293] In addition to the functions of the event processing / output circuit 4B, the event processing / output circuit 4C has the following function: for each row, it outputs to the grayscale output circuit 5C the determination result information about whether an event has occurred for each pixel 21 in that row (i.e., for each column position).

[0294] In addition to the functions of the grayscale output circuit 5B, the grayscale output circuit 5C also has the function of selectively outputting the grayscale signal of the pixel 21 for each row that has determined that an event has occurred, based on the determination result information from the event processing / output circuit 4C.

[0295] Figure 25 and Figure 26 This is a flowchart illustrating an example of the processing procedure for implementing the reading method as a third embodiment. Figure 25 The process of the event processing / output circuit 4C is shown, and Figure 26 The processing procedure of the grayscale output circuit 5C is shown.

[0296] Figure 25 The processing is repeatedly performed line-by-line by the event processing / output circuit 4C at each read moment of the event signal, and Figure 26 The processing is repeatedly performed by the grayscale output circuit 5C at each reading moment of the grayscale signal, in rows.

[0297] It should be noted that the event processing / output circuit 4C and Figure 25The processing shown is performed separately, outputting an event presence / absence determination signal to... Figure 15 The processing of the row control circuit 3B shown.

[0298] exist Figure 25 In step S110, the event processing / output circuit 4C determines whether an event has occurred for each pixel 21 in the row. In the subsequent step S111, the determination result for each pixel 21 is output to the grayscale output circuit 5C, and the process terminates. Figure 25 A series of processes within.

[0299] exist Figure 26 In step S301, the grayscale output circuit 5C determines whether there is an output command from the row control circuit 3B for that row. Note that a row refers to the row that has just arrived at the time sequence for reading the grayscale signal. In step S301, the grayscale output circuit 5C determines whether there is an output command for the row as... Figure 17 The output command executed by the row control circuit 3B in step S213.

[0300] If there is no output command from the vertical control circuit 3B, the grayscale output circuit 5C will terminate. Figure 26 The series of processes shown. That is, in this case, no processing related to grayscale signal output is performed on this row.

[0301] On the other hand, when there is an output command from the row control circuit 3B, the grayscale output circuit 5C enters step S302, and the event processing / output circuit 4C processes the grayscale signals of the pixels whose events occurred in that row based on the determination result for each pixel 21, and then ends the process. Figure 26 The series of processes shown here. That is, performing processing to output only the grayscale signals of the pixels where the event occurred in the grayscale signal read from that row.

[0302] By selectively outputting the grayscale signal of the pixel 21 in which the event is determined to have occurred from the grayscale signal read in rows as described above, it is not necessary to perform circuit operations for outputting grayscale signals at column positions where the event is not recognized.

[0303] Therefore, the power consumption of the grayscale signal output stage can be reduced, and energy saving of the sensor device IC can be achieved.

[0304] <4. Fourth Embodiment>

[0305] In the fourth embodiment, each pixel unit outputs an event signal to the line control unit as a signal indicating whether a request to read a grayscale signal exists or not.

[0306] Figure 27This is a block diagram illustrating an example of the internal configuration of the sensor device ID as a fourth embodiment.

[0307] Unlike the sensor device 1 of the first embodiment, a row control circuit 3D is provided instead of a row control circuit 3, and a request signal line Lrq for wiring in the row direction is provided in each row of the pixel array unit 2.

[0308] like Figure 28 As shown, in each pixel 21, there are two request signal lines Lrq configured in each row. One line is connected to the output line of the first polarity event signal Vop of the event detection circuit 24, and the other line is connected to the output line of the second polarity event signal Vom of the event detection circuit 24.

[0309] Therefore, if either a first polarity event or a second polarity event is detected in a certain pixel 21, the occurrence of the event is notified to the row control circuit 3D via the request signal line Lrq of the row configured with pixel 21.

[0310] exist Figure 27 In the middle, the row control circuit 3D determines the row of grayscale signal to be read based on the event signals (first polarity event signal Vop, second polarity event signal Vom) input through the request signal line Lrq. Then, for the row determined to be read, row selection for reading grayscale signal is performed sequentially row by row.

[0311] Figure 29 This is a timing diagram used to illustrate the reading method as a fourth embodiment.

[0312] First, the row control circuit 3D performs the process of reading event signals into the event processing / output circuit 4. Specifically, in this case, the event signals are read sequentially row by row, and for rows where no event has occurred based on the event signals input via the request signal line Lrq, the row selection for reading the event signals is skipped.

[0313] In addition, the line control circuit 3D reads grayscale signals sequentially line by line and skips line selection for reading grayscale signals of lines that are determined to have no events based on the event signal input via the request signal line Lrq.

[0314] It should be noted that, Figure 29 In the example, for rows where no event is determined to have occurred, the electronic shutter operation is also skipped. However, similar to Figure 23 In some cases, electronic shutter operation can also be performed on rows where no event has occurred (i.e., rows where grayscale signals are determined not to be read).

[0315] Figure 30 This is a block diagram showing an example of the internal configuration of the sensor device 1E as a variant of the fourth embodiment.

[0316] and Figure 27 The difference in the sensor device ID shown is that the event processing / output circuit 4C' and the grayscale output circuit 5C are set to replace the event processing / output circuit 4 and the grayscale output circuit 5, respectively.

[0317] Event processing / output circuit 4C' and event processing / output circuit 4C (see Figure 24 The difference is that the function of outputting a signal to determine the presence / absence of an event is omitted.

[0318] The grayscale output circuit 5C selectively outputs the grayscale signal of the pixel 21 for which the event has occurred, based on the determination result information of the presence or absence of each pixel 21 output by the event processing / output circuit 4C' in rows.

[0319] Therefore, similar to the third embodiment, there is no need to perform circuit operations for outputting grayscale signals at column positions where no event has been detected, and the power consumption of the grayscale signal output stage can be reduced.

[0320] <5. Fifth Implementation Method>

[0321] In the fifth embodiment, the ability to perform grayscale reading is determined based on the number of times the event occurs.

[0322] It should be noted that in the fifth embodiment, since the configuration of the sensor device is similar to that of the sensor device 1B in the second embodiment, redundant descriptions are avoided.

[0323] In the fifth embodiment, the event processing / output circuit 4B counts the number of times an event occurs based on the event signal of the unit pixel region Rp1 including the pixel unit 20 in the predetermined row, and determines whether the grayscale signal of the unit pixel region Rp1 can be read based on the counted number of events.

[0324] Here, the unit pixel region Rp1 refers to, for example, the region comprising n (n≥1) rows of pixel units 20 (20A). It should be noted that, if a pixel unit in pixel unit 20A extends over two rows, the unit pixel region Rp1 is the region comprising 2n rows of pixels.

[0325] If the number of events that have occurred for a unit pixel region Rp1 (the number of pixels for which events have occurred) is equal to or greater than a predetermined threshold (hereinafter referred to as "threshold m"), then the event processing / output circuit 4B obtains a determination result of reading the grayscale signal for the unit pixel region Rp1.

[0326] In this case, the line control circuit 3B determines the unit pixel region Rp1 to read the grayscale signal based only on the determination result of whether the event processing / output circuit 4B reads for each unit pixel region Rp1.

[0327] Figure 31 This is a flowchart illustrating an example of a processing procedure executed by the event processing / output circuit 4B to implement the reading method as a fifth embodiment.

[0328] It should be noted that the event processing / output circuit 4B executes repeatedly at a predetermined cycle. Figure 31 The processing shown.

[0329] exist Figure 31 In step S120, the event processing / output circuit 4B waits until it acquires the event signals of n rows. That is, under the control of the row control circuit 3B, it waits for the event signals read line by line until it acquires the event signals of n rows.

[0330] Upon acquiring n rows of event signals, the event processing / output circuit 4B proceeds to step S121 and performs noise removal processing on the acquired signals. Noise removal processing removes events that are estimated to be incorrectly detected due to noise.

[0331] Typically, since an event occurs in a pixel region that has clustered to a certain extent, it can be estimated that a pixel that has experienced an event, where no other pixels nearby have also experienced an event, is a pixel whose event was mistakenly detected due to noise. In noise removal processing, pixels estimated to have been mistakenly detected due to noise are specified in this way, and these pixels are excluded from the list of pixels where the event occurred.

[0332] In response to the noise removal process performed in step S121, the event processing / output circuit 4B counts the number of pixels that have experienced an event in step S122, and determines in subsequent step S123 whether the count value (i.e., the number of pixels that have experienced an event) is equal to or greater than the threshold m.

[0333] If the count value is equal to or greater than the threshold m, the event processing / output circuit 4B proceeds to step S124 to instruct the row control circuit 3B to read the grayscale signal of n rows, and terminates at... Figure 31 The series of processes shown in the figure.

[0334] On the other hand, if the count value is not equal to or greater than the threshold m, then the event processing / output circuit 4B terminates. Figure 31 The series of processes shown. That is, in this case, no instruction to read the grayscale signal is issued to the line control circuit 3B, and the grayscale signal is not read for the unit pixel area Rp1.

[0335] It is important to note that, in Figure 31 The noise removal process in step S121 is not necessarily required in the process shown.

[0336] <6. Sixth Implementation Method>

[0337] In the sixth embodiment, whether to read the grayscale signal is determined based on whether an object is identified through object recognition processing based on event signals.

[0338] It should be noted that, also in the sixth embodiment, the configuration of the sensor device is similar to that of the sensor device 1B in the second embodiment, and therefore redundant descriptions are avoided.

[0339] In the sixth embodiment, the event processing / output circuit 4B performs object recognition processing based on the event signal of a unit pixel region Rp2 comprising pixel units 20 in a plurality of predetermined rows, and determines whether a grayscale signal can be read for the unit pixel region Rp2 based on whether the object identified by the object recognition processing exists or not.

[0340] Specifically, when an object is identified as a result of object recognition processing based on an event signal of a unit pixel region Rp2, the event processing / output circuit 4B obtains a determination result of reading the grayscale signal for the unit pixel region Rp2.

[0341] In this case, the line control circuit 3B determines the unit pixel region Rp2 to read the grayscale signal based only on the determination result of whether the event processing / output circuit 4B reads the unit pixel region Rp2.

[0342] Figure 32 This is a flowchart illustrating an example of a processing procedure executed by the event processing / output circuit 4B to implement the reading method as the sixth embodiment.

[0343] Event processing / output circuit 4B executes repeatedly at a predetermined cycle. Figure 32 The processing shown.

[0344] exist Figure 32 In step S130, the event processing / output circuit 4B waits until multiple predetermined row event signals are acquired. That is, for event signals read sequentially row by row under the control of the row control circuit 3B, it waits until multiple predetermined row event signals are acquired.

[0345] Upon receiving event signals from multiple predetermined rows, the event processing / output circuit 4B proceeds to step S131 and performs object recognition processing. It should be noted that, as object recognition processing, the process of identifying predetermined target objects (e.g., people, animals, and vehicles) is performed. Many methods are known as specific methods for this object recognition processing. Applying a known method is sufficient, and the method is not limited to a particular method.

[0346] In step S132, which follows step S131, the event processing / output circuit 4B determines whether an object has been identified.

[0347] Upon confirming the identification of an object, the event processing / output circuit 4B proceeds to step S133, instructing the line control circuit 3B to read the grayscale signal and terminate. Figure 32 The diagram illustrates a series of processes. Specifically, it shows the issuance of instructions to read grayscale signals for unit pixel regions Rp2 of multiple predetermined rows.

[0348] On the other hand, if an unidentified object is determined, the event processing / output circuit 4B terminates. Figure 32 The series of processes shown. That is, in this case, no instruction to read the grayscale signal is issued to the line control circuit 3B, and the grayscale signal is not read for the unit pixel area Rp2.

[0349] It should be noted that, Figure 32 In the processing, after obtaining multiple event signals for predetermined rows in step S130, the above-mentioned noise removal processing can be performed on the obtained signals.

[0350] Therefore, in the object recognition process performed in step S131, the object recognition accuracy can be improved through noise removal.

[0351] <7. Variations>

[0352] It should be noted that the specific examples described so far are merely examples, and this technique can be modified with various configurations.

[0353] For example, the specific circuit configuration of each unit in the above example is just an example, and other configurations may be used.

[0354] For example, regarding Figure 3 The logarithmic transformation unit 22 shown in the figure can be adopted as follows: Figure 33 In the logarithmic transformation unit 22' shown, transistors Q15 and Q16 are stacked on a set of transistors Q1 and Q3 to form a double-stacked configuration.

[0355] Furthermore, regarding buffer 23, a configuration can be adopted that uses N-type transistors Q4' and Q5' instead of P-type transistors Q4 and Q5, such as... Figure 34In the buffer 23' shown.

[0356] Furthermore, regarding the quantizer 26 in the event detection circuit 24 (see...) Figure 7 ),like Figure 35 The quantizer 26' shown can be configured as follows: by providing only one comparator and switching and inputting voltages Vhigh and Vlow to the comparator, the first polarity event signal Vop and the second polarity event signal Vom can be output in a time-division manner by a shared comparator.

[0357] Furthermore, regarding the configuration of the output stage for event signals in pixels, such as... Figure 36 As shown, a configuration can be adopted for memory 30 that provides temporary holding of event signals. Note that, although Figure 36 An example is shown where memory 30 is provided for the first polarity event signal Vop, but a similar memory 30 is provided for the second polarity event signal Vom.

[0358] Here, in the sensor devices (1, 1B, 1C, 1D, 1E) of the embodiments described so far, the circuit portions in the pixel array unit 2 including the photodiode PD, grayscale transfer transistor Qtt, event transfer transistor Qti, floating diffuser FD, reset transistor Qr, amplification transistor Qat, and grayscale selection transistor Qst, as well as the event detection-related circuit portions including the event detection circuit 24, can be configured as separate chips. These chips can then be stacked, for example, in the vertical direction (chip thickness direction). At this time, bonding between the chips can be performed, for example, by Cu-Cu connections (copper-copper connections), connections using micro-pumps, connections using through-silicon vias (TSVs), etc.

[0359] Figures 37 to 39 A specific example of the connection point D between chips is shown.

[0360] Figure 37 This illustrates an example where the connection point between the event transfer transistor Qti and the logarithmic conversion unit 22 is set as the chip-to-chip connection point D.

[0361] Figure 38 An example is shown where the connection point between transistors Q2 and Q3 in the logarithmic conversion unit 22 is set as the connection point D between chips.

[0362] Figure 39 Showing the use of Figure 33 In the case of the logarithmic transformation unit 22' shown, the connection point between the event transfer transistor Qti and the logarithmic transformation unit 22' is set as an example of the connection point D between chips.

[0363] Figure 40This illustrates an example where, in a similar case using logarithmic conversion unit 22', the connection point between transistors Q2 and Q16 in logarithmic conversion unit 22' is set as the connection point D between chips.

[0364] Here, in the above description, an example of the application of this technique to a situation where both the first polarity event and the second polarity event are detected as event signals has been described, but this technique can also be appropriately applied to a configuration that detects only the first polarity event (or only the second polarity event).

[0365] <8. Overview of Implementation Methods>

[0366] As described above, the sensor device (1, 1B, 1C, 1D, 1E) as an embodiment includes: a pixel array unit (2 or 2A), wherein a plurality of pixel units (20 or 20A) are arranged in two dimensions, each pixel unit (20 or 20A) having one or more pixels (21, or 21-T and 21-I) and capable of generating a grayscale signal indicating the intensity of light reception and an event signal indicating a change in the light reception; and a row control unit (row control circuits 3, 3B, 3C, 3D), which is capable of sequentially performing row selection of pixels to be read event signals and selection of pixels to be read grayscale signals at different times.

[0367] Based on the above configuration, both the event signal and grayscale signal of each target pixel unit can be read within a frame period.

[0368] Therefore, grayscale signals and event signals can be read simultaneously in units of frame periods.

[0369] Furthermore, in the sensor device implemented as an example, the pixel unit (20) includes a pixel (21), and grayscale signal generation and event signal generation are performed alternately based on the charge generated in the light receiving element (photodiode PD) included in the pixel.

[0370] Therefore, event signals and grayscale signals can be read in pixels.

[0371] Therefore, the resolution of each of the event signal and the grayscale signal can be increased.

[0372] Furthermore, in the sensor device (1) implemented as an example, the row control unit (row control circuit 3) performs row selection for reading event signals and row selection for reading grayscale signals for all pixel rows.

[0373] Therefore, it is possible to simultaneously read event signals and grayscale signals for all pixel units in frame periods.

[0374] Furthermore, in the sensor devices (1B, 1C, 1D, 1E) implemented as embodiments, the line control unit (line control circuits 3B, 3C, 3D) performs line selection for reading grayscale signals based on a determination result of whether an event has occurred, the determination result being based on the event signal.

[0375] Therefore, for pixel rows where no event has been identified, grayscale signal reading may not be performed.

[0376] Therefore, power consumption associated with grayscale signal reading can be reduced, and energy saving of sensor devices can be achieved.

[0377] Furthermore, in the sensor devices (1B, 1C) implemented as embodiments, the line control unit (line control circuit 3B, 3C) performs line selection for reading event signals of all pixel units, and performs line selection for reading grayscale signals based on the determination result of whether an event has occurred in the event signals read through the line selection.

[0378] Therefore, based on the determination of whether an event has occurred in all pixel units, grayscale signals can be read only for the pixel units where the event has been identified.

[0379] Furthermore, the sensor device (1C) as an embodiment includes: a grayscale output unit (grayscale output circuit 5C) that outputs grayscale signals read from pixel units, and the grayscale output unit selectively outputs grayscale signals of pixel units whose events have occurred from grayscale signals read in rows by the row selection of the row control unit.

[0380] Therefore, it is not necessary to perform circuitry operations for outputting grayscale signals at column positions where no event has been identified.

[0381] Therefore, the power consumption of the grayscale signal output stage can be reduced, and energy saving of the sensor device can be achieved.

[0382] Furthermore, the sensor device as an embodiment includes a first readout determination unit (event processing / output circuit 4B), which counts the number of times an event occurs based on an event signal of a unit pixel region (Rp1) including the pixel unit in a predetermined row, and determines whether the grayscale signal of the unit pixel region can be read based on the counted number of events occurring (see fifth embodiment).

[0383] Typically, events occur in a certain number of pixels. Therefore, when the number of times an event occurs is very small (e.g., once), it can be estimated that the event has been incorrectly detected due to noise. Thus, by determining whether a grayscale signal can be read based on the number of times an event occurs, as described above, it is possible to prevent the reading of grayscale signals in unit pixel regions where events have been incorrectly detected due to noise.

[0384] Therefore, unnecessary reading operations of grayscale signals can be prevented, and energy saving of the sensor device can be achieved.

[0385] Furthermore, the sensor device as an embodiment includes a second readout determination unit (event processing / output circuit 4B), which performs object recognition processing on a unit pixel region (Rp2) including pixel units in a plurality of predetermined rows based on an event signal, and determines whether a grayscale signal for the unit pixel region can be read based on whether object recognition is performed through the object recognition processing (see sixth embodiment).

[0386] Therefore, for a unit pixel region comprising multiple predetermined rows, the grayscale signal can be read if the object is identified, and not read if the object is not identified.

[0387] In a unit pixel area, if the detected event is merely a false detection event caused by noise, the probability of identifying an object in the object recognition process is extremely low. Therefore, according to the above configuration, it is possible to prevent the reading of grayscale signals for unit pixel areas where events are falsely detected due to noise. That is, unnecessary grayscale signal reading operations can be prevented, and energy saving of the sensor device can be achieved.

[0388] Furthermore, in the sensor device implemented as an example, the first readout determination unit performs noise removal processing on a unit pixel region. This noise removal processing removes events that are estimated to be erroneously detected due to noise, and determines whether a grayscale signal can be read from the unit pixel region based on the number of times the events occur after the noise removal processing.

[0389] Therefore, it is possible to prevent reading grayscale signals for unit pixel areas where events are erroneously detected due to noise.

[0390] Therefore, unnecessary reading operations of grayscale signals can be prevented, and energy saving of the sensor device can be achieved.

[0391] Furthermore, in the sensor device (1B) as an example, the line control unit (line control circuit 3B) skips the line selection for reading grayscale signals for lines determined not to read grayscale signals (see [link]). Figure 22 , Figure 23 wait).

[0392] In row-order selection, by skipping row selection, the interval between the selection time of the row immediately before skipping and the selection time of the row immediately after skipping can be shortened to a single row interval.

[0393] Therefore, the time required until the grayscale signal of the necessary row is fully read can be shortened.

[0394] Furthermore, in the sensor device (1B) implemented as an example, the line control unit (line control circuit 3B) resets the charge of all line reset light receiving elements, regardless of whether a grayscale signal is read (see...). Figures 18 to 21 wait).

[0395] Therefore, the charge of the optical receiving element can be periodically reset during the frame period.

[0396] Therefore, it can prevent the charge of the light receiving element from not being reset, which would have an adverse effect on the generation of grayscale and event signals in another pixel, and can improve the accuracy of grayscale and event signals.

[0397] Furthermore, in the sensor device implemented as an example, the pixel unit can generate a first polarity event signal indicating a change on the side of increasing light reception and a second polarity event signal indicating a change on the side of decreasing light reception as event signals.

[0398] Therefore, it is possible to identify whether the generated event is an event on the side of increasing or decreasing light reception.

[0399] Furthermore, in the sensor device (1B) implemented as an example, the line control unit (line control circuit 3B) determines the charge accumulation time related to the generation of the grayscale signal based on the change in the polarity of the event signal (see [link]). Figure 20 , Figure 21 wait).

[0400] Since the first polarity event occurs on the side of increased light reception, a short charge accumulation time for the grayscale signal is sufficient. Conversely, since the second polarity event occurs on the side of decreased light reception, it is desirable to increase the charge accumulation time for the grayscale signal.

[0401] By changing the charge accumulation time according to the polarity of the event signal as described above, the charge accumulation time can be adjusted to an appropriate charge accumulation time based on the polarity of the event.

[0402] Furthermore, in the sensor device implemented as an example, the line control unit performs line selection for reading event signals before charge accumulation related to the generation of grayscale signals begins (see...). Figure 8 , Figure 10 , Figure 11 , Figure 14 , Figure 18 , Figure 20 , Figure 22 , Figure 23 wait).

[0403] By performing row selection for reading event signals before charge accumulation begins, it is possible to determine whether a row's grayscale signal can be read based on the event signals prior to the start of charge accumulation.

[0404] Therefore, charge accumulation can be prevented when no event is detected and grayscale signal reading is not required. Consequently, circuitry operations related to charge accumulation can be omitted, and power savings can be achieved in the sensor device.

[0405] Furthermore, since only grayscale signals need to be generated after an event signal is generated within a frame period, it is preferable to adopt the configuration of pixel unit 20 (where each pixel 21 can alternately generate event signals and grayscale signals).

[0406] Furthermore, in the sensor device implemented as an example, during the execution cycle of charge accumulation related to the generation of grayscale signals, the line control unit performs line selection for reading event signals (see...). Figure 9 , Figure 10 , Figure 19 , Figure 21 wait).

[0407] Therefore, compared to reading the event signal before charge accumulation begins, the event signal can be read at a time closer to the start of reading the grayscale signal. For example, the event signal can be read immediately before the grayscale signal is read.

[0408] Therefore, the time lag between reading event signals and reading grayscale signals can be reduced, and the accuracy of the correspondence between event signals and grayscale signals on the time axis can be improved.

[0409] Furthermore, in the sensor device implemented as an example, the row control unit simultaneously begins charge accumulation related to the generation of grayscale signals for all pixel units (see [link]). Figure 11 wait).

[0410] Therefore, for the charge accumulation of grayscale signals, it is not necessary to perform independent timing control on a line-by-line basis.

[0411] Therefore, the configuration of control circuits related to charge accumulation can be simplified for the line control unit.

[0412] Furthermore, in the sensor devices (1D, 1E) implemented as embodiments, each pixel unit outputs the generated event signal to the line control unit (line control circuit 3D), and the line control unit determines the line from which the grayscale signal will be read based on the event signal input from each pixel unit.

[0413] Therefore, before reading the event signals of all pixel units in row order, it is possible to determine which row from which the grayscale signal should be read.

[0414] Furthermore, in the sensor devices (1D, 1E) implemented as embodiments, the line control unit performs line selection for reading grayscale signals sequentially for the lines determined to be read.

[0415] Therefore, even when events occur in multiple pixel units, the grayscale signal of each pixel unit is not read randomly at each event occurrence time, but is read sequentially row by row.

[0416] Therefore, a general grayscale output circuit corresponding to row-sequence reading can be used.

[0417] Furthermore, in the sensor device implemented as an example, the pixel unit (20A) includes a pixel that generates an event signal (21-I) and a pixel that generates a grayscale signal (21-T).

[0418] In this case, pixel units generate event signals and grayscale signals in different pixels.

[0419] Therefore, it is not necessary to alternately generate event signals and grayscale signals, and the degree of freedom in signal generation timing can be increased.

[0420] Furthermore, as an implementation method, the reading method is a reading method in a sensor device, which includes: a pixel array unit, wherein a plurality of pixel units are arranged in two dimensions, each pixel unit having a pixel and capable of generating a grayscale signal representing the intensity of light reception and an event signal representing a change in light reception; the reading method includes: sequentially performing the selection of rows of pixels for which event signals are to be read and pixels for which grayscale signals are to be read at different times.

[0421] According to this reading method as described in the embodiment, functions and effects similar to those of the sensor device described in the above embodiment can also be obtained.

[0422] It should be noted that the effects described in this specification are merely illustrative and not limited, and other effects may exist.

[0423] <9. This technology>

[0424] It should be noted that this technology may also have the following configurations. (1)

[0426] A sensor device, comprising:

[0427] A pixel array unit, in which multiple pixel units are arranged in two dimensions, each pixel unit having one or more pixels and capable of generating a grayscale signal indicating the intensity of light reception and an event signal indicating changes in light reception; and

[0428] The row control unit can sequentially perform row selection for the pixels to be read event signals and selection for the pixels to be read grayscale signals at different times. (2)

[0430] According to the sensor device in (1),

[0431] Here, a pixel unit includes one pixel; and

[0432] The generation of grayscale signals and event signals are performed alternately based on the charge generated in the light-receiving element included in a pixel. (3)

[0434] According to the sensor device in (1) or (2),

[0435] Among them, the line control unit:

[0436] Perform row selection for reading event signals and row selection for reading grayscale signals for all pixel rows. (4)

[0438] According to the sensor device in (1) or (2),

[0439] Among them, the line control unit:

[0440] Row selection is performed to read grayscale signals based on the determination of whether or not an event has occurred; this determination is based on the event signal. (5)

[0442] According to the sensor device in (4),

[0443] Among them, the line control unit:

[0444] Perform row selection for reading event signals for all pixel units; and

[0445] Based on the determination of whether an event has occurred, row selection is performed to read the grayscale signal, the determination being based on the event signal read through the row selection. (6)

[0447] The sensor device according to any one of (1) to (5) further includes:

[0448] Grayscale output unit, which outputs the grayscale signal read from the pixel unit;

[0449] Among them, the grayscale output unit:

[0450] Among the grayscale signals read row by row through row selection by the row control unit, the grayscale signals of the pixel units whose events have been determined to have occurred are selectively output. (7)

[0452] The sensor device according to any one of (1) to (6) further includes:

[0453] The first reading determination unit counts the number of times an event occurs based on an event signal of a unit pixel region including pixel units in a predetermined row, and determines whether a grayscale signal can be read for the unit pixel region based on the counted number of events. (8)

[0455] The sensor device according to any one of (1) to (6) further includes:

[0456] The second reading determination unit performs object recognition processing based on an event signal of a unit pixel region including pixel units in multiple predetermined rows, and determines whether a grayscale signal can be read for the unit pixel region based on whether the object is identified as existing or not through the object recognition processing. (9)

[0458] According to the sensor device in (7),

[0459] The first read determination unit is as follows:

[0460] Noise removal processing is performed on a unit pixel region. Noise removal processing removes events that are estimated to be erroneously detected due to noise. The number of occurrences of the events after noise removal processing determines whether the grayscale signal can be read on a unit pixel region. (10)

[0462] According to any one of (4) to (9),

[0463] Among them, the line control unit:

[0464] Skip the row selection used to read grayscale signals for rows that are determined not to read grayscale signals. (11)

[0466] According to any one of (4) to (10),

[0467] Among them, the line control unit:

[0468] Regardless of whether grayscale signals are read, the charge of all line-reset optical receiver elements is checked. (12)

[0470] According to any one of (1) to (11),

[0471] Wherein, pixel unit:

[0472] It can generate a first polarity event signal indicating a change on the side of increasing light reception and a second polarity event signal indicating a change on the side of decreasing light reception as event signals. (13)

[0474] According to the sensor device in (12),

[0475] Among them, the line control unit:

[0476] The charge accumulation time associated with the generation of grayscale signals is changed according to the polarity of the event signal. (14)

[0478] According to any one of (1) to (13),

[0479] Among them, the line control unit:

[0480] Before starting charge accumulation associated with the generation of grayscale signals, row selection for reading event signals is performed. (15)

[0482] According to any one of (1) or (3) to (13),

[0483] Among them, the line control unit:

[0484] During the execution cycle of charge accumulation associated with the generation of grayscale signals, row selection for reading event signals is performed. (16)

[0486] According to any one of (1) to (15),

[0487] Among them, the line control unit:

[0488] Charge accumulation related to the generation of grayscale signals begins simultaneously for all pixel units. (17)

[0490] According to the sensor device in (1),

[0491] Each pixel unit outputs the generated event signal to the row control unit; and

[0492] Line control unit:

[0493] Based on the event signal input from each pixel unit, determine the row of grayscale signal to be read. (18)

[0495] According to the sensor device in (14),

[0496] Among them, the line control unit:

[0497] The row selection process is executed sequentially to read grayscale signals line by line for the rows that are determined to be read. (19)

[0499] According to the sensor device of any one of (1) or (3) to (18),

[0500] Wherein, pixel unit:

[0501] This includes pixels that generate event signals and pixels that generate grayscale signals. (20)

[0503] A reading method in a sensor device, the sensor device including a pixel array unit, wherein multiple pixel units are arranged in two dimensions, each pixel unit having one or more pixels and capable of generating a grayscale signal indicating the intensity of light received and an event signal indicating a change in the amount of light received.

[0504] Reading methods include:

[0505] At different times, row selection for the pixels whose event signals are to be read and selection for the pixels whose grayscale signals are to be read are performed sequentially.

[0506] List of reference numerals

[0507] Sensor devices 1, 1B, 1C, 1D, 1E

[0508] 2. 2A pixel array unit

[0509] 3, 3B, 3C, 3D line control circuits

[0510] 4, 4B, 4C, 4C' Event processing / output circuits

[0511] 5, 5B, 5C Grayscale Output Circuit

[0512] Lc, Lc1, Lc2, Lc3, Lc4, Lc5, Lc6 Line control lines

[0513] Li event vertical signal line

[0514] Lip First Event Vertical Signal Line

[0515] Lim Second Event Vertical Signal Line

[0516] Lt grayscale vertical signal line

[0517] 20, 20A pixel units

[0518] 21, 21-I, 21-T pixels

[0519] PD photodiode

[0520] FD floating diffusion

[0521] QTT grayscale transfer transistor

[0522] Qti Event Transfer Transistor

[0523] Qr reset transistor

[0524] Qat Amplifying Transistor

[0525] Qst grayscale selection transistor

[0526] QSIP first event selects transistors

[0527] Qsim Second Event Select Transistor

[0528] TG-T grayscale transfer drive signal

[0529] RST-T charge reset signal

[0530] TG-I event transfer drive signal

[0531] RST-I reference level reset signal

[0532] SLC-T grayscale selection signal

[0533] SLC-I Event Selection Signal

[0534] L Reference Level

[0535] SWr Reset Switch

[0536] Vop First Polarity Event Signal

[0537] Vom Second Polarity Event Signal

[0538] Lrq request signal line.

Claims

1. A sensor device, comprising: A pixel array unit in which multiple pixel units are arranged in two dimensions, each pixel unit having one or more pixels and capable of generating a grayscale signal indicating the intensity of light reception and an event signal indicating a change in the amount of light reception; as well as The row control unit is capable of sequentially performing row selection for pixels whose event signals are to be read and selection for pixels whose grayscale signals are to be read at different times, wherein both the event signal and the grayscale signal are read from each pixel unit within a frame period.

2. The sensor device according to claim 1, in, The pixel unit comprises one pixel; and The generation of the grayscale signal and the generation of the event signal are performed alternately based on the charge generated in the light-receiving element included in the pixel.

3. The sensor device according to claim 1, in, The row control unit: Perform row selection for reading the event signal and row selection for reading the grayscale signal for all pixel rows.

4. The sensor device according to claim 1, in, The row control unit: Row selection is performed to read the grayscale signal based on the determination result of whether or not an event has occurred, the determination result being based on the event signal.

5. The sensor device according to claim 4, in, The row control unit: Perform row selection for reading the event signal for all of the aforementioned pixel units; and Based on the determination of whether an event has occurred, a row selection for reading the grayscale signal is performed, the determination being based on the event signal read through the row selection.

6. The sensor device according to claim 1, further comprising: A grayscale output unit outputs the grayscale signal read from the pixel unit; Wherein, the grayscale output unit: Among the grayscale signals read row by row through the row selection performed by the row control unit, the grayscale signals of the pixel units for which an event has been determined to have occurred are selectively output.

7. The sensor device according to claim 1, further comprising: The first reading determination unit counts the number of times the event occurs based on the event signal of the unit pixel region including the pixel unit in the predetermined row, and determines whether the grayscale signal can be read for the unit pixel region based on the counted number of times the event occurs.

8. The sensor device according to claim 1, further comprising: The second reading determination unit performs object recognition processing based on the event signal of a unit pixel region comprising the pixel units in a plurality of predetermined rows, and determines whether the grayscale signal can be read for the unit pixel region based on whether the object is identified as existing or not through the object recognition processing.

9. The sensor device according to claim 7, in, First read determination unit: A noise removal process is performed on the unit pixel region, which removes events that are estimated to be erroneously detected due to noise. Based on the number of occurrences of the events after the noise removal process, it is determined whether the grayscale signal can be read for the unit pixel region.

10. The sensor device according to claim 4, in, The row control unit: Skip the row selection used to read the grayscale signal for rows that are determined not to read the grayscale signal.

11. The sensor device according to claim 4, in, The row control unit: Regardless of whether the grayscale signal is read, the charge of all line-reset optical receiving elements is calculated.

12. The sensor device according to claim 1, in, The pixel unit: It is capable of generating a first polarity event signal indicating a change on the side of increasing light reception and a second polarity event signal indicating a change on the side of decreasing light reception as the event signal.

13. The sensor device according to claim 12, in, The row control unit: The charge accumulation time associated with the generation of the grayscale signal is changed according to the polarity of the event signal.

14. The sensor device according to claim 1, in, The row control unit: Before starting charge accumulation associated with the generation of the grayscale signal, row selection for reading the event signal is performed.

15. The sensor device according to claim 1, in, The row control unit: During the execution cycle of charge accumulation associated with the generation of the grayscale signal, row selection for reading the event signal is performed.

16. The sensor device according to claim 1, in, The row control unit: Simultaneously, charge accumulation related to the generation of the grayscale signal begins for all the aforementioned pixel units.

17. The sensor device according to claim 1, in, Each pixel unit outputs the generated event signal to the row control unit; and The row control unit: Based on the event signal input from each of the pixel units, the row of grayscale signal to be read is determined.

18. The sensor device according to claim 14, in, The row control unit: The row selection process is executed sequentially to read the grayscale signal row by row for the rows determined to be read.

19. The sensor device according to claim 1, in, The pixel unit: This includes the pixels that generate the event signal and the pixels that generate the grayscale signal.

20. A reading method in a sensor device, the sensor device comprising a pixel array unit, wherein a plurality of pixel units are arranged in two dimensions, each pixel unit having one or more pixels and capable of generating a grayscale signal indicating the intensity of light reception and an event signal indicating a change in the light reception. The reading method includes: At different times, row selection for the pixel whose event signal is to be read and selection for the pixel whose grayscale signal is to be read are performed sequentially, wherein both the event signal and the grayscale signal are read from each pixel unit within a frame period.

Images