Sensor device and method for operating the sensor device

Abstract

To provide a sensor device having improved imaging capabilities. [Solution] The sensor device includes a plurality of pixels, each configured to receive light and perform photoelectric conversion to generate an electrical signal; an event detection circuit configured to detect an intensity change exceeding a predetermined threshold in the light received by each of the pixels of a first subset as event data; a pixel signal generation circuit configured to generate a pixel signal indicating an intensity value of the light received for each of the pixels of a second subset; and a control unit configured to start detecting event data at a first time point prior to generation of a pixel signal at a second time point, and to reconstruct intensity values ​​for the pixels of the second subset during a period between the first time point and the second time point using the pixel signal generated after the second time point and the event data detected before the second time point.

Description

[Technical field]

[0001] The present technology relates to a sensor device and a method of operating the sensor device, and in particular to a sensor device and a method of operating the sensor device that uses an event-based vision sensor (EVS) to enable reconstruction of an image frame prior to the start time of a camera recording. [Background technology]

[0002] For example, a typical digital video camera used as a standalone device or integrated into other electronic devices such as a mobile phone typically includes a sensor device that starts capturing video in response to a command from a user, such as pressing a button. However, if a sudden or unusual event occurs, a person may want to start recording the event, but by the time they trigger the start of the video, they may miss the beginning of the scene.

[0003] It is therefore desirable to improve the imaging capabilities of the sensor device, such as by allowing reconstruction of image frames prior to the start time of a video recording. Summary of the Invention [Problem to be solved by the invention]

[0004] To provide a sensor device having improved imaging capabilities. [Means for solving the problem]

[0005] To this end, there is provided a sensor device comprising a plurality of pixels, each configured to receive light and perform photoelectric conversion to generate an electrical signal, an event detection circuit configured to detect an intensity change exceeding a predetermined threshold of the received light by each of the pixels of a first subset as event data, and a pixel signal generation circuit configured to generate a pixel signal indicative of an intensity value of the received light for each of the pixels of a second subset, the sensor device further comprising a control unit configured to start detecting event data at a first time point prior to generation of a pixel signal at a second time point, and to reconstruct intensity values ​​for the pixels of the second subset for a period between the first and second time points using the pixel signals generated after the second time point and the event data detected before the second time point.

[0006] The present invention also provides a method of operating a sensor device, which comprises receiving light and photoelectrically converting the light to generate electrical signals by a plurality of pixels of the sensor device, detecting intensity changes of the received light by each of the pixels of a first subset exceeding a predetermined threshold as event data, generating pixel signals indicative of intensity values ​​of the received light for each of the pixels of a second subset, starting detection of event data at a first time point prior to generation of pixel signals at a second time point, and reconstructing intensity values ​​for the pixels of the second subset during a period between the first and second times using the pixel signals generated after the second time point and the event data detected before the second time point. Thus, conventional video camera pixels (pixels of the second subset) that generate pixel signals indicative of the intensity of the received light are supplemented with EVS pixels (pixels of the first subset) that are sensitive to intensity changes exceeding a predetermined threshold. The EVS pixels are triggered to detect events before a full intensity signal is obtained from the camera pixels. Once the full intensity signal is recorded, it is used together with the detected event data to reconstruct intensity values ​​for the camera pixels at a time point prior to the start of the full intensity frame capture. In this way, the captured video can be extended back in time. In this respect, the use of EVS pixels is particularly useful, since the event detection circuitry consumes less energy than the pixel signal generation circuitry. Thus, longer periods of event detection do not unduly deplete the energy reserves of the sensor device. Furthermore, the high temporal resolution of the event data allows compensation for motion blur that may occur shortly after the full intensity frame capture begins. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram of a sensor device. [Diagram 2] FIG. 2 is a schematic block diagram of the sensor unit. [Diagram 3] FIG. 3 is a schematic block diagram of the pixel array unit. [Figure 4] FIG. 4 is a schematic circuit diagram of a pixel block. [Diagram 5] FIG. 5 is a schematic block diagram showing the event detection unit. [Figure 6] FIG. 6 is a schematic circuit diagram of the current-voltage conversion unit. [Figure 7] FIG. 7 is a schematic circuit diagram of a subtraction unit and a quantization unit. [Figure 8] FIG. 8 is a schematic timing chart showing an example of the operation of the sensor unit. [Figure 9] FIG. 9 is a schematic diagram showing a method for generating frame data based on event data. [Figure 10] FIG. 10 is a schematic block diagram showing another quantization unit. [Figure 11] FIG. 11 is a schematic diagram showing another event detection unit. [Figure 12] FIG. 12 is a schematic block diagram showing another pixel array unit. [Figure 13] FIG. 13 is a schematic circuit diagram showing another pixel block. [Figure 14] FIG. 14 is a schematic block diagram showing a scanning type imaging device. [Figure 15] FIG. 15 is a schematic block diagram showing the sensor device and its functions. [Figure 16] FIG. 16 is a schematic diagram illustrating frame reconstruction. [Figure 17] FIG. 17 is a schematic diagram showing frame reconstruction according to a comparative example. [Figure 18] FIG. 18 is a schematic diagram illustrating frame reconstruction. [Figure 19] FIG. 19 is a different schematic block diagram of the sensor device. [Figure 20] FIG. 20 is a schematic diagram showing a process flow of a method for operating the sensor device. [Figure 21] FIG. 21 is a schematic block diagram showing a smartphone. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The present disclosure aims to mitigate the problems that arise at the start of video capture. In particular, this problem is addressed by a method that ensures that the sensor device starts operating early enough to capture the desired scene without excessively depleting the energy storage of the sensor device. The method that solves this problem described below is applicable to any kind of sensor that can capture a scene by generating an event data stream along with a full intensity frame sequence. However, for ease of explanation and to cover important applications, the following description focuses, without prejudice, on a specific implementation of a dynamic vision sensor / event-based vision sensor (DVS / EVS).

[0009] So, first we will describe a possible implementation of DVS / EVS, which is, of course, merely an example, as it will be understood that DVS / EVS can be implemented in different ways.

[0010] Fig. 1 is a diagram showing a configuration example of a sensor device 10. In the example of Fig. 1, the sensor device 10 is configured by a sensor chip.

[0011] The sensor device 10 is a single-chip semiconductor chip and includes a sensor die (substrate) 11 that functions as multiple dies (substrates) and a logic die 12, and the sensor die 11 and the logic die 12 are stacked. Note that the sensor device 10 may include only one die or three or more stacked dies.

[0012] In the sensor device 10 shown in FIG. 1, the sensor die 11 includes a sensor unit 21 (a circuit functioning as a sensor unit), and the logic die 12 includes a logic unit 22. The sensor unit 21 may be partially formed on the logic die 12. The logic unit 22 may be partially formed on the sensor die 11. The sensor unit 21 includes pixels configured to photoelectrically convert incident light to generate an electric signal, and generates event data indicating the occurrence of an event, which is a change in the electric signal of the pixel. The sensor unit 21 supplies the event data to the logic unit 22. That is, the sensor unit 21 performs imaging in which incident light is photoelectrically converted in a pixel to generate an electric signal, similar to, for example, a synchronous image sensor. However, the sensor unit 21 generates image data (frame data) in a frame format, and also generates event data indicating the occurrence of an event, which is a change in the electric signal of the pixel. The sensor unit 21 outputs the event data obtained by imaging to the logic unit 22.

[0013] Here, the synchronous image sensor is an image sensor configured to capture an image in synchronization with a vertical synchronization signal and output frame data, which is image data in a frame format. In contrast to the synchronous image sensor, the event detection function of the sensor unit 21 can be considered to be of an asynchronous type (asynchronous image sensor). This is because, as far as event detection is concerned, the sensor unit 21 does not operate in synchronization with the vertical synchronization signal when outputting event data.

[0014] Like a synchronous image sensor, the sensor unit 21 generates and outputs frame data in addition to event data. The sensor unit 21 can also output, together with the event data, an electrical signal of a pixel in which an event has occurred as a pixel signal that is the pixel value of the pixel in the frame data.

[0015] The logic unit 22 controls the sensor unit 21 as necessary. In addition, the logic unit 22 performs various data processing such as data processing for generating frame data based on event data from the sensor unit 21 and image processing on the frame data from the sensor unit 21 or the frame data generated based on the event data from the sensor unit 21, and outputs the data processing results obtained by performing the various data processing on the event data and frame data.

[0016] FIG. 2 is a block diagram showing an example of the configuration of the sensor unit 21 shown in FIG.

[0017] The sensor unit 21 includes a pixel array unit 31, a drive unit 32, an arbiter 33, an AD (analog-to-digital) conversion unit , and an output unit .

[0018] The pixel array unit 31 preferably includes a plurality of pixels 51 (FIG. 3) arranged in a two-dimensional lattice. When a change greater than a predetermined threshold (including a change equal to or greater than the threshold, as necessary) occurs in a photocurrent (corresponding voltage), which is an electrical signal generated by photoelectric conversion in the pixel 51, the pixel array unit 31 detects the change in the photocurrent as an event. When an event is detected, the pixel array unit 31 outputs a request to the arbiter 33 to request output of event data indicating the occurrence of the event. Then, when a response indicating permission to output the event data is received from the arbiter 33, the pixel array unit 31 outputs the event data to the drive unit 32 and the output unit 35. The pixel array unit 31 may also output an electrical signal of the pixel 51 from which the event has been detected as a pixel signal to the AD conversion unit 34. The drive unit 32 drives the pixel array unit 31 by supplying a control signal to the pixel array unit 31. For example, the drive unit 32 drives the pixel 51 from which the pixel array unit 31 has output the event data, and the pixel 51 supplies (outputs) the pixel signal to the AD conversion unit 34. However, the driving section 32 may synchronously drive the pixels 51 to generate frame data.

[0019] The arbiter 33 arbitrates requests for outputting event data from the pixel array unit 31, and returns a response to the pixel array unit 31 indicating whether output of the event data is permitted or prohibited.

[0020] The AD conversion unit 34 includes, for example, a single-slope ADC (AD converter) (not shown) for each column of a pixel block 41 (FIG. 3) described later. The AD conversion unit 34 performs AD conversion on pixel signals of pixels 51 in the pixel block 41 of each column using the ADC of the column, and supplies the result to the output unit 35. The AD conversion unit 34 can perform CDS (correlated double sampling) in addition to the AD conversion of the pixel signals.

[0021] The output section 35 performs necessary processing on the pixel signals from the AD conversion section 34 and the event data from the pixel array section 31, and supplies the results to the logic section 22 (FIG. 1).

[0022] Here, since a change in the photocurrent generated in pixel 51 can be recognized as a change in the amount of light incident on pixel 51, the event can also be said to be a change in the amount of light in pixel 51 (a change in the amount of light greater than a threshold).

[0023] The event data indicating the occurrence of an event includes at least position information (coordinates, etc.) indicating the position of the pixel block where the change in light amount, which is the event, occurred. The event data may also include the polarity (positive or negative) of the change in light amount.

[0024] With regard to a series of event data output from the pixel array unit 31 at the timing of an event occurrence, as long as the event data interval is the same as the event occurrence interval, it can be said that the event data implicitly includes time information indicating the (relative) time when the event occurred. However, for example, when the event data is stored in a memory and the event data interval is no longer the same as the event occurrence interval, the time information implicitly included in the event data is lost. Therefore, the output unit 35 includes, in the event data, time information such as a timestamp indicating the (relative) time when the event occurred before the event data interval changes from the event occurrence interval. The process of including the time information in the event data can be performed in any block other than the output unit 35, as long as the process is performed before the time information implicitly included in the event data is lost.

[0025] FIG. 3 is a block diagram showing an example of the configuration of the pixel array unit 31 shown in FIG.

[0026] The pixel array unit 31 may include a plurality of pixel blocks 41. Each pixel block 41 may include I×J pixels 51, which are one or more pixels arranged in I rows and J columns (I and J integers), an event detection unit 52, and a pixel signal generation unit 53. One or more pixels 51 in the pixel block 41 share the event detection unit 52 and the pixel signal generation unit 53. In addition, in each column of the pixel block 41, a VSL (vertical line) is wired to connect the pixel block 41 to the ADC of the AD conversion unit 34.

[0027] The pixel 51 receives light incident from an object and performs photoelectric conversion to generate a photocurrent as an electric signal. The pixel 51 supplies the generated photocurrent to the event detection unit 52 under the control of the drive unit 32.

[0028] Under the control of the driving unit 32, the event detection unit 52 detects, as an event, a change in the photocurrent from each pixel 51 that is greater than a predetermined threshold. When detecting an event, the event detection unit 52 may supply a request to the arbiter 33 (FIG. 2) requesting output of event data indicating the occurrence of the event. Then, upon receiving a response to the request from the arbiter 33 indicating permission to output the event data, the event detection unit 52 outputs the event data to the driving unit 32 and the output unit 35.

[0029] When the event detection unit 52 detects an event, the pixel signal generation unit 53 generates a voltage corresponding to the photocurrent from the pixel 51 as a pixel signal under the control of the drive unit 32, and supplies the generated voltage to the AD conversion unit 34 via VSL.

[0030] Here, detecting a change in photocurrent greater than a predetermined threshold as an event can also be recognized as detecting no change in photocurrent greater than the predetermined threshold as an event. The pixel signal generator 53 can generate a pixel signal not only when a change in photocurrent greater than a predetermined threshold is detected as an event, but also when no change in photocurrent greater than the predetermined threshold is detected as an event.

[0031] Also, the event detector 52 and the pixel signal generator 53 may operate independently of each other. This means that pixel signals are generated synchronously by the pixel signal generator 53 to generate frame data while event data is generated by the event detector 52. The event detector 52 and the pixel signal generator 53 may operate in a time-multiplexed manner such that each unit receives photocurrent from the pixel 51 only at different time intervals.

[0032] Fig. 4 is a circuit diagram showing an example of the configuration of the pixel block 41. The configuration shown in Fig. 4 is sometimes called a "shared pixel configuration."

[0033] The pixel block 41 includes a pixel 51, an event detection unit 52, and a pixel signal generation unit 53, as described with reference to FIG.

[0034] The pixel 51 includes a photoelectric conversion element 61 and transfer transistors 62 and 63 .

[0035] The photoelectric conversion element 61 includes, for example, a photodiode (PD). The photoelectric conversion element 61 receives incident light and performs photoelectric conversion to generate electric charges.

[0036] The transfer transistor 62 includes, for example, an N (negative) type MOS (metal oxide semiconductor) FET (field effect transistor). The transfer transistor 62 of the n-th pixel 51 of the I×J pixels 51 in the pixel block 41 is turned on or off in response to a control signal OFGn supplied from the drive unit 32 (FIG. 2). When the transfer transistor 62 is turned on, the charge generated in the photoelectric conversion element 61 is transferred (supplied) to the event detection unit 52 as a photocurrent.

[0037] The transfer transistor 63 includes, for example, an N-type MOSFET. The transfer transistor 63 of the n-th pixel 51 of the I×J pixels 51 in the pixel block 41 is turned on or off in response to a control signal TRGn supplied from the drive unit 32. When the transfer transistor 63 is turned on, the charge generated in the photoelectric conversion element 61 is transferred to the FD 74 of the pixel signal generation unit 53.

[0038] The I×J pixels 51 in the pixel block 41 are connected to the event detection unit 52 of the pixel block 41 via the node 60. As a result, the photocurrent generated in the pixel 51 (the photoelectric conversion element 61) is supplied to the event detection unit 52 via the node 60. As a result, the event detection unit 52 receives the entire photocurrent from all the pixels 51 in the pixel block 41. Therefore, the event detection unit 52 detects a change in the entire photocurrent supplied from the I×J pixels 51 in the pixel block 41 as an event.

[0039] The pixel signal generation unit 53 includes a reset transistor 71, an amplification transistor 72, a selection transistor 73, and an FD (floating diffusion) 74.

[0040] The reset transistor 71, the amplifying transistor 72, and the selection transistor 73 each include, for example, an N-type MOSFET.

[0041] The reset transistor 71 is turned on or off in response to a control signal RST supplied from the drive unit 32 (FIG. 2). When the reset transistor 71 is turned on, the FD 74 is connected to the power supply VDD, and the charge accumulated in the FD 74 is discharged to the power supply VDD. This resets the FD 74.

[0042] The amplifying transistor 72 has a gate connected to the FD 74, a drain connected to a power supply VDD, and a source connected to VSL via the selection transistor 73. The amplifying transistor 72 is a source follower, and outputs a voltage (electrical signal) corresponding to the voltage of the FD 74 supplied to the gate to VSL via the selection transistor 73.

[0043] The selection transistor 73 is turned on or off in response to a control signal SEL supplied from the drive unit 32. When the selection transistor 73 is turned on, a voltage corresponding to the voltage of the FD 74 from the amplification transistor 72 is output to VSL.

[0044] The FD 74 accumulates the charge transferred from the photoelectric conversion element 61 of the pixel 51 via the transfer transistor 63, and converts the accumulated charge into a voltage. With respect to the pixel 51 and pixel signal generation unit 53 configured as described above, the drive unit 32 turns on the transfer transistor 62 using a control signal OFGn, and the transfer transistor 62 supplies a photocurrent based on the charge generated in the photoelectric conversion element 61 of the pixel 51 to the event detection unit 52. As a result, the event detection unit 52 receives a current that is the total photocurrent from all the pixels 51 in the pixel block 41 (which may be only a single pixel).

[0045] When the event detection unit 52 detects a change in photocurrent (total photocurrent) in the pixel block 41 as an event, the drive unit 32 may turn off the transfer transistors 62 of all pixels 51 in the pixel block 41 to stop the supply of photocurrent to the event detection unit 52. Then, the drive unit 32 may use a control signal TRGn to sequentially turn on the transfer transistors 63 of the pixels 51 in the pixel block 41 in which the event is detected, so that the transfer transistors 63 transfer the charge generated in the photoelectric conversion element 61 to the FD 74. The FD 74 accumulates the charge transferred from the pixel 51 (the photoelectric conversion element 61). A voltage corresponding to the charge accumulated in the FD 74 is output to the VSL as a pixel signal of the pixel 51 via the amplification transistor 72 and the selection transistor 73.

[0046] As described above, in the sensor unit 21 (FIG. 2), only the pixel signals of the pixels 51 in the pixel blocks 41 in which an event has been detected are sequentially output to the VSL in this manner. The pixel signals output to the VSL are supplied to the AD conversion unit 34 and are AD converted.

[0047] Here, the transfer transistors 63 can be turned on simultaneously, not sequentially, in the pixels 51 in the pixel block 41. In this case, the entire pixel signals of all the pixels 51 in the pixel block 41 can be output.

[0048] It is also possible for the pixel signal generator 53 to operate independently of event detection, either by controlling the transfer transistor 63 independently of event detection, or by omitting the transfer transistor 63 entirely. In this case, frame data can be generated by the pixel signal generator 53 in a conventional synchronous manner, and event detection can occur simultaneously. If desired, the transfer of photocurrent to the event detector 52 and pixel signal generator 53 may be time multiplexed, such that only one of the two transfer transistors 62 and 63 per pixel 51 is open at a given time.

[0049] 3, pixel block 41 includes one or more pixels 51, and one or more pixels 51 share an event detection unit 52 and a pixel signal generation unit 53. As a result, when pixel block 41 includes multiple pixels 51, the number of event detection units 52 and pixel signal generation units 53 can be reduced compared to when an event detection unit 52 and a pixel signal generation unit 53 are provided for each pixel 51, and as a result, the size of pixel array unit 31 can be reduced.

[0050] When the pixel block 41 includes multiple pixels 51, an event detection unit 52 can be provided for each pixel 51. When multiple pixels 51 in the pixel block 41 share an event detection unit 52, an event is detected in units of pixel block 41. However, when an event detection unit 52 is provided for each pixel 51, an event can also be detected in units of pixel 51.

[0051] Even if multiple pixels 51 in a pixel block 41 share an event detection unit 52, events can be detected on a pixel 51 basis by temporarily turning on the transfer transistors 62 of the multiple pixels 51 in a time-division manner.

[0052] Furthermore, if there is no need to output a pixel signal, the pixel block 41 may be formed without the pixel signal generation unit 53. When the pixel block 41 is formed without the pixel signal generation unit 53, the sensor unit 21 can be formed without the AD conversion unit 34 and the transfer transistor 63. In this case, the size of the sensor unit 21 can be reduced. The sensor then outputs the address of the pixel (block) in which the event has occurred, together with a timestamp as necessary.

[0053] Furthermore, the sensor die 11 may be provided with additional pixels 51 connected to the pixel signal generation unit 53 but not to the event detection unit 52. These pixels 51 may be interleaved with the pixels 51 connected to the event detection unit 52, but may also constitute a separate pixel array for generating frame data. The separation of the event detection and pixel signal generation functions may for example be implemented as two pixel arrays on different dies arranged to observe the same scene. Thus, in principle any pixel arrangement / pixel circuitry may be used that allows obtaining event data with a first subset of pixels 51 and generating pixel signals with a second subset of pixels 51.

[0054] FIG. 5 is a block diagram showing an example of the configuration of the event detection unit 52 shown in FIG.

[0055] The event detection unit 52 includes a current-voltage conversion unit 81 , a buffer 82 , a subtraction unit 83 , a quantization unit 84 , and a transfer unit 85 .

[0056] The current-voltage converter 81 converts the (total) photocurrent from the pixel 51 into a voltage (hereinafter also referred to as “photovoltage”) corresponding to the logarithm of the photocurrent, and supplies the voltage to a buffer 82.

[0057] The buffer 82 buffers the photovoltage from the current-voltage conversion unit 81 , and supplies the buffered photovoltage to the subtraction unit 83 .

[0058] The subtraction unit 83 calculates the difference between the current optical voltage at a timing indicated by the column drive signal, which is a control signal from the drive unit 32, and the optical voltage at a timing slightly shifted from the current time, and supplies a differential signal corresponding to the difference to the quantization unit 84.

[0059] The quantization unit 84 quantizes the difference signal from the subtraction unit 83 into a digital signal, and supplies the quantized value of the difference signal as event data to the transfer unit 85. The transfer unit 85 transfers (outputs) the event data to the output unit 35 based on the event data from the quantization unit 84. That is, the transfer unit 85 supplies a request to the arbiter 33 to request output of the event data. Then, upon receiving a response to the request from the arbiter 33 indicating permission to output the event data, the transfer unit 85 outputs the event data to the output unit 35.

[0060] FIG. 6 is a circuit diagram showing an example of the configuration of the current-voltage conversion unit 81 shown in FIG.

[0061] The current-voltage conversion unit 81 includes transistors 91 to 93. For example, an N-type MOSFET can be used as the transistors 91 and 93. For example, a P-type MOSFET can be used as the transistor 92.

[0062] The source of transistor 91 is connected to the gate of transistor 93, and photocurrent is supplied from pixel 51 to the connection point between the source of transistor 91 and the gate of transistor 93. The drain of transistor 91 is connected to a power supply VDD, and the gate is connected to the drain of transistor 93.

[0063] The source of the transistor 92 is connected to a power supply VDD, and the drain is connected to the connection point between the gate of the transistor 91 and the drain of the transistor 93. A predetermined bias voltage Vbias is supplied to the gate of the transistor 92. The bias voltage Vbias turns the transistor 92 on or off, and the operation of the current-voltage conversion unit 81 is turned on or off depending on whether the transistor 92 is on or off.

[0064] The source of transistor 93 is grounded.

[0065] In the current-voltage conversion unit 81, the transistor 91 has a drain connected to the power supply VDD side and serves as a source follower. The source of the transistor 91, which is a source follower, is connected to the pixel 51 (FIG. 4), and a photocurrent based on charges generated in the photoelectric conversion element 61 of the pixel 51 flows through the transistor 91 (from the drain to the source). The transistor 91 operates in the subthreshold region, and a photovoltage corresponding to the logarithm of the photocurrent flowing through the transistor 91 is generated at the gate of the transistor 91. As described above, in the current-voltage conversion unit 81, the transistor 91 converts the photocurrent from the pixel 51 into a photovoltage corresponding to the logarithm of the photocurrent.

[0066] In the current-voltage conversion section 81, the gate of the transistor 91 is connected to the connection point between the drain of the transistor 92 and the drain of the transistor 93, and a photovoltage is output from this connection point.

[0067] FIG. 7 is a circuit diagram showing an example of the configuration of the subtraction unit 83 and the quantization unit 84 shown in FIG.

[0068] The subtraction unit 83 includes a capacitor 101 , an operational amplifier 102 , a capacitor 103 , and a switch 104 .

[0069] The quantization unit 84 includes a comparator 111. One end of the capacitor 101 is connected to the output terminal of the buffer 82 (FIG. 5), and the other end is connected to the input terminal (inverting input terminal) of the operational amplifier 102. Therefore, the photovoltage is input to the input terminal of the operational amplifier 102 via the capacitor 101.

[0070] The output terminal of the operational amplifier 102 is connected to the non-inverting input terminal (+) of a comparator 111 .

[0071] One end of the capacitor 103 is connected to the input terminal of the operational amplifier 102 , and the other end is connected to the output terminal of the operational amplifier 102 .

[0072] The switch 104 is connected to the capacitor 103 and switches the connection between both ends of the capacitor 103. The switch 104 is turned on or off in response to a row drive signal, which is a control signal from the drive unit 32, thereby switching the connection between both ends of the capacitor 103.

[0073] When the switch 104 is on, the optical voltage on the buffer 82 (FIG. 5) side of the capacitor 101 is denoted as Vinit, and the capacitance (electrostatic capacitance) of the capacitor 101 is denoted as C1. The input terminal of the operational amplifier 102 functions as a virtual ground terminal, and the charge Qinit accumulated in the capacitor 101 when the switch 104 is on is expressed by equation (1).

[0074] Qinit = C1 × Vinit (1)

[0075] Moreover, when the switch 104 is on, the connection between both ends of the capacitor 103 is cut (shorted), and no charge is stored in the capacitor 103.

[0076] Thereafter, when the switch 104 is turned off, the photovoltage on the buffer 82 (FIG. 5) side of the capacitor 101 is denoted as Vafter. The charge Qafter accumulated in the capacitor 101 when the switch 104 is off is expressed by the formula (2).

[0077] Qafter=C1×Vafter (2)

[0078] If the capacitance of the capacitor 103 is C2 and the output voltage of the operational amplifier 102 is Vout, then the charge Q2 stored in the capacitor 103 is expressed by equation (3).

[0079] Q2=-C2×Vout (3)

[0080] Since the total amount of charge in the capacitors 101 and 103 does not change before and after the switch 104 is turned off, the formula (4) holds.

[0081] Qinit=Qafter+Q2 (4)

[0082] By substituting equations (1) to (3) into equation (4), equation (5) is obtained.

[0083] Vout=-(C1 / C2)×(Vafter-Vinit) (5)

[0084] According to the formula (5), the subtraction unit 83 subtracts the photovoltage Vinit from the photovoltage Vafter. That is, the subtraction unit 83 calculates a difference signal (Vout) corresponding to the difference Vafter-Vinit between the photovoltages Vafter and Vinit. According to the formula (5), the subtraction gain of the subtraction unit 83 is C1 / C2. Since the maximum gain is usually desirable, it is preferable to set C1 to a large value and C2 to a small value. On the other hand, if C2 is too small, kTC noise increases and noise characteristics may deteriorate. Therefore, the capacitance C2 can only be made small within a range where noise is tolerable. In addition, since the event detection unit 52 including the subtraction unit 83 is provided in each pixel block 41, the capacitances Cl and C2 are subject to space restrictions. The values ​​of the capacitances C1 and C2 are determined taking these factors into consideration.

[0085] The comparator 111 compares the difference signal from the subtraction unit 83 with a predetermined threshold (voltage) Vth (>0) applied to the inverting input terminal (-), and quantizes the difference signal. The comparator 111 outputs the quantized value obtained by the quantization to the transfer unit 85 as event data.

[0086] For example, if the difference signal is greater than the threshold Vth, the comparator 111 outputs an H (High) level indicating 1 as event data indicating the occurrence of an event. If the difference signal is not greater than the threshold Vth, the comparator 111 outputs an L (Low) level indicating 0 as event data indicating the absence of an event.

[0087] When it is confirmed that a change in the amount of light, which is an event, has occurred based on the event data from the quantization unit 84, that is, when the differential signal (Vout) is greater than the threshold value Vth, the transfer unit 85 supplies a request to the arbiter 33. When a response indicating permission to output event data is received, the transfer unit 85 outputs event data (e.g., H level) indicating the occurrence of an event to the output unit 35.

[0088] The output unit 35 includes, in the event data from the transfer unit 85, position / address information on the pixel 51 (including the pixel block 41) where the event indicated by the event data occurred, time point information indicating the time when the event occurred, and, as necessary, the polarity of the change in light amount that is the event, i.e., whether the intensity has increased or decreased. The output unit 35 outputs the event data.

[0089] As a data format of the event data including the position information on the pixel 51 where the event occurred, the time point information indicating the time point when the event occurred, and the polarity of the change in the amount of light which is the event, for example, a data format called "Address Event Representation (AER)" can be adopted. The gain A of the entire event detection unit 52 is expressed by the following formula (6). In formula (6), the gain of the current-voltage conversion unit 81 is expressed by CG log and the gain of the buffer 82 is set to 1.

[0090] A=CG log C1 / C2(Σi photo _n) (6)

[0091] where i photo n represents the photocurrent of the n-th pixel 51 of the I×J pixels 51 in the pixel block 41. In equation (6), Σ represents the sum of n, which is an integer from 1 to I×J.

[0092] Note that the pixel 51 can receive any light as incident light by using an optical filter such as a color filter that transmits predetermined light. For example, when the pixel 51 receives visible light as incident light, the event data indicates the occurrence of a change in pixel value in an image including a visible object. Also, for example, when the pixel 51 receives infrared light or millimeter waves for distance measurement as incident light, the event data indicates the occurrence of a change in the distance to the object. Also, for example, when the pixel 51 receives infrared light for temperature measurement as incident light, the event data indicates the occurrence of a change in the temperature of the object. In this embodiment, the pixel 51 receives visible light as incident light.

[0093] FIG. 8 is a timing chart showing an example of the operation of the sensor unit 21 shown in FIG.

[0094] At timing T0, the drive unit 32 changes all the control signals OFGn from the L level to the H level to turn on the transfer transistors 62 of all the pixels 51 in the pixel block 41. This causes the entire photocurrent from all the pixels 51 in the pixel block 41 to be supplied to the event detection unit 52. Here, since all the control signals TRGn are at the L level, the transfer transistors 63 of all the pixels 51 are off.

[0095] For example, when an event is detected at timing T1, the event detection section 52 outputs H-level event data in response to the detection of the event.

[0096] At timing T2, the drive unit 32 sets all control signals OFGn to L level based on the H level event data, and stops the supply of photocurrent from the pixels 51 to the event detection unit 52. The drive unit 32 also sets the control signal SEL to H level, sets the control signal RST to H level for a certain period of time, and controls the FD 74 to release charge to the power supply VDD, thereby resetting the FD 74. The pixel signal generation unit 53 outputs a pixel signal corresponding to the voltage of the FD 74 when the FD 74 is reset as a reset level, and the AD conversion unit 34 AD converts the reset level.

[0097] At timing T3 after AD conversion of the reset level, the drive unit 32 sets the control signal TRG1 to H level for a certain period to control the first pixel 51 in the pixel block 41 in which an event has been detected (or triggered for other reasons such as time multiplexed output and / or synchronous readout of frame data) so as to transfer charge generated by photoelectric conversion in (the photoelectric conversion element 61 of) the first pixel 51 to the FD 74. The pixel signal generation unit 53 outputs a pixel signal corresponding to the voltage of the FD 74 to which the charge has been transferred from the pixel 51 as a signal level, and the AD conversion unit 34 performs AD conversion on the signal level.

[0098] The AD conversion section 34 outputs the difference between the signal level obtained by the AD conversion and the reset level to the output section 35 as a pixel signal that is a pixel value (frame data) of the image.

[0099] Here, the process of obtaining the difference between the signal level and the reset level as a pixel signal, which is the pixel value of an image, is called "CDS." CDS can be performed after AD conversion of the signal level and the reset level, or, if the AD conversion unit 34 performs single-slope AD conversion, it can be performed simultaneously with AD conversion of the signal level and the reset level. In the latter case, the AD conversion of the signal level is performed using the result of the AD conversion of the reset level as an initial value.

[0100] At timing T4 after AD conversion of the pixel signal of the first pixel 51 in the pixel block 41, the drive unit 32 controls the second pixel 51 in the pixel block 41 in which the event was detected to set the control signal TRG2 to an H level for a certain period of time and output a pixel signal.

[0101] The sensor unit 21 subsequently executes the same process, and sequentially outputs pixel signals from the pixels 51 in the pixel blocks 41 in which events have been detected.

[0102] When the pixel signals of all the pixels 51 in the pixel block 41 have been output, the drive unit 32 sets all the control signals OFGn to the H level, and turns on the transfer transistors 62 of all the pixels 51 in the pixel block 41 .

[0103] FIG. 9 is a diagram illustrating an example of a method for generating frame data based on event data.

[0104] The logic unit 22 sets the frame interval and frame width based on, for example, an externally input command. Here, the frame interval represents the frame interval of frame data generated based on event data. The frame width represents the time width of the event data used to generate the frame data of a single frame. The frame interval and frame width set by the logic unit 22 are also called the "set frame interval" and the "set frame width", respectively.

[0105] The logic unit 22 generates frame data, which is image data in a frame format, based on the set frame interval, set frame width, and event data from the sensor unit 21, thereby converting the event data into frame data.

[0106] That is, the logic section 22 generates frame data for each set frame interval based on event data for the set frame width from the beginning of the set frame interval.

[0107] Here, the event data includes time information t indicating the time when the event occurred (hereinafter, also referred to as the "event time"). i and coordinates (x, y) which are position information (hereinafter also referred to as "event position") regarding the pixel 51 (or pixel block 41 including the pixel 51) where the event occurred. In Fig. 9, points representing event data are plotted in a three-dimensional space (space-time) having an x-axis, a y-axis, and a time axis t, based on the event time t and event position (coordinates) (x, y) included in the event data.

[0108] In other words, if the position (x, y, t) in three-dimensional space indicated by the event time t and event position (x, y) contained in the event data is regarded as the spatiotemporal position of the event, then in FIG. 9, a point representing the event data is plotted at the spatiotemporal position (x, y, t) of the event.

[0109] The logic unit 22 starts generating frame data based on the event data, using a predetermined point in time, for example, the point in time when frame data generation is instructed from outside or the point in time when the power of the sensor device 10 is turned on, as the start point in time for starting frame data generation.

[0110] Here, a rectangular parallelepiped having a set frame width in the direction of the time axis t of the set frame interval that appears from the generation start time point is called a "frame volume." The size of the frame volume in the x-axis direction and y-axis direction is equal to, for example, the number of pixel blocks 41 or pixels 51 in the x-axis direction or y-axis direction.

[0111] The logic unit 22 generates frame data for a single frame in each set frame interval based on event data in a frame volume having the set frame width from the beginning of the set frame interval.

[0112] The frame data can be generated, for example, by setting the pixels (pixel values) of the frame to white at the event position (x, y) included in the event data, and setting the pixels to a predetermined color such as gray at other positions in the frame.

[0113] Furthermore, if the event data includes the polarity of the change in the amount of light that is the event, the frame data can be generated taking into consideration the polarity included in the event data. For example, in the case of positive polarity, the pixel can be set to white, and in the case of negative polarity, the pixel can be set to black.

[0114] 3 and 4, if the pixel signal of pixel 51 is also output when the event data is output, frame data can also be generated based on the event data using the pixel signal of pixel 51. That is, in a frame, the pixel at the event position (x, y) (in the block corresponding to pixel block 41) included in the event data is set to the pixel signal of pixel 51 at position (x, y), and a predetermined color such as gray is set to pixels at other positions, thereby generating frame data.

[0115] In addition, a frame volume may contain multiple event data with different event times t but the same event position (x, y). In this case, for example, priority can be given to the event data with the newest or oldest event time t. In addition, when the event data includes polarity, the polarities of multiple event data with different event times t but the same event position (x, y) can be added, and a pixel value based on the value obtained by the addition can be set to the pixel at the event position (x, y). Here, when the frame width and frame interval are the same, the frame volumes are adjacent to each other with no gaps. In addition, when the frame interval is larger than the frame width, the frame volumes are arranged with gaps. When the frame width is larger than the frame interval, the frame volumes are arranged so as to overlap partially.

[0116] As mentioned above, the pixel signal generator 53 may generate the frame data in a conventional synchronous manner.

[0117] FIG. 10 is a block diagram showing another example of the configuration of the quantization unit 84 shown in FIG.

[0118] In FIG. 10, parts corresponding to those in FIG. 7 are given the same reference numerals, and the description thereof will be omitted below as appropriate.

[0119] 10, the quantization unit 84 includes comparators 111 and 112 and an output unit 113.

[0120] Thus, the quantization unit 84 shown in FIG. 10 is similar to the quantization unit 84 shown in FIG. 7 in that it includes a comparator 111. However, the quantization unit 84 shown in FIG. 10 is different from the quantization unit 84 shown in FIG. 7 in that it newly includes a comparator 112 and an output unit 113.

[0121] The event detection unit 52 (FIG. 5) including the quantization unit 84 shown in FIG. 10 detects the polarity of the change in the amount of light, which is an event, in addition to the event.

[0122] In the quantization unit 84 shown in FIG. 10, when the differential signal is greater than the threshold value Vth, the comparator 111 outputs an H level indicating 1 as event data indicating the occurrence of an event having a positive polarity. When the differential signal is not greater than the threshold value Vth, the comparator 111 outputs an L level indicating 0 as event data indicating that no event having a positive polarity has occurred.

[0123] Also, in the quantization unit 84 shown in FIG. 10, a threshold value Vth' (<Vth) is supplied to the non-inverting input terminal (+) of the comparator 112, and the differential signal is supplied from the subtraction unit 83 to the inverting input terminal (-) of the comparator 112. Here, for simplicity of explanation, the threshold value Vth' is assumed to be equal to, for example, -Vth, but it is not necessarily so.

[0124] The comparator 112 compares the differential signal from the subtraction unit 83 with the threshold value Vth' applied to the inverting input terminal (-), thereby quantizing the differential signal. The comparator 112 outputs the quantization value obtained by quantization as event data.

[0125] For example, when the differential signal is smaller than the threshold Vth' (the absolute value of the differential signal having a negative value is larger than the threshold Vth), the comparator 112 outputs an H level indicating 1 as event data indicating the occurrence of an event having negative polarity. On the other hand, when the differential signal is not smaller than the threshold Vth' (the absolute value of the differential signal having a negative value is not larger than the threshold Vth), the comparator 112 outputs an L level indicating 0 as event data indicating the absence of an event having negative polarity.

[0126] Based on the event data output from comparators 111 and 112, output unit 113 outputs to transfer unit 85 event data indicating the occurrence of an event having positive polarity, event data indicating the occurrence of an event having negative polarity, or event data indicating that no event has occurred.

[0127] For example, when the event data from the comparator 111 is at an H level indicating 1, the output unit 113 outputs +V volts indicating +1 to the transfer unit 85 as event data indicating the occurrence of an event having positive polarity. When the event data from the comparator 112 is at an H level indicating 1, the output unit 113 outputs -V volts indicating -1 to the transfer unit 85 as event data indicating the occurrence of an event having negative polarity. When the event data from the comparators 111 and 112 is at an L level indicating 0, the output unit 113 outputs 0 volts (GND level) indicating 0 to the transfer unit 85 as event data indicating the absence of an event.

[0128] When it is confirmed that a change in light amount, which is an event having positive or negative polarity, has occurred based on the event data from the output unit 113 of the quantization unit 84, the transfer unit 85 supplies a request to the arbiter 33. After receiving a response indicating permission to output event data, the transfer unit 85 outputs to the output unit 35 event data indicating the occurrence of an event having positive or negative polarity (+V volts indicating 1 or -V volts indicating -1).

[0129] The quantization section 84 preferably has a configuration as shown in FIG.

[0130] FIG. 11 is a diagram showing another example of the configuration of the event detection unit 52. In FIG.

[0131] 11, the event detection unit 52 includes a subtractor 430, a quantizer 440, a memory 451, and a controller 452. The subtractor 430 and the quantizer 440 correspond to the subtraction unit 83 and the quantization unit 84, respectively.

[0132] In FIG. 11, the event detection unit 52 further includes blocks corresponding to a current-voltage conversion unit 81 and a buffer 82, but these blocks are omitted from FIG.

[0133] The subtractor 430 includes a capacitor 431, an operational amplifier 432, a capacitor 433, and a switch 434. The capacitor 431, the operational amplifier 432, the capacitor 433, and the switch 434 correspond to the capacitor 101, the operational amplifier 102, the capacitor 103, and the switch 104, respectively.

[0134] The quantizer 440 includes a comparator 441. The comparator 441 corresponds to the comparator 111.

[0135] The comparator 441 compares the voltage signal (difference signal) from the subtractor 430 with a predetermined threshold voltage Vth applied to an inverting input terminal (-) of the comparator 441. The comparator 441 outputs a signal indicating the comparison result as a detection signal (quantized value).

[0136] The voltage signal from the subtractor 430 may be input to an input terminal (−) of a comparator 441 , and the predetermined threshold voltage Vth may be input to an input terminal (+) of the comparator 441 .

[0137] The controller 452 supplies a predetermined threshold voltage Vth to be applied to the inverting input terminal (-) of the comparator 441. The supplied threshold voltage Vth may vary in a time-division manner. For example, the controller 452 supplies a threshold voltage Vth1 corresponding to an on-event (e.g., a positive change in photocurrent) and a threshold voltage Vth2 corresponding to an off-event (e.g., a negative change in photocurrent) at different timings, allowing a single comparator to detect multiple types of address events.

[0138] The memory 451 accumulates the output from the comparator 441 based on a sample signal supplied from the controller 452. The memory 451 may be a sampling circuit such as a switch, a plastic, or a capacitor, or a digital memory circuit such as a latch or a flip-flop. For example, the memory 451 may hold a comparison result by the comparator 441 using a threshold voltage Vth1 corresponding to an on-event during a period in which a threshold voltage Vth2 corresponding to an off-event is supplied to the inverting input terminal (-) of the comparator 441. Note that the memory 451 may be omitted, or may be provided inside or outside the pixel (pixel block 41).

[0139] FIG. 12 is a block diagram showing another example of the configuration of the pixel array unit 31 shown in FIG.

[0140] In addition, in FIG. 12, parts corresponding to those in FIG. 3 are given the same reference numerals, and the description thereof will be omitted below as appropriate.

[0141] 12, the pixel array section 31 includes a plurality of pixel blocks 41. The pixel block 41 includes one or more I×J pixels 51 and an event detection section 52.

[0142] 12 is similar to the pixel array section 31 shown in Fig. 3 in that the pixel array section 31 includes a plurality of pixel blocks 41, and each pixel block 41 includes one or more pixels 51 and an event detection section 52. However, the pixel array section 31 shown in Fig. 12 differs from the pixel array section 31 shown in Fig. 3 in that the pixel block 41 does not include the pixel signal generation section 53.

[0143] 12, since the pixel block 41 does not include the pixel signal generation unit 53, the sensor unit 21 (FIG. 2) can be formed without the AD conversion unit 34. Furthermore, as described above, the pixel signal generation unit 53 and the associated pixels 51 may be disposed in a different part of the die 11, or may be disposed in another sensor chip.

[0144] Fig. 13 is a circuit diagram showing an example of the configuration of the pixel block 41 shown in Fig. 12. As described with reference to Fig. 12, the pixel block 41 includes the pixel 51 and the event detection unit 52, but does not include the pixel signal generation unit 53.

[0145] In this case, the pixel 51 may include only the photoelectric conversion element 61 without the transfer transistors 62 and 63 .

[0146] When the pixel 51 has the configuration shown in FIG. 13, the event detection section 52 can output a voltage corresponding to the photocurrent from the pixel 51 as a pixel signal.

[0147] The sensor device 10 has been described above as an asynchronous imaging device configured to read out events by an asynchronous readout method. However, the event readout method is not limited to the asynchronous readout method, and may be a synchronous readout method. An imaging device to which the synchronous readout method is applied is a scan-type imaging device similar to a general imaging device configured to capture images at a predetermined frame rate. Also, event data detection may be performed asynchronously, and pixel signal generation may be performed synchronously.

[0148] FIG. 14 is a block diagram showing an example of the configuration of a scan-type imaging device.

[0149] As shown in FIG. 14, the imaging device 510 includes a pixel array section 521, a drive section 522, a signal processing section 525, a readout region selection section 527, and a signal generation section 528.

[0150] The pixel array unit 521 includes a plurality of pixels 530. Each of the plurality of pixels 530 outputs an output signal in response to a selection signal from the readout region selection unit 527. Each of the plurality of pixels 530 may include an in-pixel quantizer, for example, as shown in FIG. 11. The plurality of pixels 530 outputs an output signal corresponding to an amount of change in light intensity. The plurality of pixels 530 may be two-dimensionally arranged in a matrix, as shown in FIG. 14.

[0151] The driving unit 522 drives a plurality of pixels 530, and the pixels 530 output pixel signals generated in the pixels 530 to a signal processing unit 525 via an output line 514. The driving unit 522 and the signal processing unit 525 are circuit units for acquiring grayscale information. Therefore, when only event information (event data) is acquired, the driving unit 522 and the signal processing unit 525 may be omitted.

[0152] The readout region selection unit 527 selects a part of the multiple pixels 530 included in the pixel array unit 521. For example, the readout region selection unit 527 selects one or more rows included in a two-dimensional matrix structure corresponding to the pixel array unit 521. The readout region selection unit 527 sequentially selects one or more rows based on a preset cycle. The readout region selection unit 527 may also determine the selected region based on a request from the pixel 530 in the pixel array unit 521. The signal generation unit 528 generates an event signal corresponding to an active pixel in which an event has been detected among the selected pixels 530 based on the output signal of the pixel 530 selected by the readout region selection unit 527. An event means an event in which the intensity of light changes. An active pixel means a pixel 530 in which the amount of change in the intensity of light corresponding to the output signal exceeds or falls below a preset threshold. For example, the signal generation unit 528 compares the output signal from the pixel 530 with a reference signal, and detects a pixel that outputs an output signal larger or smaller than the reference signal as an active pixel. The signal generating unit 528 generates an event signal (event data) corresponding to the active pixel.

[0153] The signal generator 528 may include, for example, a column selection circuit configured to arbitrate the signals input to the signal generator 528. The signal generator 528 may also output information regarding active pixels where an event is detected as well as information regarding inactive pixels where no event is detected, i.e., the signal generator 528 may operate as a conventional synchronous image sensor generating a continuous series of image frames.

[0154] The signal generator 528 outputs address information and timestamp information (e.g., (X,Y,T)) relating to the active pixel in which an event is detected via output line 515. However, the data output from the signal generator 528 may be not only address information and timestamp information, but also information in frame format (e.g., (0,0,1,0,...)).

[0155] The above describes a sensor device 10 in which event data generation and pixel signal generation may be interdependent or independent of each other. Furthermore, pixels 51 may be shared between the event detection circuitry and the pixel signal generation by respective circuitry or time multiplexing. However, pixels 51 may also be split such that there are event detection pixels and pixel signal generation pixels. These pixels 51 may be interleaved in the same pixel array or may be part of different pixel arrays on the same die or may be located on different dies.

[0156] In all these configurations, the event detection circuitry operates on the pixels 51 of the first subset and is configured to detect as event data a change in intensity of the light received at each of the pixels 51 of the first subset that exceeds a predefined threshold, and the pixel signal generation circuitry operates on the pixels 51 of the second subset and is configured to generate a pixel signal indicative of an intensity value of the light received for each of the pixels 51 of the second subset. Here, at least some of the pixels 51 may belong to the pixels 51 of the first subset and to the pixels 51 of the second subset. That is, at least some of the pixels 51 may be shared pixels 51 or pixels 51 whose output is distributed in a time multiplexed manner. However, the pixels 51 in the pixels 51 of the first subset may be different from the pixels 51 in the pixels 51 of the second subset. That is, the sensor device 10 may include pixels 51 that are divided for event detection and intensity frame generation.

[0157] FIG. 15 shows a schematic diagram of an exemplary embodiment of the sensor device 10. As mentioned above, the sensor device 10 comprises a number of pixels 51, each configured to receive light and perform photoelectric conversion to generate an electrical signal. Here, as explained with respect to FIG. 3, the pixels 51 may be arranged in pixel blocks 41. For ease of explanation, it is assumed below that there is one pixel 51 per pixel block 41, i.e., the pixel 51 and the pixel block 41 refer to the same structural element, but this is not limiting. However, the following explanation can be easily generalized by understanding the reference to a pixel 51 as a reference to a pixel block 41 including several pixels 51.

[0158] The sensor device 10 comprises an event detection circuit 20 configured to detect as event data D a change in intensity of the light received by each of the pixels 51 of the first subset S1 that exceeds a predefined threshold. The event detection circuit 20 is essentially constituted by an event detection unit 52 that is able to receive the photocurrent of the pixels 51 as described above and detect an event due to a change in intensity of the received light that is greater than a predefined (but possibly dynamically adjustable) event threshold. Although the event detection circuit 20 is shown in FIG. 15 as a block adjacent to the pixel 51, this should be understood as merely symbolic. As described above, the event detection unit 52 of the event detection circuit 20 may be located near or next to the pixel 51. However, the event detection units 52 may also be grouped in a dedicated area on the sensor chip. In particular, the event detection circuit 20 may be located on a different wafer / die than the pixels 51 and vertically connected to the pixels 51.

[0159] The pixels 51 comprise a first subset S1. In Figure 15, the pixels 51 of the first subset S1 are indicated by vertical hatching. These pixels 51 are connected to the event detection circuitry 20, for example as described above, to enable event detection on the light they receive.

[0160] The pixels 51 further comprise a second subset S2 consisting of the pixels 51 that are not hatched in Fig. 15. The pixels 51 of the second subset S2 are used to generate full intensity information, i.e. data representative of the intensity of light received by the pixels 51. It is understood that each pixel 51 of the second subset S2 may be provided with a suitable colour filter to enable the generation of a colour image. The colour filters may be provided on the pixels 51 of the first subset S1.

[0161] The sensor device 10 comprises a pixel signal generating circuit 30 configured to generate pixel signals indicative of received light intensity values ​​for the pixels 51 in the second subset S2 of pixels 51. For example, the pixel signal generating circuit 30 may be constituted by a pixel signal generator 53 capable of generating pixel signals indicative of received light intensity in an asynchronous (event triggered) manner or in a conventional synchronous manner, as described above. In particular, the pixels 51 of the second subset S2 may operate according to known principles of active pixel sensors (APS). As with the event detection circuit 20, the pixel signal generating circuit 30 may also be formed in a distributed manner, in which case, for example, one pixel signal generator 53 is located next to or close to one pixel 51 of the second subset S2 or in a dedicated area of ​​the sensor chip. It is also conceivable that the pixel signal generating circuit 30 includes all control, reset and / or selection lines etc. necessary to operate the sensor device 10 as an APS capable of generating a continuous stream of pixel signals indicative of received light intensity over a given period of time. The sensor device 10 further comprises a controller 40 configured to start detecting the event data D at a first time point t1 and to start generating the pixel signal at a second time point t2, the first time point t1 being prior to the second time point t2, i.e., during the period between t1 and t2, only the event data D is generated but no pixel signal indicative of the intensity of the received light is generated.

[0162] Here, the control unit 40 may be any circuit configuration capable of executing the above-mentioned functions. For example, the control unit 40 may be configured by a processor. The control unit 40 may be a part of the pixel unit of the sensor device 10, and may be arranged on the same die as other components of the sensor device 10. However, the control unit 40 may be arranged separately, for example, on a separate die. The functions of the control unit 40 may be implemented completely in hardware or software, or may be implemented as a mixture of hardware and software functions.

[0163] The control unit 40 is further configured to reconstruct intensity values ​​for the pixels 51 of the second subset S2 in the period between the first and second points in time t1 and t2 using the pixel signals generated after the second point in time t2 and the event data D detected before the second point in time t2. The control unit 40 thus uses the intensity difference information present in the event data D to reconstruct intensity data back in time for the pixels 51 of the second subset S2, for example by integration, to generate data that would have been present if the generation of pixel signals by the pixels of the second subset D2 had started before the second point in time t2.

[0164] This is shown in the lower part of FIG. 15. Here, event detection starts at time t1 and generates a stream of event data D spatially distributed across a number of pixels 51 of the first subset S1. For example, events of positive and negative polarity can be detected, as shown in FIG. 15 by the white and black circles. At t2, the pixels of the second subset S2 start capturing a stream of intensity data, directed towards the same scene as observed during the event data generation. As shown in FIG. 15, the stream of intensity data may form a continuous series of image frames F1. That is, the stream of intensity data may be a conventional video stream. However, the pixels of the second subset S2 do not necessarily need to be located adjacent to each other, but may be sparsely distributed. In this case, full frames are not generated, but rather point-like or patch-like image portions are generated. The intensity data may also be generated asynchronously, for example by event triggering as described above.

[0165] The pixel signals containing intensity information generated after time t2 can be used to generate corresponding intensity information for a previous time point by referring to the event data D. Indeed, since the event data D contains information indicating the time at which the intensity change exceeded the event threshold, it is in principle possible to track the intensity change of each pixel 51 of the second subset S2 if there is a corresponding pixel 51 of the first subset S1 that observed the same part of the captured scene.

[0166] In this way, the first intensity information acquired after time t2 can be extrapolated back in time. For example, a series of intensity frames F1 captured after time t2 can be supplemented with a series of previous frames F2, as shown in Fig. 15. The previous intensity information may in principle date back to the time t1 when the event detection started. The reconstructed period may have a fixed length, for example between 1 and 5 seconds, or may be freely adjusted by the user of the sensor device 10.

[0167] Thus, by reconstruction, the intensity information prior to the start of the capture of the intensity information, e.g. via video frame capture, can be recovered, e.g. in the form of video frame F2. As mentioned above, the reconstruction is possible by event data from pixels 51 of the first subset S1, which event data is preferably recorded continuously at least between a first time point t1 and a second time point t2.

[0168] The event-based data is then used together with the actual recorded intensity information to reconstruct the lost intensity information back in time, so that the reconstruction has a more realistic and continuous appearance compared to when the intensity information is reconstructed from event data alone, i.e. without reference to pixel signals from pixels 51 of the second subset S2.

[0169] Generating event data inherently consumes less power than generating intensity information, for example because fewer pixels 51 are operated per time instance. This effect is enhanced when fewer pixels 51 are used for event detection than for capturing intensity information. In principle, the event detection circuit 20 can therefore operate continuously to always allow reconstruction of intensity information back in time when a capture of intensity information, preferably on a frame basis, was triggered. Thus, it can be avoided in a power-saving manner that, for example, a time instance is lost at the beginning of a video because the user failed to trigger the video capture early enough.

[0170] Now, in principle, it is possible to perform event detection whenever the sensor device 10 is on, i.e. the first time point t1 may be the time point when the sensor device 10 is switched on.

[0171] Alternatively, event detection may be switched on based on a certain trigger to reduce energy consumption. That is, the first time point t1 may be the time point when the sensor device 10 senses a predetermined condition. To this end, the sensor device 10 may be equipped with or connected to various sensors measuring sound, temperature, humidity, movement of the sensor device, etc. Based on the measurements of these sensors, the control unit 40 decides whether to start event detection or not.

[0172] 15 by a dashed line, the sensor device 10 may further comprise components including, for example, an accelerometer unit 50 for detecting acceleration of the sensor device 10, the first time point t1 being the time point at which a predefined acceleration pattern is detected by the accelerometer unit 50. In particular, any motion pattern normally performed by the sensor device 10 before video capture begins may be used to trigger the start of event detection.

[0173] For example, it may sense that the sensor device 10 (or an electronic device comprising the same) has been taken out of a storage position, such as being placed in a pocket or bag, or hanging from a neck strap, and placed in a stable position to focus on a scene of interest. If the sensor device 10 is then in a stable position for a long enough time, it may begin event detection even if the sensor device 10 has not received a trigger to begin capturing intensity information. Additionally or alternatively, the accelerometer unit 50 may be used to record movement of the sensor device 10 to enable the control unit 40 to remove the effects of such movement (e.g., camera shake, etc.) from event data processing and pixel signal generation / frame generation.

[0174] The sensor device may comprise a brightness sensor 65 which recognises changes in brightness and triggers event detection in response to such changes. The brightness sensor 65 may be replaced or realised by a group or small group of pixels 51 which provide intensity information to the controller 40 and triggers event detection when a predefined intensity is measured by said at least one pixel 51. Of course, the number of pixels 51 involved in this kind of brightness detection should be small enough to keep energy consumption low, for example around 1 to 10 pixels.

[0175] Additionally or alternatively, the control unit 40 may be able to recognize that a particular program that runs on the sensor device 10 (or an electronic device including the sensor device 10) and is used to control image capture is opened. When the program is opened, event detection is initiated even if an image capture instruction has not been received from a user. For example, event detection may be initiated by the control unit 40 when a camera app on a smartphone is opened.

[0176] Additionally or alternatively, the sensor device 10 may comprise an input unit 70 for receiving a user input to the sensor device 10. The first time t1 and / or the second time t2 may be a time when the user provides a predefined input to the input unit 70. The input unit 70 may take any conceivable form for interaction with a human user. For example, the input unit 70 may take the form of one or several mechanical buttons, or a touchpad or a touchscreen resembling such mechanical buttons. The input unit 70 may for example be capable of recording sounds, such as the user's voice, or of recognizing certain (hand) gestures of the user. The control unit 40 is then configured to recognize user commands received via the input unit 70 and to control the sensor device 10 accordingly. Most typically, the user directly triggers the video capture by pressing a button or a certain area on the touchpad, uttering a certain phrase, or showing a certain hand gesture. However, the event detection may also be explicitly triggered by a (different) user command.

[0177] The control unit 40 may be configured to terminate the detection of event data when the event detection rate per hour falls below a predetermined value. This is particularly beneficial in situations where the start of event detection is automatically triggered. In such situations, the user of the sensor device 10 may not be aware that event detection is being performed. If the user has no intention of capturing intensity information and places the sensor device 10 in a position where no intensity information or only static intensity information is received, such as in a pocket or bag or in a fixed position on a table, the intensity image capture is not expected to start immediately, so event detection may be switched off to save energy. This improves the possibility of energy saving, especially when combined with the automatic start of event detection, since event detection is not performed when it is not necessary.

[0178] As mentioned above, event detection can provide information of the net intensity difference between successive instances in time. Thus, the event stream can be integrated backwards in time. In principle, it is even possible to reconstruct a video sequence from the event data stream without reference to absolute intensity information. Here, we rely on the fact that we know the regularities of natural images, i.e., what the true image looks like in terms of possible intensity gradients, possible borders, etc. However, in this process, coarse image features are usually lost due to the additional invariance provided by the time difference operator (i.e., the event data stream is not subject to constant intensity changes in time).

[0179] This drawback is mitigated by using pixel signals recorded after the second time point t2, including absolute intensity information, as boundary conditions. These boundary conditions also provide image style information and therefore more natural images. Furthermore, the spatial image resolution can be improved if event detection is performed at a lower spatial resolution to reduce energy consumption.

[0180] From a methodological point of view, the absolute intensity information used as a boundary condition brings about a convex constraint on the event data stream-based reconstruction applied in the framework of compressive sensing or trained neural networks. From a mathematical point of view, the generalized reconstruction problem can be formulated as follows:

[0181] A sequence of reconstructed pixel signals Xt is obtained such that the corresponding event generation model EVSmodel applied to Xt corresponds to the actual detected event data Et.

[0182] EVSmodel(Xt) ≒ Et

[0183] Incorporating boundary conditions into the problem in the form of absolute intensity values, the above can be reformulated as follows:

[0184] EVSmodel(Xt)≈Et and X0=Y0, where Y0 represents the boundary condition.

[0185] This highlights the fact that data from regular frame-based cameras can be easily incorporated as additional or supplementary information into existing reconstruction algorithms that are based solely on event data.

[0186] FIG. 16 shows a schematic diagram of such a reconstruction of intensity values ​​by starting from pixel signals generated after the second time point t2 and interpolating these back in time on the basis of event data D detected before the second time point t2, which is compared in FIG. 17 with the situation of a reconstruction based on pure event data.

[0187] FIG. 16 shows the first two frames F1-b and F1 of a video sequence. Here, the first frame F1-b is blurred due to excessive motion during image capture or may indicate under- or overexposure. FIG. 16 also shows a stream of event data D whose capture started before the capture of the first frame F1-b. Here, the first frame F1-b is discarded by the control unit 40 as a starting point for image reconstruction due to excessive motion blur. As a boundary condition for image reconstruction, the first frame F1 that is sufficiently sharp, i.e., the first frame F1 whose degree of motion blur is below a given limit, is used. Then, frame F2 is reconstructed backward in time using the intensity differences encoded in the detected events. Here, information from the blurred image F1-b can also be used to check the interpolation results of these frames. Meanwhile, the interpolation removes the effects of motion blur, since it improves the dynamic range and reduces the delay of event generation. In this way, a series of frames F2 can be generated that show, for example, the movement of an object of interest O in the scene preceding the actual trigger of the video capture.

[0188] By adopting the true color and contrast information from the boundary condition frame F1, the reconstructed frame F2 shares the natural color and contrast and thus has maximum similarity with the actual captured image frame.

[0189] The improvement of the reconstruction result is illustrated diagrammatically by the comparative example shown in Fig. 17. Here, it is shown that the frame F3 reconstructed only from the event data stream contains in principle all the details of the captured scene, but does not provide the correct contrast or the correct colors of said scene. Therefore, with regard to a purely event-based reconstruction algorithm, it is advantageous in terms of reconstruction quality to take into account absolute intensity information and not omit this information in order to save space on the sensor chip or to further reduce energy consumption.

[0190] The control unit 40 may be configured to reconstruct the intensity values ​​based on an artificial intelligence model that receives as input pixel signals generated after the second time point t2 and event data detected before the second time point t2, and provides as output reconstructed intensity values. Thus, according to this implementation, the control unit 40 does not perform an explicit extrapolation starting from intensity information obtained after the second time point t2. Instead, the control unit 40 relies on the artificial intelligence model to "know" what reconstruction data to generate when fed a given set of captured intensity values ​​and event data.

[0191] This can be achieved by properly training an artificial intelligence model, for example using a neural network NN to estimate the time segments [0,T END ], where the NN receives as input the event data EVS for the entire time segment and the intensity information I captured after the second time point t2. cap 0 <t2<T END Then, the output is the actually captured intensity information I cap The weights of the network need to be adjusted and trained to resemble

[0192] I cap [0,T END ]=NN(EVS[0,T END ],I cap [t2,T END ])

[0193] Here, the captured intensity information I cap may be actually captured or may consist of a rendered 2D image overlaid on a moving 3D plane. From this captured intensity information, the event data EVS can be generated by an event data simulator. In this way, the input EVS[0,T END ] and I cap [t2,T END ], as well as reference I cap [0,T END ] can be generated, and the weights of the network NN can be adjusted based on different training sets to satisfy the above relationship as closely as possible (e.g., with respect to a perceptual image metric such as the Learned Perceptual Image Patch Similarity (LPIPS) metric).

[0194] Intuitively, this means that given a complete event data stream and a partial intensity value stream, the complete intensity value stream should be restored. When the generated intensity value stream matches the known intensity value stream sufficiently, the training can be terminated. Also, to achieve a continuous optimal combination of the reconstructed intensity information and the captured intensity information, the motion field from the event data stream to transfer information from the actual intensity value record as well as the mismatch of the perceptual style between the event data and the actual intensity value stream may be considered. Here, as shown in FIG. 18, a model-aware neural network with a motion field estimation block 80 may be trained while automatically transferring the perceptual style information during the training of the neural network.

[0195] 18 shows a sequence of five frames F2 centered around the Nth frame F2:N. Based on the previous frame F2:N-1 and the next frame F2:N+1, warped frames F2':N-1 and F2':N+1 are created and then fed to a motion field estimation block 80 to generate motion compensated frames, for example via a regularization network that relies on natural image regularities.

[0196] In this way, a neural network can be provided that allows for the automatic assignment of reconstructed intensity information to captured intensity information and event data streams.

[0197] In the schematic diagram of the sensor device 10 shown in Figure 15 and Figure 19a), the pixels 51 in the first subset S1 are different from the pixels 51 in the second subset S2. Thus, in this example, the event detection circuit 20 receives signals only from the first subset S1, whereas the pixel signal generation circuit 30 receives signals only from the second subset S2.

[0198] One implementation of such a division of the pixels 51 is to arrange all the pixels 51 in a single pixel array, as shown in Fig. 15 and Fig. 19a). Then, by providing a circuit as described above with respect to Fig. 13, a certain number of pixels 51 can be designed for event detection, for example for one pixel of an NxM pixel group, for example a 2x2 pixel group as shown in Fig. 15 and Fig. 19a. Then, these pixels 51 constitute the pixels 51 of a first subset S1. The remaining pixels 51 constitute the pixels 51 of a second subset S2 and can be provided with conventional circuitry for synchronous readout. Due to the sparse distribution of the pixels 51 of the first subset S1, a larger energy saving effect can be achieved during the period when only event detection is active.

[0199] However, at least some of the pixels 51 may belong to the first subset of pixels 51 as well as the second subset of pixels 51. For example, all pixels 51 of the pixel array may function as pixels 51 of the first subset S1 and the second subset S2, as shown in Fig. 19 b) in a schematic manner. The pixels 51 may then be connected to the event detection circuit 20 and the pixel signal generation circuit 30, for example as described with reference to Fig. 4. That is, each pixel 51 can be selected to transfer its photocurrent to the event detection circuit 20 or to the pixel signal generation circuit 30, for example in a time multiplexed manner. This shared pixel architecture allows to have the same spatial resolution for event detection and intensity information generation, improving the quality of the reconstructed intensity information.

[0200] In the above description, it has been assumed that the pixels 51 of both pixel subsets S1 and S2 are part of the same pixel array. However, it is also conceivable to use two different pixel arrays, each containing only the pixels 51 of one pixel subset, in different parts of a single die or on different sensor chips, as shown diagrammatically in Fig. 19 c). Nevertheless, as long as both pixel arrays observe the same scene, intensity information can be reconstructed for the pixels 51 of the second subset S2 based on events detected by the pixels 51 of the first subset S1.

[0201] In general, any geometric arrangement of pixels 51 with shared or divided features is sufficient to obtain the same scene event data and intensity information. If there is a mismatch in spatial resolution or the solid angles observed by different subsets of pixels 51 cannot be mapped one-to-one, this can in principle be resolved by using interpolation techniques to accommodate / adjust the two different data sets. Also, when using artificial intelligence models, such differences are irrelevant as long as the training data show the same differences.

[0202] Also, in the above description, it has been assumed that the pixel signals of the second subset of pixels 51 form a successive series of image / intensity frames F1 and that the reconstructed intensity values ​​form an image / intensity frame F2 preceding said successive series of image / intensity frames F1. It should be understood that this is for ease of explanation only. In principle, any set of intensity information generated after the second point in time t2 can be used together with the event data stream to reconstruct intensity information corresponding to a time prior to the second point in time t2 when event detection was already active.

[0203] Various implementations of the basic idea of ​​reconstructing earlier intensity information based on event data and later captured intensity information have been described above. The basic methodology underlying these implementations is summarized below with respect to FIG. 20.

[0204] Here, in S101, as described above, the plurality of pixels 51 of the sensor device 10 receive light and perform photoelectric conversion to generate an electric signal.

[0205] In S102, a change in intensity of the light received by each of the pixels 51 of the first subset that exceeds a predetermined threshold is detected as event data.

[0206] At S103, for each of the pixels 51 of the second subset, a pixel signal is generated indicative of the intensity value of the received light.

[0207] In S104, detection of event data is started at a first point in time t1 prior to generation of a pixel signal at a second point in time t2.

[0208] In S105, intensity values ​​are reconstructed for the second subset of pixels 51 in the period between the first point in time t1 and the second point in time t2 using pixel signals generated after the second point in time t2 and event data detected before the second point in time t2.

[0209] In this way, intensity information obtained after a given trigger can be supplemented with reconstructed intensity information preceding that trigger time.

[0210] 21 is a block diagram showing an example of a schematic configuration of a smartphone 900 that may constitute or include the above-described sensor device 10. The smartphone 900 includes a processor 901, a memory 902, a storage 903, an external connection interface 904, a camera 906, a sensor 907, a microphone 908, an input device 909, a display device 910, a speaker 911, a wireless communication interface 912, one or more antenna switches 915, one or more antennas 916, a bus 917, a battery 918, and an auxiliary controller 919.

[0211] The processor 901 may be, for example, a CPU or a system on chip (SoC), and controls functions such as an application layer of the smartphone 900. The memory 902 includes a RAM and a ROM, and stores programs and data executed by the processor 901. The storage 903 may include a storage medium such as a semiconductor memory or a hard disk. The external connection interface 904 is an interface for connecting an external device such as a memory card or a universal serial bus (USB) device to the smartphone 900. The processor may function as the control unit 40.

[0212] The camera 906 includes an image sensor as described above. The sensor 907 may include a group of sensors such as a positioning sensor, a gyro sensor, a geomagnetic sensor, and an acceleration sensor. The microphone 908 converts a voice input to the smartphone 900 into a voice signal. The input device 909 includes a touch sensor that detects a touch on the screen of the display device 910, a keypad, a keyboard, a button, a switch, and other devices, and accepts operations and information input from a user. The display device 910 includes a screen such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display, and outputs an output image of the smartphone 900. The speaker 911 converts an audio signal output from the smartphone 900 into audio.

[0213] The wireless communication interface 912 supports a cellular communication method such as LTE or LTE-Advanced, and performs wireless communication. Typically, the wireless communication interface 912 includes a BB processor 913, an RF circuit 914, and the like. The BB processor 913 may perform various signal processing for wireless communication, such as encoding / decoding, modulation / demodulation, multiplexing / demultiplexing, and the like. On the other hand, the RF circuit 914 may include components such as a mixer, a filter, an amplifier, and the like, and transmits and receives wireless signals via an antenna 916. The wireless communication interface 912 may be a one-chip module in which the BB processor 913 and the RF circuit 914 are integrated. The wireless communication interface 912 may include a plurality of BB processors 913 and a plurality of RF circuits 914. Note that, although FIG. 21 shows an example of the wireless communication interface 912 including a plurality of BB processors 913 and a plurality of RF circuits 914, the wireless communication interface 912 may include a single BB processor 913 or a single RF circuit 914.

[0214] Furthermore, the wireless communication interface 912 may support other types of wireless communication schemes, such as a short-range wireless communication scheme, a short-range wireless communication scheme, a wireless local area network (LAN) scheme, etc. In this case, the BB processor 913 and the RF circuit 914 may be included for each wireless communication scheme.

[0215] Each antenna switch 915 switches the transmission destination of an antenna 916 between multiple circuits (eg, circuits for different wireless communication methods) included in the wireless communication interface 912.

[0216] Each antenna 916 includes one or more antenna elements (for example, multiple antenna elements constituting a MIMO antenna) and is used to transmit and receive wireless signals via the wireless communication interface 912. The smartphone 900 may include multiple antennas 916 as shown in Fig. 21. Note that although Fig. 21 shows an example of the smartphone 900 including the antenna 916, the smartphone 900 may include a single antenna 916.

[0217] Furthermore, the smartphone 900 may include an antenna 916 for each wireless communication system. In this case, the antenna switch 915 may be omitted from the configuration of the smartphone 900.

[0218] The bus 917 interconnects the processor 901, the memory 902, the storage 903, the external connection interface 904, the camera 906, the sensor 907, the microphone 908, the input device 909, the display device 910, the speaker 911, the wireless communication interface 912, and the auxiliary controller 919. The battery 918 supplies power to each block of the smartphone 900 shown in Fig. 21 via a power supply line partially indicated by a dashed line in the figure. The auxiliary controller 919 operates the minimum functions of the smartphone 900, for example, in a sleep mode.

[0219] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0220] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may be achieved.

[0221] The present technology can also be configured as follows.

[0222] (1) a plurality of pixels, each configured to receive light and perform photoelectric conversion to generate an electrical signal; an event detection circuit configured to detect a change in intensity of the light received by each of the pixels of a first subset that exceeds a predetermined threshold as event data; a pixel signal generating circuit configured to generate a pixel signal indicative of an intensity value of the received light for each of the pixels of the second subset; a control unit configured to begin detecting event data at a first time point prior to generation of a pixel signal at a second time point, and to reconstruct intensity values ​​for the pixels of the second subset during a period between the first time point and the second time point using the pixel signal generated after the second time point and the event data detected before the second time point; Equipped Sensor device. (2) The sensor device according to (1), The first time point is the time point when the sensor device is switched on. Sensor device. (3) The sensor device according to (1), The first time point is a time point when the sensor device senses a predetermined condition. Sensor device. (4) The sensor device according to (3), further comprising: an accelerometer unit for detecting an acceleration of the sensor device; The first time point is a time point at which a predetermined acceleration pattern is detected by the accelerometer unit. Sensor device. (5) A sensor device according to (3) or (4), The first time point is a time point when a predetermined intensity is measured by at least one of the plurality of pixels. Sensor device. (6) The sensor device according to any one of (1) to (5), further comprising: an input unit that receives an input from a user to the sensor device; The second time point is a time point when the user provides a predetermined input to the input unit. Sensor device. (7) A sensor device according to any one of (1) to (6), The control unit is configured to terminate detection of the event data when an event detection rate per unit time falls below a predetermined value. Sensor device. (8) A sensor device according to any one of (1) to (7), The control unit is configured to reconstruct the intensity value by interpolating back in time the pixel signal based on the event data detected before the second time point, starting from the pixel signal generated after the second time point. Sensor device. (9) A sensor device according to any one of (1) to (7), The controller is configured to reconstruct the intensity values ​​based on an artificial intelligence model that receives as inputs the pixel signals generated after the second time point and the event data detected before the second time point and provides as an output the reconstructed intensity values. Sensor device. (10) A sensor device according to any one of (1) to (9), At least some of the pixels belong to the first subset of pixels and the second subset of pixels. Sensor device. (11) A sensor device according to any one of (1) to (9), The pixels of the first subset are different from the pixels of the second subset. Sensor device. (12) A sensor device according to any one of (1) to (11), the pixel signals of said second subset of pixels forming a successive series of image frames; The reconstructed intensity values ​​form a preceding image frame in the series of successive image frames. Sensor device. (13) A method for operating a sensor device according to any one of (1) to (12), comprising: A plurality of pixels of the sensor device receive light and perform photoelectric conversion to generate an electrical signal; detecting a change in intensity of the light received by each of the pixels of the first subset that exceeds a predetermined threshold as event data; generating pixel signals indicative of intensity values ​​of the light received for each of the pixels of the second subset; starting detecting event data at a first time point prior to generation of a pixel signal at a second time point; reconstructing intensity values ​​for the second subset of pixels during a period between the first and second time points using the pixel signals generated after the second time point and the event data detected before the second time point. How it works.

Claims

1. A plurality of pixels (51), each configured to receive light and perform photoelectric conversion to generate an electrical signal; an event detection circuit (20) configured to detect, as event data (D), a change in intensity of the light received by each of the pixels (51) of a first subset that exceeds a predetermined threshold; a pixel signal generating circuit (30) configured to generate a pixel signal indicative of an intensity value of the received light for each of the pixels (51) of the second subset; a control unit (40) configured to start detecting event data (D) at a first time point (t1) prior to generation of a pixel signal at a second time point (t2), and to reconstruct intensity values ​​for the second subset of pixels (51) in a period between the first time point (t1) and the second time point (t2) using the pixel signal generated after the second time point (t2) and the event data (D) detected before the second time point (t2); Equipped with A sensor device (10).

2. A sensor device (10) according to claim 1, The first time point (t1) is the time point when the sensor device (10) is switched on. A sensor device (10).

3. A sensor device (10) according to claim 1, The first time point (t1) is a time point when the sensor device (10) senses a predetermined condition. A sensor device (10).

4. The sensor device (10) according to claim 3, further comprising: The sensor device (10) includes an accelerometer unit (50) for detecting the acceleration of the sensor device (10), The first time point (t1) is a time point when a predetermined acceleration pattern is detected by the accelerometer unit (50). A sensor device (10).

5. A sensor device (10) according to claim 3, The first time point (t1) is a time point when a predetermined intensity is measured by at least one of the plurality of pixels (51). A sensor device (10).

6. The sensor device (10) according to claim 1, further comprising: The sensor device (10) includes an input unit (70) that accepts a user's input to the sensor device (10), The second time point (t2) is a time point when the user provides a predetermined input to the input unit (70). A sensor device (10).

7. A sensor device (10) according to claim 1, The control unit (40) is configured to terminate detection of the event data (D) when an event detection rate per unit time falls below a predetermined value. A sensor device (10).

8. A sensor device (10) according to claim 1, The control unit (40) is configured to reconstruct the intensity value by interpolating the pixel signal back in time based on the event data (D) detected before the second time point (t2), starting from a pixel signal generated after the second time point (t2). A sensor device (10).

9. A sensor device (10) according to claim 1, The control unit (40) is configured to reconstruct the intensity values ​​based on an artificial intelligence model that receives as input the pixel signals generated after the second time point (t2) and the event data (D) detected before the second time point (t2) and provides as output the reconstructed intensity values. A sensor device (10).

10. A sensor device (10) according to claim 1, At least a portion of the plurality of pixels (51) belong to the first subset of pixels (51) and the second subset of pixels (51). A sensor device (10).

11. A sensor device (10) according to claim 1, The pixels (51) of the first subset are different from the pixels (51) of the second subset. A sensor device (10).

12. A sensor device (10) according to claim 1, the pixel signals of said second subset of pixels (51) form a successive series of image frames (F1); The reconstructed intensity values ​​form an image frame (F2) preceding the successive series of image frames (F1). A sensor device (10).

13. A method of operating a sensor device (10), comprising: A plurality of pixels (51) of the sensor device (10) receive light and perform photoelectric conversion to generate an electrical signal; detecting a change in intensity of the light received by each of the pixels of a first subset that exceeds a predetermined threshold as event data; generating pixel signals indicative of intensity values ​​of the light received for each of the pixels of a second subset; starting detection of event data at a first time point (t1) prior to generation of a pixel signal at a second time point (t2); reconstructing intensity values ​​for the second subset of pixels (51) in a time period between the first time point (t1) and the second time point (t2) using the pixel signals generated after the second time point (t2) and the event data detected before the second time point (t2). How it works.

Images