Sensor device and method for operating a sensor device
By detecting the intensity changes of light before the video recording starts and generating event data, the sensor device can reconstruct the initial image frame, solving the problem of missing the initial part of the scene at the beginning of the video recording, and reducing energy consumption and extending the detection time.
Patent Information
- Application Number
- CN202380029709.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-31
- Filing Date
- 2023-03-14
- Publication Date
- 2024-11-08
AI Technical Summary
The sensor device is difficult to start early at the beginning of video shooting, resulting in the possibility of missing the initial part of the scene, and the sensor device consumes too much energy when detecting emergencies, resulting in the exhaustion of the device.
Design a sensor device, including a plurality of pixels and an event detection circuit, generate event data by detecting the intensity change of light before the video recording begins, and using the control unit to generate pixel signals before the event occurs, reconstructing the initial part of the video frame , reduce energy consumption.
Reconstructing the image frame before the video recording begins, reducing the energy consumption of the sensor device, extending the detection time, and avoiding the initial partial loss of the scene due to energy depletion.
Smart Images

Figure CN118923124A_ABST
Abstract
Description
Technical Field
[0001] The present technology relates to a sensor device and a method for operating the sensor device, and in particular, to a sensor device and a method for operating the sensor device that allow an image frame before a start time of camera recording to be reconstructed using an event-based vision sensor (EVS). Background Art
[0002] For example, a typical digital camera used as a stand-alone device or integrated into other electronic devices (e.g., mobile phones) includes sensor devices that usually start capturing video after a corresponding instruction from a user (e.g., pressing a button). However, when an emergency or unusual event occurs, people may want to start recording the event, but when they trigger the video start, they may have missed the initial part of the scene.
[0003] It is therefore desirable to improve the imaging capabilities of sensor devices in order to allow reconstruction of image frames prior to the start time of video recording. Summary of the invention
[0004] To this end, a sensor device is provided, comprising: a plurality of pixels, each pixel being configured to receive light and perform photoelectric conversion to generate an electrical signal; an event detection circuit configured to detect an intensity change of light received by each pixel in a first subset of pixels that is higher than a predetermined threshold as event data; and a pixel signal generation circuit configured to generate a pixel signal indicating an intensity value of light received by each pixel in a second subset of pixels. The sensor device also includes a control unit configured to start detecting event data at a first time point earlier than a second time point at which the pixel signal is generated, and to reconstruct the intensity value of the second subset of pixels for a time period between the first time point and the second time point by using the pixel signal generated after the second time point and the event data detected before the second time point.
[0005] In addition, a method for operating a sensor device is provided, the method comprising: receiving light using multiple pixels of the sensor device and performing photoelectric conversion to generate an electrical signal; detecting an intensity change of light received by each pixel in a first subset of pixels that is higher than a predetermined threshold as event data; generating a pixel signal indicating an intensity value of light received by each pixel in a second subset of pixels; starting to detect event data at a first time point earlier than a second time point at which the pixel signal is generated; and reconstructing the intensity values of the second subset of pixels in a time period between the first time point and the second time point by using the pixel signal generated after the second time point and the event data detected before the second time point.
[0006] Therefore, conventional camera pixels (a second subset of pixels) that generate a pixel signal representing the intensity of the received light are supplemented by EVS pixels (a first subset of pixels) that are sensitive to intensity changes exceeding a certain threshold. Before the full intensity signal is obtained from the camera pixel, the EVS pixel is triggered to detect an event that has occurred. Once the full intensity signal is recorded, the full intensity signal will be used together with the detected event data to reconstruct the intensity value of the camera pixel at the moment before the start of capturing the full intensity frame. In this way, the captured video can be extended forward in time. The use of EVS pixels is particularly useful in this regard because the event detection circuit consumes less energy than the pixel signal generation circuit, so that a longer duration of event detection does not excessively deplete the energy storage of the sensor device. In addition, the high temporal resolution of the event data allows correction of motion blur that may occur immediately after the start of full intensity frame capture. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 is a schematic diagram of the sensor device.
[0008] Figure 2 is a schematic block diagram of the sensor unit.
[0009] Figure 3 is a schematic block diagram of a pixel array section.
[0010] Figure 4 is a schematic circuit diagram of a pixel block.
[0011] Figure 5 is a schematic block diagram showing an event detection section.
[0012] Figure 6 It is a schematic circuit diagram of the current-voltage conversion unit.
[0013] Figure 7 It is a schematic circuit diagram of the subtraction part and the quantization part.
[0014] Figure 8 is a schematic timing diagram of an example operation of the sensor section.
[0015] Fig. 9 It is a schematic diagram of a method for generating frame data based on event data.
[0016] Fig.10 It is a schematic block diagram of another quantization unit.
[0017] Fig.11 is a schematic diagram of another event detection unit.
[0018] Fig.12 is a schematic block diagram of another pixel array section.
[0019] Fig.13 is a schematic circuit diagram of another pixel block.
[0020] Fig.14 is a schematic block diagram of a scanning type imaging device.
[0021] Fig.15 is a schematic block diagram of a sensor device and its functions.
[0022] Fig.16 is a schematic diagram of frame reconstruction.
[0023] Fig.17 is a schematic diagram of frame reconstruction according to a comparative example.
[0024] Fig.18 is a schematic diagram of frame reconstruction.
[0025] Fig.19 Different schematic block diagrams of sensor devices are shown.
[0026] Fig. 20 The process flow of a method for operating a sensor device is schematically shown.
[0027] Fig.21 is a schematic block diagram of a smart phone. DETAILED DESCRIPTION
[0028] The present disclosure is directed to mitigating problems that occur during the start of video capture. In particular, the problem of how to ensure that the sensor device starts operating early enough to allow the desired scene to be captured without excessively depleting the energy storage of the sensor device is solved. The solution to this problem discussed below is applicable to all sensor types that allow a scene to be captured by generating a full intensity frame sequence and an event data stream. However, in order to simplify the description and also to cover important application examples, the following description focuses on the specific implementation of a dynamic vision sensor / event-based vision sensor (DVS / EVS), but without any prejudice.
[0029] Therefore, first, a possible implementation of DVS / EVS will be described. This is of course purely exemplary. It should be understood that DVS / EVS may also be implemented differently.
[0030] Figure 1 The sensor chip is shown as Figure 1 2 is a diagram of a configuration example of the sensor device 10 in the example of FIG.
[0031] The sensor device 10 is a single-chip semiconductor chip including a sensor die (substrate) 11 and a logic die 12 serving as multiple dies (substrates), the sensor die and the logic die being stacked. Note that the sensor device 10 may also include only one die, or three or more stacked dies.
[0032] exist Figure 1 In the sensor device 10, the sensor die 11 includes a sensor section 21 (a circuit serving as a sensor section), and the logic die 12 includes a logic section 22. Note that the sensor section 21 may be partially formed on the logic die 12. In addition, the logic section 22 may be partially formed on the sensor die 11.
[0033] The sensor unit 21 includes pixels configured to perform photoelectric conversion on incident light to generate electrical signals, and generates event data indicating the occurrence of an event as a change in the electrical signal of the pixel. The sensor unit 21 provides the event data to the logic unit 22. That is, the sensor unit 21 performs imaging in which photoelectric conversion is performed on incident light in pixels to generate electrical signals, similar to a synchronous image sensor, for example. However, in addition to generating image data (frame data) in a frame format, the sensor unit 21 also generates event data indicating the occurrence of an event as a change in the electrical signal of the pixel. The sensor unit 21 outputs the event data obtained by imaging to the logic unit 22.
[0034] Here, the synchronous image sensor is an image sensor that is configured to perform imaging and output frame data, i.e., image data in a frame format, in synchronization with a vertical synchronization signal. Compared to the synchronous image sensor, the event monitoring function of the sensor section 21 can be considered to be asynchronous (asynchronous image sensor) because, in terms of event detection, the sensor section 21 does not operate in synchronization with the vertical synchronization signal when outputting event data.
[0035] Note that the sensor unit 21 generates and outputs frame data other than event data, similar to a synchronous image sensor. In addition, the sensor unit 21 may output the electrical signal of the pixel where the event occurred as a pixel signal (ie, the pixel value of the pixel in the frame data) together with the event data.
[0036] The logic unit 22 controls the sensor unit 21 as needed. In addition, the logic unit 22 performs various data processing, such as data processing for generating frame data based on event data from the sensor unit 21 and image processing for frame data from the sensor unit 21 or frame data generated based on event data from the sensor unit 21, and outputs data processing results obtained by performing various data processing on event data and frame data.
[0037] Figure 2 It is shown Figure 1 A block diagram of a configuration example of the sensor section 21 is shown.
[0038] The sensor section 21 includes a pixel array section 31 , a drive section 32 , an arbitrator 33 , an AD (Analog to Digital) conversion section 34 , and an output section 35 .
[0039] The pixel array section 31 preferably includes a plurality of pixels 51 ( Figure 3 ). In the case where the photocurrent (the voltage corresponding thereto) as the electrical signal generated by the photoelectric conversion in the pixel 51 undergoes a change greater than a predetermined threshold value (including a change equal to or greater than the threshold value as required), the pixel array unit 31 detects the change in the photocurrent as an event. In the case where an event is detected, the pixel array unit 31 outputs a request to the arbitrator 33 to request output of event data indicating the occurrence of the event. Then, in the case of receiving a response indicating permission for output of event data from the arbitrator 33, the pixel array unit 31 outputs the event data to the drive unit 32 and the output unit 35. In addition, the pixel array unit 31 can output the electrical signal of the pixel 51 that has detected the event as a pixel signal to the AD conversion unit 34.
[0040] The driving section 32 supplies a control signal to the pixel array section 31 to drive the pixel array section 31. For example, the driving section 32 drives the pixel 51, about which the pixel array section 31 outputs the event data, so that the relevant pixel 51 supplies (outputs) a pixel signal to the AD conversion section 34. However, the driving section 32 may also drive the pixel 51 in a synchronous manner to generate frame data.
[0041] The arbitrator 33 arbitrates requests for output of event data from the pixel array section 31 , and returns a response indicating permission or prohibition of output of event data to the pixel array section 31 .
[0042] The AD conversion unit 34 is, for example, a pixel block 41 ( Figure 3 ) includes, for example, a single slope ADC (AD converter) (not shown) in each column. The AD conversion section 34 performs AD conversion on the pixel signal of the pixel 51 of the pixel block 41 in the column using the ADC in the column, and supplies the result to the output section 35. Note that the AD conversion section 34 can perform CDS (correlated double sampling) together with the pixel signal AD conversion.
[0043] The output section 35 performs necessary processing on the pixel signal from the AD conversion section 34 and the event data from the pixel array section 31, and supplies the result to the logic section 22 ( Figure 1 ).
[0044] Here, the change in the photocurrent generated in the pixel 51 can be identified as a change in the amount of light entering the pixel 51, so it can also be said that the event is a change in the amount of light in the pixel 51 (a change in the amount of light greater than a threshold).
[0045] The event data indicating the occurrence of an event includes at least position information (coordinates, etc.) indicating the position of the pixel block where the change in light amount as the event occurred. In addition, the event data may also include the polarity (positive or negative) of the light amount change.
[0046] Regarding the series of event data output from the pixel array unit 31 at the time when an event has occurred, it can be said that as long as the event data interval is the same as the event occurrence interval, the event data implicitly includes time point information indicating the (relative) time point at which the event has occurred. However, for example, when the event data is stored in a memory and the event data interval is no longer the same as the event occurrence interval, the time point information implicitly included in the event data is lost. Therefore, before the event data interval changes from the event occurrence interval, the output unit 35 includes time point information indicating the (relative) time point at which the event occurred, such as a timestamp, in the event data. As long as the processing of including the time point information in the event data is performed before the time point information implicitly included in the event data is lost, the processing can be performed in any block outside the output unit 35.
[0047] Figure 3 It is shown Figure 2 A block diagram of a configuration example of the pixel array section 31 is shown.
[0048] The pixel array section 31 may include a plurality of pixel blocks 41. Each pixel block 41 may include I×J pixels 51 as one or more pixels arranged in I rows and J columns (I and J are integers), an event detection section 52, and a pixel signal generation section 53. One or more pixels 51 in the pixel block 41 share the event detection section 52 and the pixel signal generation section 53. In addition, in each column of the pixel block 41, a VSL (vertical signal line) for connecting the pixel block 41 to the ADC of the AD conversion section 34 is wired.
[0049] The pixel 51 receives light incident from an object and performs photoelectric conversion to generate a photocurrent serving as an electric signal. The pixel 51 supplies the photocurrent to the event detection section 52 under the control of the drive section 32 .
[0050] The event detection unit 52 detects a change in the photocurrent from each pixel 51 that is greater than a predetermined threshold as an event under the control of the drive unit 32. When an event is detected, the event detection unit 52 may report to the arbitrator 33 ( Figure 2 ) provides a request for outputting event data indicating the occurrence of an event. Then, when the event detection unit 52 receives a response indicating that the output of the requested event data is permitted from the arbitrator 33, the event detection unit 52 outputs the event data to the drive unit 32 and the output unit 35.
[0051] When the event detection unit 52 detects an event, the pixel signal generation unit 53 generates a voltage corresponding to the photocurrent from the pixel 51 as a pixel signal under the control of the drive unit 32 , and supplies the voltage to the AD conversion unit 34 through VSL.
[0052] Here, detecting a change in photocurrent greater than a predetermined threshold as an event may also be considered as detecting no change in photocurrent greater than a predetermined threshold as an event. The pixel signal generating unit 53 may generate a pixel signal in the case where no change in photocurrent greater than a predetermined threshold is detected as an event and in the case where a change in photocurrent greater than a predetermined threshold is detected as an event.
[0053] In addition, the event detection section 52 and the pixel signal generation section 53 can operate independently of each other. This means that while the event data is generated by the event detection section 52, the pixel signal generation section 53 generates the pixel signal in a synchronous manner to generate the frame data. The event detection section 52 and the pixel signal generation section 53 can also operate in a time multiplexing manner so that each section receives the photocurrent from the pixel 51 only during different time intervals.
[0054] Figure 4 is a circuit diagram showing a configuration example of the pixel block 41 . Figure 4 The configuration can be referred to as a "shared pixel configuration".
[0055] As reference Figure 3 As described above, the pixel block 41 includes pixels 51 , an event detection unit 52 , and a pixel signal generation unit 53 .
[0056] The pixel 51 includes a photoelectric conversion element 61 and transfer transistors 62 and 63 .
[0057] The photoelectric conversion element 61 includes, for example, a PD (Photodiode) The photoelectric conversion element 61 receives incident light and performs photoelectric conversion to generate electric charges.
[0058] The transfer transistor 62 includes, for example, an N (negative) type MOS (metal oxide semiconductor) FET (field effect transistor). Figure 2 ) is turned on or off by the transfer transistor 62 of the n-th pixel 51 among the I×J pixels 51 in the pixel block 41. When the transfer transistor 62 is turned on, the charge generated in the photoelectric conversion element 61 is transferred (supplied) to the event detection section 52 as a photocurrent.
[0059] The transfer transistor 63 includes, for example, an N-type MOSFET. The transfer transistor 63 of the n-th pixel 51 among the I×J pixels 51 in the pixel block 41 is turned on or off in response to the control signal TRGn supplied from the drive section 32. When the transfer transistor 63 is turned on, the charge generated in the photoelectric conversion element 61 is transferred to the FD 74 of the pixel signal generating section 53.
[0060] The I×J pixels 51 in the pixel block 41 are connected to the event detection section 52 of the pixel block 41 through the node 60. Therefore, the photocurrent generated in the pixel 51 (the photoelectric conversion element 61 of the pixel) is supplied to the event detection section 52 through the node 60. Thus, the event detection section 52 receives the sum of the photocurrents from all the pixels 51 in the pixel block 41. Therefore, the event detection section 52 detects a change in the sum of the photocurrents supplied from the I×J pixels 51 in the pixel block 41 as an event.
[0061] The pixel signal generating section 53 includes a reset transistor 71 , an amplifying transistor 72 , a selecting transistor 73 , and an FD (Floating Diffusion) 74 .
[0062] The reset transistor 71 , the amplifying transistor 72 , and the selecting transistor 73 include, for example, N-type MOSFETs.
[0063] In response to the slave drive unit 32 ( Figure 2 ) is provided, the reset transistor 71 is turned on or off. When the reset transistor 71 is turned on, the FD 74 is connected to the power supply VDD, and the charge accumulated in the FD 74 is thus released to the power supply VDD. Thus, the FD 74 is reset.
[0064] The amplifying transistor 72 has a gate connected to the FD 74, a drain connected to the power supply VDD, and a source connected to VSL through the selecting transistor 73. The amplifying transistor 72 is a source follower and outputs a voltage (electric signal) corresponding to the voltage of the FD 74 supplied to the gate to VSL through the selecting transistor 73.
[0065] The selection transistor 73 is turned on or off in response to a control signal SEL supplied from the drive section 32. When the selection transistor 73 is turned on, a voltage corresponding to the voltage of the FD 74 from the amplification transistor 72 is output to VSL.
[0066] The FD 74 accumulates the charge transferred from the photoelectric conversion element 61 of the pixel 51 through the transfer transistor 63 , and converts the charge into a voltage.
[0067] For the pixel 51 and the pixel signal generating section 53 configured as described above, the driving section 32 turns on the transfer transistor 62 using the control signal OFGn, so that the transfer transistor 62 supplies the event detecting section 52 with a photocurrent based on the charge generated in the photoelectric conversion element 61 of the pixel 51. Thus, the event detecting section 52 receives a current that is the sum of the photocurrents from all the pixels 51 in the pixel block 41, which may also be only a single pixel.
[0068] When the event detection section 52 detects a change in the photocurrent (the sum of the photocurrents) in the pixel block 41 as an event, the drive section 32 may turn off the transfer transistors 62 of all the pixels 51 in the pixel block 41, thereby stopping the supply of the photocurrent to the event detection section 52. Then, the drive section 32 may sequentially turn on the transfer transistors 63 of the pixels 51 in the pixel block 41 where the event is detected using the control signal TRGn, so that the transfer transistors 63 transfer the charge generated in the photoelectric conversion element 61 to the FD 74. The FD 74 accumulates the charge transferred from (the photoelectric conversion element 61 of) the pixel 51. The voltage corresponding to the charge accumulated in the FD 74 is output to VSL as a pixel signal of the pixel 51 through the amplification transistor 72 and the selection transistor 73.
[0069] As described above, in the sensor section 21 ( Figure 2 ), in this way, only the pixel signals of the pixels 51 in the pixel block 41 where the event is detected are sequentially output to the VSL. The pixel signals output to the VSL are provided to the AD conversion section 34 for AD conversion.
[0070] Here, in the pixels 51 in the pixel block 41, the transfer transistors 63 may be turned on simultaneously instead of sequentially. In this case, the sum of the pixel signals of all the pixels 51 in the pixel block 41 may be output.
[0071] In addition, the pixel signal generation unit 53 can also be operated independently of event detection by controlling the transfer transistor 63 without regard to event detection, or by omitting the transfer transistor 63 entirely. In this case, frame data can be generated by the pixel signal generation unit 53 in a conventional synchronous manner, and event detection can be performed simultaneously. If desired, the photocurrent transmission to the event detection unit 52 and the pixel signal generation unit 53 can be time-multiplexed so that only one of the two transfer transistors 62, 63 of each pixel 51 is turned on at a given point in time.
[0072] exist Figure 3 In the pixel array section 31, the pixel block 41 includes one or more pixels 51, and the one or more pixels 51 share the event detection section 52 and the pixel signal generation section 53. Therefore, in the case where the pixel block 41 includes a plurality of pixels 51, the number of the event detection section 52 and the pixel signal generation section 53 can be reduced compared to the case where the event detection section 52 and the pixel signal generation section 53 are provided for each pixel 51, and as a result, the scale of the pixel array section 31 can be reduced.
[0073] Note that, in the case where the pixel block 41 includes a plurality of pixels 51, the event detection section 52 may be provided for each pixel 51. In the case where a plurality of pixels 51 in the pixel block 41 share the event detection section 52, an event is detected in units of the pixel block 41. However, in the case where the event detection section 52 is provided for each pixel 51, an event may be detected in units of the pixel 51.
[0074] However, even in the case where a plurality of pixels 51 in the pixel block 41 share a single event detecting section 52 , an event can be detected in units of pixels 51 when the transfer transistors 62 of the plurality of pixels 51 are temporarily turned on in a time-division manner.
[0075] Furthermore, in the case where it is not necessary to output a pixel signal, the pixel block 41 may be formed without the pixel signal generating section 53. In the case where the pixel block 41 is formed without the pixel signal generating section 53, the sensor section 21 may be formed without the AD conversion section 34 and the transfer transistor 63. In this case, the scale of the sensor section 21 can be reduced. The sensor then outputs the address of the pixel (block) where the event occurs, with a time stamp if necessary.
[0076] Furthermore, additional pixels 51 connected to the pixel signal generation section 53 but not to the event detection section 52 may be provided on the sensor die 11. These pixels 51 may be interleaved with the pixels 51 connected to the event detection section 52, but may also form a separate pixel array for generating frame data. The separation of the event detection function and the pixel signal generation function may even be implemented as two pixel arrays on different dies that are arranged to observe the same scene. Thus, essentially any pixel arrangement / pixel circuit that allows event data to be obtained using a first subset of pixels 51 and pixel signals to be generated using a second subset of pixels 51 may be used.
[0077] Figure 5 It is shown Figure 3 A block diagram of a configuration example of the event detection section 52 is shown.
[0078] The event detection section 52 includes a current-voltage conversion section 81 , a buffer 82 , a subtraction section 83 , a quantization section 84 , and a transmission section 85 .
[0079] The current-voltage conversion section 81 converts the photocurrent (sum) from the pixel 51 into a voltage corresponding to the logarithm of the photocurrent (hereinafter also referred to as “photovoltage”), and supplies the voltage to the buffer 82 .
[0080] The buffer 82 buffers the photovoltage from the current-voltage conversion section 81 and supplies the result to the subtraction section 83 .
[0081] The subtraction unit 83 calculates the difference between the current photovoltage and the photovoltage at a time slightly shifted from the current moment at the time indicated by the row drive signal as the control signal from the drive unit 32 , and provides a differential signal corresponding to the difference to the quantization unit 84 .
[0082] The quantization section 84 quantizes the difference signal from the subtraction section 83 into a digital signal, and supplies the quantized value of the difference signal to the transmission section 85 as event data.
[0083] The transmission section 85 transmits (outputs) the event data to the output section 35 based on the event data from the quantization section 84. That is, the transmission section 85 provides a request to the arbitrator 33 to output the event data. Then, when receiving a response indicating permission of output of the requested event data from the arbitrator 33, the transmission section 85 outputs the event data to the output section 35.
[0084] Figure 6 It is shown Figure 5 A circuit diagram of a configuration example of the current-voltage conversion unit 81.
[0085] The current-voltage conversion unit 81 includes transistors 91 to 93. As the transistors 91 and 93, for example, N-type MOSFETs can be used. As the transistor 92, for example, P-type MOSFETs can be used.
[0086] The source of transistor 91 is connected to the gate of transistor 93, and a photocurrent is supplied from pixel 51 to the connection point between the source of transistor 91 and the gate of transistor 93. The drain of transistor 91 is connected to power supply VDD, and the gate thereof is connected to the drain of transistor 93.
[0087] The source of the transistor 92 is connected to the power supply VDD, and the gate thereof is connected to the connection point between the gate of the transistor 91 and the drain of the transistor 93. A predetermined bias voltage Vbias is applied to the gate of the transistor 92. With the bias voltage Vbias, the transistor 92 is turned on or off, and the operation of the current-voltage conversion section 81 is enabled or disabled depending on whether the transistor 92 is turned on or off.
[0088] The source of the transistor 93 is grounded.
[0089] In the current-voltage conversion unit 81, the drain of the transistor 91 is connected to the power supply VDD side, so the transistor is a source follower. The source of the transistor 91 as a source follower is connected to the pixel 51 ( Figure 4), so that the photocurrent based on the charge generated in the photoelectric conversion element 61 of the pixel 51 flows through the transistor 91 (from the drain to the source). The transistor 91 operates in the subthreshold region, and at the gate of the transistor 91, a photovoltage corresponding to the logarithm of the photocurrent flowing through the transistor 91 is generated. As described above, in the current-voltage conversion unit 81, the transistor 91 converts the photocurrent from the pixel 51 into a photovoltage corresponding to the logarithm of the photocurrent.
[0090] In the current-voltage conversion section 81 , the gate of the transistor 91 is connected to a connection point between the drain of the transistor 92 and the drain of the transistor 93 , and a photovoltage is output from the relevant connection point.
[0091] Figure 7 It is shown Figure 5 A circuit diagram of a configuration example of the subtraction section 83 and the quantization section 84.
[0092] The subtraction section 83 includes a capacitor 101 , an operational amplifier 102 , a capacitor 103 , and a switch 104 . The quantization section 84 includes a comparator 111 .
[0093] One end of the capacitor 101 is connected to the buffer 82 ( Figure 5 ) is connected to an output terminal of the capacitor 101, and the other end is connected to an input terminal (inverting input terminal) of the operational amplifier 102. Therefore, the photovoltage is input to the input terminal of the operational amplifier 102 through the capacitor 101.
[0094] The output terminal of the operational amplifier 102 is connected to the non-inverting input terminal (+) of the comparator 111 .
[0095] One end of the capacitor 103 is connected to the input terminal of the operational amplifier 102 , and the other end is connected to the output terminal of the operational amplifier 102 .
[0096] The switch 104 is connected to the capacitor 103 to switch the connection between both ends of the capacitor 103. The switch 104 is turned on or off in response to a row driving signal as a control signal from the driving section 32, thereby switching the connection between both ends of the capacitor 103.
[0097] When the switch 104 is turned on, the capacitor 101 is in the buffer 82 ( Figure 5 ) side is represented by Vinit, and the capacitance (electrostatic capacitance) of the capacitor 101 is represented by C1. The input terminal of the operational amplifier 102 serves as a virtual ground terminal, and the charge Qinit accumulated in the capacitor 101 when the switch 104 is turned on is represented by expression (1).
[0098] Qinit=C1×Vinit(1)
[0099] In addition, when the switch 104 is turned on, the connection between both ends of the capacitor 103 is cut off (short-circuited), so that electric charge is not accumulated in the capacitor 103.
[0100] When the switch 104 is disconnected thereafter, the capacitor 101 is in the buffer 82 ( Figure 5 ) side is represented by Vafter, when the switch 104 is turned off, the charge Qafter accumulated in the capacitor 101 is represented by expression (2).
[0101] Qafter=C1×Vafter(2)
[0102] Where the capacitance of the capacitor 103 is represented by C2 and the output voltage of the operational amplifier 102 is represented by Vout, the charge Q2 accumulated in the capacitor 103 is represented by Expression (3).
[0103] Q2=-C2×Vout(3)
[0104] Since the total amount of charge in the capacitors 101 and 103 does not change before and after the switch 104 is turned off, expression (4) holds.
[0105] Qinit=Qafter+Q2(4)
[0106] When Expression (1) to Expression (3) are substituted into Expression (4), Expression (5) is obtained.
[0107] Vout=-(C1 / C2)×(Vafter-Vinit)(5)
[0108] Using expression (5), the subtraction unit 83 subtracts the photovoltage Vinit from the photovoltage Vafter, that is, calculates the differential signal (Vout) corresponding to the difference Vafter-Vinit between the photovoltage Vafter and Vinit. Using expression (5), the subtraction gain of the subtraction unit 83 is C1 / C2. Because the maximum gain is usually expected, C1 is preferably set to a large value and C2 is preferably set to a small value. At the same time, when C2 is too small, kTC noise increases, resulting in the risk of noise characteristic degradation. Therefore, the capacitor C2 can be reduced only within the range of achieving acceptable noise. In addition, since an event detection unit 52 including a subtraction unit 83 is installed in each pixel block 41, capacitors C1 and C2 have space limitations. Taking these factors into account, the values of capacitors C1 and C2 are determined.
[0109] The comparator 111 compares the differential signal from the subtraction section 83 with a predetermined threshold value (voltage) Vth (>0) applied to the inverting input terminal (-) to quantize the differential signal. The comparator 111 outputs the quantized value obtained by quantization to the transmission section 85 as event data.
[0110] For example, when the differential signal is greater than the threshold value Vth, the comparator 111 outputs an H (high) level indicating 1 as event data indicating the occurrence of an event. When the differential signal is not greater than the threshold value Vth, the comparator 111 outputs an L (low) level indicating 0 as event data indicating that no event has occurred.
[0111] When it is confirmed that a change in the amount of light as an event has occurred based on the event data from the quantization section 84, that is, when the differential signal (Vout) is greater than the threshold value Vth, the transmission section 85 provides a request to the arbiter 33. When receiving a response indicating permission for output of event data, the transmission section 85 outputs event data indicating the occurrence of the event (e.g., H level) to the output section 35.
[0112] The output section 35 includes, in the event data from the transmission section 85, position / address information about the pixel 51 (including the pixel block 41 of the pixel) where the event indicated by the event data has occurred and time point information indicating the time point when the event has occurred, and also includes, as necessary, the polarity of the change in the amount of light as the event, that is, whether the intensity increases or decreases. The output section 35 outputs the event data.
[0113] As a data format that includes position information about the pixel 51 where an event has occurred, time point information indicating the time point when the event has occurred, and event data of the polarity of the change in the amount of light as the event, for example, a data format called "AER (Address Event Representation)" can be adopted.
[0114] Note that the gain A of the entire event detection section 52 is represented by the following expression, where the gain of the current-voltage conversion section 81 is given by CG log Indicates that, and the gain of buffer 82 is 1.
[0115] A=CG log C1 / C2(∑i photo _n)(6)
[0116] Here, i photo_ n represents the photocurrent of the n-th pixel 51 among the I×J pixels 51 in the pixel block 41. In Expression (6), Σ represents the sum of n taking integers ranging from 1 to I×J.
[0117] Note that the pixel 51 can receive any light as incident light using a filter (e.g., a color filter) through which predetermined light passes. For example, in the case where the pixel 51 receives visible light as incident light, the event data indicates the occurrence of a change in the pixel value in the image including a visible object. In addition, for example, in the case where the pixel 51 receives infrared light, millimeter waves, etc. for ranging as incident light, the event data indicates that the distance to the object has changed. In addition, for example, in the case where the pixel 51 receives infrared light for temperature measurement as incident light, the event data indicates a change in the temperature of the object. In the present embodiment, it is assumed that the pixel 51 receives visible light as incident light.
[0118] Figure 8 It is shown Figure 2 A timing chart of an example of the operation of the sensor section 21.
[0119] At time T0, the driving section 32 changes all the control signals OFGn from the L level to the H level, thereby turning on the transfer transistors 62 of all the pixels 51 in the pixel block 41. Thus, the sum of the photocurrents from all the pixels 51 in the pixel block 41 is provided to the event detection section 52. At this point, the control signals TRGn are all at the L level, and thus the transfer transistors 63 of all the pixels 51 are turned off.
[0120] For example, at time T1 , when an event is detected, the event detection section 52 outputs event data having an H level in response to the detection of the event.
[0121] At time T2, the drive section 32 sets all control signals OFGn to L level based on the event data having H level to stop the photocurrent from being supplied from the pixel 51 to the event detection section 52. In addition, the drive section 32 sets the control signal SEL to H level and sets the control signal RST to H level for a certain period of time to control the FD 74 to discharge the charge to the power supply VDD, thereby resetting the FD 74. The pixel signal generation section 53 outputs a pixel signal corresponding to the voltage of the FD 74 when the FD 74 is reset as a reset level, and the AD conversion section 34 performs AD conversion on the reset level.
[0122] At time T3 after AD conversion of the reset level, the driving section 32 sets the control signal TRG1 to the H level for a certain period of time to control the first pixel 51 in the pixel block 41 in which an event is detected (or triggered due to other reasons, such as time-division multiplexing output and / or synchronous readout of frame data), thereby transferring the charge generated by the photoelectric conversion in the first pixel 51 (the photoelectric conversion element 61 of the first pixel) to the FD 74. The pixel signal generating section 53 outputs a pixel signal corresponding to the voltage of the FD 74 to which the charge is transferred from the pixel 51 as a signal level, and the AD conversion section 34 performs AD conversion on the signal level.
[0123] The AD conversion section 34 outputs the difference between the signal level obtained after the AD conversion and the reset level to the output section 35 as a pixel signal serving as a pixel value of an image (frame data).
[0124] Here, the process of obtaining the difference between the signal level and the reset level as a pixel signal used as a pixel value of an image is referred to as "CDS". CDS may be performed after AD conversion of the signal level and the reset level, or may be performed simultaneously with AD conversion of the signal level and the reset level in the case where the AD conversion section 34 performs single slope AD conversion. In the latter case, AD conversion is performed on the signal level by using the AD conversion result of the reset level as an initial value.
[0125] At time T4 after AD conversion of the pixel signal of the first pixel 51 in the pixel block 41 , the drive section 32 sets the control signal TRG2 to H level for a certain period of time to control the second pixel 51 in the pixel block 41 where the event is detected to output the pixel signal.
[0126] In the sensor section 21 , subsequently, similar processing is performed so that pixel signals of the pixels 51 in the pixel block 41 that have detected an event are sequentially output.
[0127] When outputting pixel signals of all the pixels 51 in the pixel block 41 , the driving section 32 sets all the control signals OFGn to the H level to turn on the transfer transistors 62 of all the pixels 51 in the pixel block 41 .
[0128] Fig. 9 is a diagram illustrating an example of a frame data generation method based on event data.
[0129] The logic unit 22 sets the frame interval and the frame width, for example, based on a command input from the outside. Here, the frame interval represents the interval of frames of frame data generated based on event data. The frame width represents the time width of event data used to generate frame data in a single frame. The frame interval and frame width set by the logic unit 22 are also referred to as "set frame interval" and "set frame width", respectively.
[0130] The logic section 22 generates frame data as image data in a frame format based on the set frame interval, the set frame width, and the event data from the sensor section 21 , thereby converting the event data into the frame data.
[0131] That is, the logic section 22 generates frame data in each set frame interval based on the event data in a set frame width starting from the set frame interval.
[0132] Here, it is assumed that the event data includes time point information t indicating a time point at which the event has occurred.i (hereinafter also referred to as the "event time point") and coordinates (x, y) serving as position information about the pixel 51 (the pixel block 41 including the pixel) where the event occurred (hereinafter also referred to as the "event position").
[0133] exist Fig. 9 In a three-dimensional space (time and space) having an x-axis, a y-axis and a time axis t, points representing event data are plotted based on the event time point t and the event position (coordinates) (x, y) contained in the event data.
[0134] That is, when the position (x, y, t) in the three-dimensional space represented by the event time point t and the event position (x, y) contained in the event data is regarded as the spatiotemporal position of the event, Fig. 9 In , points representing event data are plotted at the event's spatiotemporal location (x, y, t).
[0135] The logic unit 22 starts generating frame data based on the event data at a predetermined time point (for example, when frame data generation is externally instructed or when the sensor device 10 is powered on) as the generation start time point of frame data.
[0136] Here, a rectangular parallelepiped having a set frame width in the time axis t direction in a set frame interval appearing from the generation start time point is referred to as a "frame volume". The size of the frame volume in the x-axis direction or the y-axis direction is, for example, equal to the number of pixel blocks 41 or pixels 51 in the x-axis direction or the y-axis direction.
[0137] The logic section 22 generates frame data on a single frame based on event data in a frame volume having a set frame width starting from the set frame interval in each set frame interval.
[0138] For example, frame data may be generated by setting pixels (pixel values) in a frame at an event position (x, y) included in the event data to white, and setting pixels at other positions in the frame to a predetermined color such as gray.
[0139] In addition, when the event data includes the polarity of the light amount change as an event, the frame data can be generated taking into account the polarity included in the event data. For example, in the case of positive polarity, the pixel can be set to white, and in the case of negative polarity, the pixel can be set to black.
[0140] In addition, in reference Figure 3 and Figure 4In the case where the pixel signal of the pixel 51 is also output when the event data is output, the frame data can be generated based on the event data using the pixel signal of the pixel 51. That is, the frame data can be generated by setting the pixel signal of the pixel 51 at the position (x, y) for the pixel at the event position (x, y) included in the event data (in the block corresponding to the pixel block 41) in the frame, and setting a predetermined color such as grayscale for the pixels at other positions.
[0141] Note that in the frame volume, in some cases, there are multiple event data with different event time points t but the same event position (x, y). In this case, for example, the event data at the latest or earliest event time point t can be prioritized. In addition, in the case where the event data includes polarity, the polarities of the multiple event data with different event times t but the same event position (x, y) can be added, and the pixel value based on the added value obtained by the addition can be set for the pixel at the event position (x, y).
[0142] Here, when the frame width and the frame interval are the same, the frame volumes are adjacent to each other without a gap. In addition, when the frame interval is greater than the frame width, there is a gap between the frame volumes. When the frame width is greater than the frame interval, the frame volumes are arranged to partially overlap each other.
[0143] As described above, the pixel signal generating section 53 may also generate frame data by a conventional synchronization method.
[0144] Fig.10 It is shown Figure 5 A block diagram of another configuration example of the quantization section 84.
[0145] In addition, Fig.10 In, with Figure 7 The corresponding parts are represented by the same figure marks and the description is appropriately omitted below.
[0146] exist Fig.10 , the quantization section 84 includes comparators 111 and 112 and an output section 113 .
[0147] therefore, Fig.10 The quantization unit 84 and Figure 7 The case is similar to that of including comparator 111. However, Fig.10 The quantization unit 84 and Figure 7 The case is different in that a comparator 112 and an output unit 113 are newly included.
[0148] In addition to detecting events, including Fig.10 The event detection unit 52 ( Figure 5 ) also detects the polarity of the change in light amount as an event.
[0149] In Fig.10 the quantization unit 84, when the differential signal is greater than the threshold Vth, the comparator 111 outputs an H level indicating 1 as event data indicating the occurrence of an event with a positive polarity. When the differential signal is not greater than the threshold Vth, the comparator 111 outputs an L level indicating 0 as event data indicating that an event with a positive polarity has not occurred.
[0150] In addition, in Fig.10 the quantization unit 84, a threshold Vth’ (<Vth) is supplied to the non-inverting input terminal (+) of the comparator 112, and the differential signal is supplied from the subtraction unit 83 to the inverting input terminal (-) of the comparator 112. Here, for simplicity of description, it is assumed that the threshold Vth’ is equal to -Vth, for example, but this does not have to be the case.
[0151] The comparator 112 compares the differential signal from the subtraction unit 83 with the threshold Vth’ applied to the inverting input terminal (-), thereby quantifying the differential signal. The comparator 112 outputs the quantization value obtained by quantization as event data.
[0152] For example, when the differential signal is less than the threshold Vth’ (the absolute value of the differential signal with a negative value is greater than the threshold Vth), the comparator 112 outputs an H level indicating 1 as event data indicating the occurrence of an event with a negative polarity. In addition, when the differential signal is not less than the threshold Vth’ (the absolute value of the differential signal with a negative value is not greater than the threshold Vth), the comparator 112 outputs an L level indicating 0 as event data indicating that an event with a negative polarity has not occurred.
[0153] The output unit 113 outputs, based on the event data output from the comparators 111 and 112, the event data indicating the occurrence of an event with a positive polarity, the event data indicating the occurrence of an event with a negative polarity, or the event data indicating that no event has occurred, to the transmission unit 85.
[0154] For example, when the event data from the comparator 111 is an H level indicating 1, the output unit 113 outputs +V volts indicating +1 to the transmission unit 85 as event data indicating the occurrence of an event with a positive polarity. In addition, when the event data from the comparator 112 is an H level indicating 1, the output unit 113 outputs -V volts indicating -1 to the transmission unit 85 as event data indicating the occurrence of an event with a negative polarity. Further, when the event data from each of the comparators 111 and 112 is an L level indicating 0, the output unit 113 outputs 0 volts (GND level) indicating 0 to the transmission unit 85 as event data indicating that no event has occurred.
[0155] When confirming the occurrence of a change in the amount of light as an event with a positive polarity or a negative polarity based on the event data from the output section 113 of the quantization section 84, the transmission section 85 provides a request to the arbitrator 33. After receiving a response indicating the permission of the output of the event data, the transmission section 85 outputs the event data indicating the occurrence of the event with a positive polarity or a negative polarity (+V volts indicating 1 or -V volts indicating -1) to the output section 35.
[0156] Preferably, the quantization unit 84 has Fig.10 Configuration shown.
[0157] Fig.11 is a diagram showing another configuration example of the event detection section 52 .
[0158] exist Fig.11 , the event detection section 52 includes a subtractor 430, a quantizer 440, a memory 451, and a controller 452. The subtractor 430 and the quantizer 440 correspond to the subtractor 83 and the quantizer 84, respectively.
[0159] Note that Fig.11 In the embodiment, the event detection unit 52 further includes a block corresponding to the current-voltage conversion unit 81 and the buffer 82, but in Fig.11 Illustration of these boxes is omitted.
[0160] The subtractor 430 includes a capacitor 431, an operational amplifier 432, a capacitor 433, and a switch 434. The capacitor 431, the operational amplifier 432, the capacitor 433, and the switch 434 correspond to the capacitor 101, the operational amplifier 102, the capacitor 103, and the switch 104, respectively.
[0161] The quantizer 440 includes a comparator 441. The comparator 441 corresponds to the comparator 111.
[0162] The comparator 441 compares the voltage signal (differential signal) from the subtractor 430 with a predetermined threshold voltage Vth applied to the inverting input terminal (-). The comparator 441 outputs a signal indicating the comparison result as a detection signal (quantized value).
[0163] The voltage signal from the subtractor 430 may be input to an input terminal (−) of the comparator 441 , and a predetermined threshold voltage Vth may be input to an input terminal (+) of the comparator 441 .
[0164] The controller 452 provides a predetermined threshold voltage Vth applied to the inverting input terminal (-) of the comparator 441. The provided threshold voltage Vth can be changed in a time-division manner. For example, the controller 452 provides a threshold voltage Vth1 corresponding to an ON event (e.g., a positive change in photocurrent) and a threshold voltage Vth2 corresponding to an OFF event (e.g., a negative change in photocurrent) at different times to allow a single comparator to detect multiple types of address events (events).
[0165] The memory 451 stores the output from the comparator 441 based on the sampling signal provided from the controller 452. The memory 451 may be a sampling circuit, such as a switch, plastic or capacitor, or a digital memory circuit, such as a latch or a trigger. For example, the memory 451 may maintain the comparison result of the comparator 441 using the threshold voltage Vth1 corresponding to the ON event during the period when the threshold voltage Vth2 corresponding to the OFF event is provided to the inverting input terminal (-) of the comparator 441. In addition, the memory 451 may be omitted, or may be set inside the pixel (pixel block 41), or may be set outside the pixel.
[0166] Fig.12 It is shown Figure 2 1 is a block diagram of another configuration example of the pixel array section 31.
[0167] Note that Fig.12 In, with Figure 3 The corresponding parts are represented by the same figure marks and the description is appropriately omitted below.
[0168] exist Fig.12 In FIG. 4 , the pixel array section 31 includes a plurality of pixel blocks 41 . The pixel block 41 includes I×J pixels 51 as one or more pixels and an event detection section 52 .
[0169] therefore, Fig.12 The pixel array section 31 and Figure 3 The case of is similar in that the pixel array section 31 includes a plurality of pixel blocks 41, and the pixel block 41 includes one or more pixels 51 and an event detection section 52. However, Fig.12 The pixel array section 31 and Figure 3 The case is different in that the pixel block 41 does not include the pixel signal generating section 53 .
[0170] As mentioned above, in Fig.12 In the pixel array section 31, the pixel block 41 does not include the pixel signal generating section 53, so that the sensor section 21 ( Figure 2). As described above, the pixel signal generating section 53 and the pixels 51 fed thereto may be placed on different parts of the die 11, or even on another sensor chip.
[0171] Fig.13 It is shown Fig.12 A circuit diagram of a configuration example of a pixel block 41.
[0172] As reference Fig.12 As described above, the pixel block 41 includes the pixels 51 and the event detection section 52 , but does not include the pixel signal generation section 53 .
[0173] In this case, the pixel 51 may include only the photoelectric conversion element 61 , without including the transfer transistors 62 and 63 .
[0174] Note that in pixel 51, Fig.13 In the case of the illustrated configuration, the event detection section 52 can output a voltage corresponding to the photocurrent from the pixel 51 as a pixel signal.
[0175] In the above, the sensor device 10 is described as an asynchronous imaging device configured to read out events through an asynchronous readout system. However, the event readout system is not limited to the asynchronous readout system, and may be a synchronous readout system. The imaging device to which the synchronous readout system is applied is a scanning type imaging device, which is the same as a general imaging device configured to perform imaging at a predetermined frame rate. In addition, event data detection may be performed asynchronously, while pixel signal generation may be performed synchronously.
[0176] Fig.14 is a block diagram showing a configuration example of a scanning type image forming apparatus.
[0177] like Fig.14 As shown, the imaging device 510 includes a pixel array section 521 , a driving section 522 , a signal processing section 525 , a readout region selection section 527 , and a signal generation section 528 .
[0178] The pixel array section 521 includes a plurality of pixels 530. Each of the plurality of pixels 530 outputs an output signal in response to a selection signal from the readout region selection section 527. Each of the plurality of pixels 530 may include, for example Fig.11 The plurality of pixels 530 output an output signal corresponding to the amount of change in light intensity. The plurality of pixels 530 may be arranged in a two-dimensional matrix, such as Fig.14 shown.
[0179] The driving section 522 drives the plurality of pixels 530 so that the pixels 530 output the pixel signals generated in the pixels 530 to the signal processing section 525 through the output lines 514. Note that the driving section 522 and the signal processing section 525 are circuit parts for acquiring grayscale information. Therefore, in the case of acquiring only event information (event data), the driving section 522 and the signal processing section 525 may be omitted.
[0180] The readout region selection section 527 selects some pixels from among the plurality of pixels 530 included in the pixel array section 521. For example, the readout region selection section 527 selects one or more rows included in a two-dimensional matrix structure corresponding to the pixel array section 521. The readout region selection section 527 sequentially selects one or more rows based on a preset cycle. In addition, the readout region selection section 527 may determine the selection region based on a request from the pixel 530 in the pixel array section 521.
[0181] The signal generation unit 528 generates an event signal corresponding to an active pixel in which an event is detected in the selected pixel 530 based on the output signal of the pixel 530 selected by the readout area selection unit 527. An event refers to an event in which light intensity changes. An active pixel refers to a pixel 530 in which the amount of change in light intensity corresponding to the output signal exceeds or falls below a preset threshold. For example, the signal generation unit 528 compares the output signal from the pixel 530 with a reference signal, and detects a pixel that outputs an output signal greater than or less than the reference signal as an active pixel. The signal generation unit 528 generates an event signal (event data) corresponding to an active pixel.
[0182] The signal generation section 528 may include, for example, a column selection circuit configured to arbitrate signals input to the signal generation section 528. In addition, the signal generation section 528 may output not only information about active pixels that have detected an event, but also information about inactive pixels that have not detected an event, i.e., may operate as a conventional synchronous image sensor that generates a series of consecutive image frames.
[0183] The signal generating section 528 outputs address information and time stamp information (e.g., (X, Y, T)) about active pixels that have detected an event through the output line 515. However, the data output from the signal generating section 528 may be not only address information and time stamp information but also information in a frame format (e.g., (0, 0, 1, 0, ...)).
[0184] In the above description, a sensor device 10 has been described in which event data generation and pixel signal generation can be dependent on each other or can be independent of each other. In addition, the pixel 51 can be shared between the event detection circuit and the pixel signal generation circuit by a corresponding circuit or by time multiplexing. However, the pixel 51 can also be divided so that there are event detection pixels and pixel signal generation pixels. These pixels 51 can be interwoven in the same pixel array, or can be part of different pixel arrays on the same tube core, or even arranged on different tube cores.
[0185] In all of these configurations, the event detection circuit operates on a first subset of pixels 51 and is configured to detect intensity changes above a predetermined threshold of light received by each pixel in the first subset of pixels 51 as event data; the pixel signal generation circuit operates on a second subset of pixels 51 and is configured to generate pixel signals indicating intensity values of received light for each pixel in the second subset of pixels 51. Here, at least a portion of the pixels 51 may belong to both the first subset of pixels 51 and the second subset of pixels 51, i.e., the portion is pixels 51 that share pixels 51 or whose outputs are distributed in a time-multiplexed manner. However, the pixels 51 in the first subset of pixels 51 may also be different from the pixels 51 in the second subset of pixels 51, i.e., the sensor device 10 may include pixels 51 that are divided for event detection and intensity frame generation.
[0186] Fig.15 Schematically illustrates a corresponding exemplary implementation of the sensor device 10. As described above, the sensor device 10 includes a plurality of pixels 51, each of which is configured to receive light and perform photoelectric conversion to generate an electrical signal. Here, the pixel 51 can be as described with respect to Figure 3 4. The pixel blocks 41 are arranged as described. For ease of description, it is assumed in the following without limitation that there is only one pixel 51 per pixel block 41, that is, the pixel 51 and the pixel block 41 refer to the same structural element. However, the following description can be easily summarized by understanding any reference to the pixel 51 as a reference to the pixel block 41 including several pixels 51.
[0187] The sensor device 10 comprises an event detection circuit 20, which is configured to detect a change in the intensity of light received by each pixel in the first subset S1 of pixels 51 above a predetermined threshold as event data D. The event detection circuit 20 is basically composed of an event detection section 52 as described above, which is capable of receiving the photocurrent of the pixel 51 and is capable of detecting an event due to a change in the intensity of the received light being greater than a predetermined (but possibly dynamically adjustable) event threshold. Although the event detection circuit 20 is Fig.1551, but this must be understood as being merely symbolic. As described above, the event detection section 52 of the event detection circuit 20 can be arranged close to or adjacent to the pixel 51. However, the event detection section 52 can also be grouped together in a dedicated area on the sensor chip. In particular, the event detection circuit 20 can be arranged on a different chip / die from the pixel 51 and connected vertically to the pixel 51.
[0188] The pixels 51 include a first subset S1. Fig.15 , the pixels 51 of the first subset S1 are indicated by vertical hatching. These pixels 51 are connected to the event detection circuit 20, for example as described above, and the event detection circuit 20 allows event detection of the light received by these pixels.
[0189] Pixel 51 also includes Fig.15 The second subset S2 formed by the unhatched pixels 51 in the first subset S1 is shown. The pixels 51 of the second subset S2 are used to generate full intensity information, i.e., to generate data representing the intensity of light received by the pixel 51. Here, it should be understood that appropriate color filters can be provided to each pixel 51 of the second subset S2 to allow the generation of color images. Color filters can also be provided to the pixels 51 of the first subset S1.
[0190] The sensor device 10 includes a pixel signal generating circuit 30, which is configured to generate a pixel signal indicating the intensity value of the received light of each pixel 51 in the second subset S2 of pixels 51. For example, the pixel signal generating circuit 30 can be composed of a pixel signal generating section 53 as described above, which can generate a pixel signal indicating the intensity of the received light in an asynchronous (event-triggered) or conventional synchronous manner. In particular, the pixels 51 of the second subset S2 can operate according to the known principles of an active pixel sensor APS.
[0191] Just like the event detection circuit 20, the pixel signal generation circuit 30 can also be formed in a distributed manner, where, for example, one pixel signal generation section 53 is arranged next to or close to one pixel 51 of the second subset S2, or is arranged in a dedicated area of the sensor chip. The pixel signal generation circuit 30 can also be considered to include all control lines, reset lines and / or selection lines etc. necessary to operate the sensor device 10 as an APS, which is capable of generating a continuous stream of pixel signals indicating the intensity of light received within a given period of time.
[0192] The sensor device 10 further includes a control unit 40 configured to start detecting event data D at a first time point t1 and to start generating pixel signals at a second time point t2, wherein the first time point t1 is earlier than the second time point t2. That is, during the time period between t1 and t2, only event data D is generated, and pixel signals indicating the intensity of received light are not generated.
[0193] Here, the control unit 40 can be any circuit arrangement capable of performing the above functions. For example, the control unit 40 can also be composed of a processor. The control unit 40 can be part of the pixel portion of the sensor device 10 and can be placed on the same die as the other components of the sensor device 10. However, the control unit 10 can also be arranged separately, for example, on a separate die. The functions of the control unit 10 can be implemented completely in hardware, in software, or as a mixture of hardware and software functions.
[0194] The control unit 40 is also configured to reconstruct the intensity values of the second subset S2 of the pixels 51 for the time period between the first time point t1 and the second time point t2 by using the pixel signals generated after the second time point t2 and the event data D detected before the second time point t2. Therefore, the control unit 40 uses the intensity difference information present in the event data D to reconstruct the intensity data of the pixels 51 of the second subset S2 forward in time, for example, by integration, for example, to generate data that already existed as if the generation of pixel signals by the pixels of the second subset D2 had begun before the second time point t2.
[0195] This is Fig.15 Here, event detection starts at time t1, generating an event data stream D spatially distributed over a plurality of pixels 51 of the first subset S1. Fig.15 As shown by the black and white circles, both negative and positive polarity events can be detected. At t2, an intensity data stream having a second subset S2 of pixels is captured, directed to the same scene as that observed during event data generation. Fig.15 As shown, the intensity data stream may form a series of continuous image frames F1, i.e., may be a conventional video stream. However, the pixels of the second subset S2 do not have to be arranged adjacent to each other, but may also be sparsely distributed. In this case, no complete frame is generated, but rather dot-like or sheet-like image portions. Furthermore, the intensity data may also be generated asynchronously, for example, by being triggered by an event as described above.
[0196] Using the pixel signals carrying intensity information generated after time t2, corresponding intensity information for the previous time point can be generated by referencing the event data D. In fact, since the event data D carries information indicating when the intensity change has crossed the event threshold, the intensity change of each pixel 51 of the second subset S2 can in principle be tracked if there are corresponding pixels 51 of the first subset S1 observing the same part of the captured scene.
[0197] In this way, the first intensity information obtained after time t2 can be extrapolated forward in time. Fig.15 As shown, for example, a series of intensity frames F1 taken after time t2 can be supplemented with a series of previous frames F2. The previous intensity information can in principle go back to time t1 when event detection began. The reconstructed time period can have a fixed length, for example, between 1 second and 5 seconds, or can be freely adjustable by the user of the sensor device 10.
[0198] Thus, it is possible to restore by reconstruction, for example, intensity information in the form of a video frame F2 before the capture of the intensity information (e.g., via video frame capture) began. As described above, the reconstruction is possible due to the event data from the pixels 51 of the first subset S1, which are preferably recorded continuously at least between the first time point t1 and the second time point t2.
[0199] Because the event-based data is used together with the subsequently actually recorded intensity information to reconstruct the lost intensity information forward in time, the reconstruction will have a more realistic and continuous appearance than if the intensity information were reconstructed only from the event data (i.e., without reference to the pixel signals from the second subset S2 of pixels 51).
[0200] The generation of event data inherently consumes less power than the generation of intensity information, for example because fewer pixels 51 are operated per time instance. This effect can be enhanced if fewer pixels 51 are used for event detection than for capturing intensity information. Thus, in principle, the event detection circuit 20 can be run continuously to allow for reconstruction of intensity information all the time ahead, and then be triggered when the preferred frame-based intensity information is captured. Thus, instances where time is lost at the start of a video, for example because the user cannot trigger the video capture quickly enough, can be avoided in a power-saving manner.
[0201] Here, event detection can in principle be performed at all times when the sensor device 10 is switched on, ie the first point in time t1 is the time when the sensor device 10 is switched on.
[0202] Alternatively, in order to reduce energy consumption, event detection may be turned on based on a certain trigger, i.e., the first time point t1 may also be the time when the sensor device 10 senses a predetermined condition. To this end, the sensor device 10 may include or be connected to various sensors that measure, for example, sound, temperature, humidity, movement, etc. of the sensor device. Based on the measurements of these sensors, the control unit 40 decides whether to start event detection.
[0203] For example, Fig.15 As shown in the dashed box, the sensor device 10 may include additional components, including, for example, an accelerometer unit 50 for detecting the acceleration of the sensor device 10, wherein the first time point t1 is the time when a predetermined acceleration pattern is detected by the accelerometer unit 50. In particular, any motion pattern that is usually performed with the sensor device 10 before starting to capture a video can be used to trigger the start of event detection.
[0204] For example, it can be sensed that the sensor device 10 (or an electronic device including the sensor device) is taken out of a storage location, such as in a pocket, bag, hanging on a neck strap, etc., and placed in a stable position to focus on a scene of interest. Then, once the sensor device 10 is in a stable position for a long enough time, event detection can be started even if the sensor device 10 does not receive a trigger to start capturing intensity information. Additionally or alternatively, the accelerometer unit 50 can also be used to record the movement of the sensor device 10 to allow the control unit 40 to filter out the effects of such movement (e.g., hand shaking, etc.) from event data processing and pixel signal generation / frame generation.
[0205] The sensor device may also include a brightness sensor 65 that identifies changes in brightness and triggers event detection based on such changes. The brightness sensor 65 may also be replaced or implemented by one or a small group of pixels 51 that provide intensity information to the control unit 40 and trigger event detection when a predetermined intensity is measured by the at least one pixel 51. Of course, the number of pixels 51 involved in such brightness detection needs to be small enough to keep energy consumption low, for example, about 1 to 10 pixels.
[0206] Additionally or alternatively, the control unit 40 can recognize that a specific program running on the sensor device 10 (or an electronic device including the sensor device 10) is opened for controlling image capture. Once the program is opened, event detection begins, even if an image capture instruction has not been received from the user. For example, once a camera application on a smartphone is opened, event detection can be started by the control unit 40.
[0207] Additionally or alternatively, the sensor device 10 may further include an input unit 70 for receiving input from a user of the sensor device 10. The first time point t1 and / or the second time point t2 may be the time at which the user provides a predetermined input to the input unit 70. The input unit 70 may take any conceivable form of interaction with a human user. For example, the input unit may have the form of one or more mechanical buttons or a touch pad or touch screen similar to such mechanical buttons. The input unit 70 may also be capable of recording sounds, such as the user's voice, or recognizing certain (gesture) postures of the user. The control unit 40 is then configured to recognize the user's commands received via the input unit 70 and control the sensor device 10 accordingly. Most commonly, the user will directly trigger video capture by pressing a button or a specific area on the touch pad, saying a specific phrase, or showing a specific gesture. However, event detection may also be explicitly triggered by (different) user commands.
[0208] The control unit 40 may also be configured to end the detection of event data when the event detection rate per time drops below a predetermined value. This is particularly beneficial in the case of automatically triggering the start of event detection. In this case, the user of the sensor device 10 may not be aware that event detection is running. If the user does not intend to capture intensity information and places the sensor device 10 in a position that does not receive or only receives static intensity information, for example, in a pocket or bag or in a fixed position on a table or the like, event detection can be turned off to save energy because it is not expected that intensity image capture will begin soon. This increases the energy saving potential, especially if used in conjunction with automatically starting event detection, because event detection will not be performed when it is not needed.
[0209] As mentioned above, event detection can provide information about the net intensity differences between consecutive time instances. Therefore, the event stream can be integrated forward in time. In principle, it is even possible to reconstruct a video sequence from the event data stream without reference to the absolute intensity information. Here, one relies on the regularities of natural images, i.e., on the fact that it is known how real images look like in terms of possible intensity gradients, possible border lines, etc. However, in this process, coarse image features are usually lost due to the additive invariance of the temporal difference operator (i.e., the event data stream is insensitive to time-constant intensity changes).
[0210] This drawback is mitigated by using pixel signals that carry absolute intensity information and are recorded after the second time point t2 as boundary conditions. These boundary conditions also provide a more natural image because they provide image style information. In addition, the spatial image resolution can be improved if event detection is performed at a low spatial resolution to reduce energy consumption.
[0211] From a methodological point of view, the absolute intensity information used as boundary conditions provides a convex constraint for event-based data stream reconstruction applied in the framework of compressed sensing or learning neural networks. From a mathematical point of view, the general reconstruction problem can be formulated as:
[0212] Find the sequence Xt of reconstructed pixel signals such that the corresponding event generation model EVSmodel applied to Xt corresponds to the actual detected event data Et:
[0213] EVSmodel(Xt)≈Et.
[0214] When boundary conditions in the form of absolute intensity values are included in the problem, the above can be restated as follows:
[0215] EVSmodel(Xt)≈Et and X0=Y0, where Y0 represents the boundary condition.
[0216] This highlights the fact that data from ordinary frame-based cameras can be easily included as additional or supplementary information to existing reconstruction algorithms based solely on event data.
[0217] Fig.16 A schematic diagram showing reconstruction of intensity values by starting from pixel signals generated after the second time point t2 and by temporally interpolating these pixel signals forward based on event data D detected before the second time point t2 is shown. Fig.17 The case of reconstruction based on pure event data is schematically compared in FIG.
[0218] Fig.16 The first frames F1-b and F1 of the video sequence are shown. Here, the first frame F1-b is blurred due to excessive motion during image capture, or may appear underexposed or overexposed. In addition, Fig.16 An event data stream D is shown which has already started filming before the first frame F1-b is filmed. Here, due to excessive motion blur, the first frame F1-b is discarded by the control unit 40 as the starting point for image reconstruction. As a boundary condition for image reconstruction, the first sufficiently clear frame F1 is used, i.e. the first frame F1 with a degree of motion blur below a given limit. The encoded intensity differences in the detected events are then used to reconstruct the frame F2 forward in time. Here, the information from the blurred image F1-b can also be used to check the interpolation results of these frames, while on the other hand, the interpolation will remove the effects of motion blur due to the improved dynamic range and reduced delays in event generation. In this way, a series of frames F2 can be generated showing, for example, the movement of an object of interest O in the scene before the actual triggering of the video filming.
[0219] By adopting the true color and contrast information from the boundary condition frame F1, the reconstructed frame F2 will share the natural color and contrast, and will therefore have maximum similarity to the actual captured image frame.
[0220] Depend on Fig.17 The comparative example of schematically illustrates this improvement of the reconstruction result. Here, it is shown that frame F3 reconstructed only from the event data stream will in principle contain all the details of the filmed scene, however, without giving the correct contrast or the correct colors of this scene. Therefore, with regard to the reconstruction algorithm based only on events and with regard to the quality of the reconstruction, it is advantageous if the absolute intensity information is taken into account and not omitted in order to save space on the sensor chip or even further reduce the energy consumption.
[0221] The control unit 40 may also be configured to reconstruct the intensity values based on an artificial intelligence model that receives, as input, pixel signals generated after the second time point t2 and event data detected before the second time point t2, and provides, as output, reconstructed intensity values. Thus, according to this implementation, the control unit 40 will not perform an explicit extrapolation starting from the intensity information obtained after the second time point t2. Instead, the control unit 40 will rely on an artificial intelligence model that, once fed with a given set of captured intensity values and event data, will "know" which reconstructed data to generate.
[0222] This can be achieved by properly training an artificial intelligence model. For example, a neural network NN can be used to create a time period [0, T END Here, NN has the event data EVS over the entire time period and the intensity information I captured after the second time point t2. cap , as input, where 0 <t2<T END Then the network needs to be trained by adjusting its weights so that its output is similar to the actual captured intensity information I cap :
[0223] I cap [0,T END ]=NN(EVS[0,T END ],I cap [t2,T END ])
[0224] Here, the intensity information I cap It may be a real shot or it may consist of a rendered 2D image imposed on a moving 3D plane. Based on the intensity information of the shot, the event data simulator can generate event data EVS. In this way, the input EVS [0, T END ] and I cap [t2,TEND ] and reference I cap [0,T END ], and the weights of the network NN can be adjusted based on various corresponding training sets in order to satisfy the above relationship as good as possible, for example, in terms of perceptual image metrics, as for example the learned perceptual image patch similarity (LPIPS) metric.
[0225] Intuitively, this means that given a complete event data stream and a partial intensity value stream, the complete intensity value stream should be recovered. When the generated intensity value stream matches the known intensity value stream sufficiently, the training can be terminated.
[0226] Furthermore, in order to achieve the best seamless combination of the reconstructed intensity information and the captured intensity information, the motion field of the transferred information from the event data stream to the actual intensity value records and the perceived style difference between the event data and the actual intensity value streams can be considered. Fig.18 The model of the motion field estimation block 80 shown in FIG. 8 is a perceptual neural network, and the perceptual style information can be automatically transferred during the training of the neural network.
[0227] Fig.18 A series of five frames F2 is shown centered around the Nth frame F2: N. Based on the previous frame F2: N-1 and the next frame F2: N+1, warped frames F2': N-1 and F2': N+1 are created and then fed to a motion field estimation block 80 to generate motion compensated frames, for example via a regularization network relying on natural image laws.
[0228] In this way, a neural network may be provided which allows for an automatic assignment of reconstructed intensity information to the captured intensity information and event data streams.
[0229] exist Fig.15 In the schematic diagram of the sensor device 10 given in Fig.19 As shown in a), the pixels 51 in the first subset S1 are different from the pixels 51 in the second subset S2. Therefore, although in this example, the event detection circuit 20 receives signals only from the first subset S1, the pixel signal generation unit 30 receives signals only from the second subset S2.
[0230] like Fig.15 and Fig.19 As shown in a), one implementation of such a division of pixels 51 is to arrange all pixels 51 in a single pixel array. Then, a specific number of pixels 51 can be designed for event detection, for example, by providing as described above with respect to Fig.13 The circuit is, for example, used for one pixel of an N×M pixel group, for example, one pixel of a 2×2 pixel group, such as Fig.15 and Fig.19These pixels 51 then form a first subset S1 of pixels 51. The remaining pixels 51 form a second subset S2 of pixels 51 and may be provided with conventional circuits for synchronous readout. Due to the sparse distribution of the pixels 51 of the first subset S1, greater energy saving can be achieved for periods when only event detection is effective.
[0231] However, at least a portion of the pixels 51 may belong to both the first subset of pixels 51 and the second subset of pixels 51. For example, all pixels 51 of the pixel array may be used as pixels 51 of the first subset S1 and the second subset S2. Fig.19 The pixel 51 can then be connected to the event detection circuit 20 and the pixel signal circuit 30, for example, as described above with respect to Figure 4 That is, each pixel 51 can be selectively selected to transmit its photocurrent to the event detection circuit 20 or the pixel signal generation circuit 30, for example in a time multiplexed manner. This shared pixel architecture allows to have the same spatial resolution for event detection and intensity information generation, which leads to improved quality of reconstructed intensity information.
[0232] In the above description, it is assumed that the pixels 51 of the two pixel subsets S1 and S2 are part of the same pixel array. Fig.19 As schematically shown in c), it is also conceivable to use two different pixel arrays, each containing only pixels 51 of a subset of pixels, either on different parts of a single die, or even on different sensor chips. As long as the two pixel arrays observe the same scene, it is still possible to reconstruct the intensity information of the pixels 51 of the second subset S2 based on the events detected by the pixels 51 of the first subset S1.
[0233] In general, any geometric arrangement of pixels 51 with shared or divided functionality will be sufficient as long as event data and intensity information for the same scene are available. If there is a mismatch in spatial resolution or if it is not possible to map the solid angles observed by pixels 51 in different subsets in a one-to-one manner, this will in principle be solved by adapting / aligning the two different data sets using interpolation techniques. Furthermore, if an artificial intelligence model is used, such differences will not matter as long as the training data show the same differences.
[0234] Furthermore, in the above description, it is assumed that the pixel signals of the second subset of pixels 51 form a continuous series of image / intensity frames F1, and the reconstructed intensity values form an image / intensity frame F2 preceding the continuous series of image / intensity frames F1. It should be understood that this is merely for ease of description. In principle, it would be possible to use any set of intensity information generated after the second time point t2 and the event data stream to reconstruct the corresponding intensity information for a time before the second time point t2, during which event detection was already valid.
[0235] Various implementations of the basic idea of reconstructing temporally preceding intensity information based on event data and temporally subsequent intensity information have been discussed above. Fig. 20 Summarize the basic approach of all these implementations.
[0236] Here, at S101 , as described above, light is received by the plurality of pixels 51 of the sensor device 10 and photoelectric conversion is performed to generate an electric signal.
[0237] At S102 , a change in intensity of light received by each pixel in the first subset of pixels 51 that is above a predetermined threshold is detected as event data.
[0238] At S103 , a pixel signal is generated, the pixel signal indicating the intensity value of the received light of each pixel 51 in the second subset of pixels 51 .
[0239] At S104 , event data starts to be detected earlier at a first time point t1 earlier than a second time point t2 at which a pixel signal is generated.
[0240] At S105 , intensity values of a second subset of pixels 51 for a period between the first time point t1 and the second time point t2 are reconstructed by using pixel signals generated after the second time point t2 and event data detected before the second time point t2 .
[0241] In this way, intensity information obtained after a predetermined trigger may be supplemented with reconstructed intensity information prior to the trigger time.
[0242] Fig.21 900 is a block diagram showing an example of a schematic configuration of a smart phone 900 that may constitute or include the sensor device 10, as described above. The smart phone 900 is equipped with a processor 901, a memory 902, a storage device 903, an external connection interface 904, a camera 906, a sensor 907, a microphone 908, an input device 909, a display device 910, a speaker 911, a radio communication interface 912, one or more antenna switches 915, one or more antennas 916, a bus 917, a battery 918, and an auxiliary controller 919.
[0243] The processor 901 may be, for example, a CPU or a system on chip (SoC), and controls functions in an application layer and other layers of the smartphone 900. The memory 902 includes a RAM and a ROM, and stores programs executed by the processor 901 and data. The storage device 903 may include a storage medium such as a semiconductor memory or a hard disk. The external connection interface 904 is an interface for connecting an external attachment device such as a memory card or a universal serial bus (USB) device to the smartphone 900. The processor may also function as the control unit 40.
[0244] The camera 906 includes an image sensor as described above. The sensor 907 may include a sensor group, for example, a positioning sensor, a gyroscope sensor, a geomagnetic sensor, and an acceleration sensor. The microphone 908 converts the audio input to the smart phone 900 into an audio signal. The input device 909 includes devices such as a touch sensor, a keypad, a keyboard, a button, or a switch that detects a touch on the screen of the display device 910, and receives an operation or information input from a user. The display device 910 includes a screen such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display, and displays an output image of the smart phone 900. The speaker 911 converts the audio signal output from the smart phone 900 into audio.
[0245] The radio communication interface 912 supports cellular communication schemes such as LTE or Advanced LTE, and performs radio communication. Typically, the radio communication interface 912 may include a BB processor 913, an RF circuit 914, and the like. For example, the BB processor 913 may perform processing such as encoding / decoding, modulation / demodulation, and multiplexing / demultiplexing, and perform various signal processing for radio communication. Meanwhile, the RF circuit 914 may include components such as mixers, filters, and amplifiers, and transmit or receive radio signals via an antenna 916. The radio communication interface 912 may also be a single-chip module in which the BB processor 913 and the RF circuit 914 are integrated. The radio communication interface 912 may also include multiple BB processors 913 and multiple RF circuits 91. Note that although Fig.21 An example of the radio communication interface 912 including a plurality of BB processors 913 and a plurality of RF circuits 914 is shown, but the radio communication interface 912 may also include a single BB processor 913 or a single RF circuit 914 .
[0246] In addition, in addition to the cellular communication scheme, the radio communication interface 912 can also support other types of radio communication schemes, such as a short-range wireless communication scheme, a near-field wireless communication scheme, or a wireless local area network (LAN) scheme. In this case, a BB processor 913 and an RF circuit 914 can be included for each radio communication scheme.
[0247] Each antenna switch 915 switches the destination of the antenna 916 between a plurality of circuits (eg, circuits for different radio communication schemes) included in the radio communication interface 912 .
[0248] Each antenna 916 includes a single or multiple antenna elements (e.g., multiple antenna elements constituting a MIMO antenna) and is used by the radio communication interface 912 to transmit and receive radio signals. The smartphone 900 may also include multiple antennas 916, such as Fig.21 Note that although Fig.21 An example of a smartphone 900 including a plurality of antennas 916 is shown, but the smartphone 900 may also include a single antenna 916 .
[0249] Furthermore, the smartphone 900 may also be equipped with an antenna 916 for each radio communication scheme. In this case, the antenna switch 915 may be omitted from the configuration of the smartphone 900.
[0250] The bus 917 interconnects the processor 901, the memory 902, the storage device 903, the external connection interface 904, the camera 906, the sensor 907, the microphone 908, the input device 909, the display device 910, the speaker 911, the wireless communication interface 912, and the auxiliary controller 919. The battery 918 is connected to the power supply line via a power line partially shown in dashed lines in the figure. Fig.21 The various blocks of the smartphone 900 are shown to be powered. For example, in the sleep mode, the auxiliary controller 919 enables the minimum functions of the smartphone 900 to operate.
[0251] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.
[0252] Furthermore, the effects described herein are merely exemplary and not restrictive, and other effects may be provided.
[0253] Note that the present technology can also adopt the following configurations.
[0254] 1. A sensor device, comprising:
[0255] a plurality of pixels, each pixel being configured to receive light and perform photoelectric conversion to generate an electrical signal;
[0256] event detection circuitry configured to detect, as event data, a change in intensity of light received by each pixel in the first subset of pixels that is above a predetermined threshold;
[0257] a pixel signal generating circuit configured to generate a pixel signal indicative of an intensity value of received light for each pixel in the second subset of pixels;
[0258] A control unit is configured to start detecting event data at a first time point earlier than a second time point at which a pixel signal is generated, and to reconstruct intensity values of a second subset of pixels in a time period between the first time point and the second time point by using the pixel signal generated after the second time point and the event data detected before the second time point.
[0259] 2. The sensor device according to 1, wherein:
[0260] The first point in time is when the sensor device is switched on.
[0261] 3. The sensor device according to 1, wherein:
[0262] The first time point is the time when the sensor device senses the predetermined condition.
[0263] 4. The sensor device according to 3, further comprising:
[0264] an accelerometer unit for detecting acceleration of the sensor device; and
[0265] The first time point is the time when a predetermined acceleration pattern is detected by the accelerometer unit.
[0266] 5. The sensor device according to 3 or 4, wherein:
[0267] The first time point is the time when a predetermined intensity is measured by at least one pixel.
[0268] 6. The sensor device according to any one of 1 to 5, further comprising:
[0269] An input unit is used to receive input from a user of the sensor device; wherein,
[0270] The second time point is a time when the user provides a predetermined input to the input unit.
[0271] 7. The sensor device according to any one of 1 to 6, wherein:
[0272] The control unit is configured to end the detection of event data when the event detection rate per time drops below a predetermined value.
[0273] 8. The sensor device according to any one of 1 to 7, wherein:
[0274] The control unit is configured to reconstruct the intensity value by starting from a pixel signal generated after the second point in time and by temporally interpolating the pixel signal forward based on event data detected before the second point in time.
[0275] 9. The sensor device according to any one of 1 to 7, wherein:
[0276] The control unit is configured to reconstruct the intensity value based on an artificial intelligence model, which receives as input the pixel signal generated after the second time point and the event data detected before the second time point, and provides a reconstructed intensity value as an output.
[0277] 10. The sensor device according to any one of 1 to 9, wherein:
[0278] At least a portion of the pixels belong to the first subset of pixels and the second subset of pixels.
[0279] 11. The sensor device according to any one of 1 to 9, wherein:
[0280] The pixels in the first subset of pixels are different from the pixels in the second subset of pixels.
[0281] 12. The sensor device according to any one of 1 to 11, wherein:
[0282] The pixel signals of the second subset of pixels form a continuous series of image frames; and
[0283] The reconstructed intensity values form an image frame preceding a consecutive series of image frames.
[0284] 13. A method for operating a sensor device according to any one of 1 to 12, the method comprising:
[0285] receiving light using a plurality of pixels of a sensor device and performing photoelectric conversion to generate an electrical signal;
[0286] detecting, as event data, a change in intensity of light received by each pixel in the first subset of pixels that is above a predetermined threshold;
[0287] generating a pixel signal indicating an intensity value of received light for each pixel in the second subset of pixels;
[0288] Starting detecting event data at a first point in time earlier than generating a pixel signal at a second point in time; and
[0289] Intensity values of a second subset of pixels are reconstructed for a time period between the first time point and the second time point using pixel signals generated after the second time point and event data detected before the second time point.
Claims
1. A sensor device (10), comprising: a plurality of pixels (51), each pixel being configured to receive light and perform photoelectric conversion to generate an electrical signal; an event detection circuit (20) configured to detect a change in intensity of the light received by each pixel in the first subset of pixels (51) above a predetermined threshold as event data (D); A pixel signal generating circuit (30) configured to generate a pixel signal indicating an intensity value of received light for each pixel (51) in the second subset of pixels (51); A control unit (40) is configured to start detecting event data (D) at a first time point (t1) earlier than a second time point (t2) at which a pixel signal is generated, and to reconstruct intensity values of the second subset of pixels (51) in a time period between the first time point (t1) and the second time point (t2) by using the pixel signal generated after the second time point (t2) and the event data (D) detected before the second time point (t2).
2. The sensor device (10) according to claim 1, wherein: The first time point (t1) is the time when the sensor device (10) is switched on.
3. The sensor device (10) according to claim 1, wherein: The first time point (t1) is the time when the sensor device (10) senses a predetermined condition.
4. The sensor device (10) according to claim 3, further comprising: an accelerometer unit (50) for detecting the acceleration of the sensor device (10); as well as The first time point (t1) is the time when a predetermined acceleration pattern is detected by the accelerometer unit (50).
5. The sensor device (10) according to claim 3, wherein: The first time point (t1) is the time when a predetermined intensity is measured by at least one of the pixels (51).
6. The sensor device (10) according to claim 1, further comprising: An input unit (70) is used to receive input from a user of the sensor device (10); wherein: The second time point (t2) is the time when the user provides a predetermined input to the input unit (70).
7. The sensor device (10) according to claim 1, wherein: The control unit (40) is configured to terminate the detection of event data when the event detection rate per time drops below a predetermined value.
8. The sensor device (10) according to claim 1, wherein The control unit (40) is configured to reconstruct the intensity value by starting from a pixel signal generated after the second time point (t2) and by temporally interpolating the pixel signal forward based on the event data detected before the second time point (t2).
9. The sensor device (10) according to claim 1, wherein: The control unit (40) is configured to reconstruct the intensity value based on an artificial intelligence model, which receives the pixel signal generated after the second time point (t2) and the event data detected before the second time point (t2) as input, and provides a reconstructed intensity value as output.
10. The sensor device (10) according to claim 1, wherein: At least a portion of the pixels (51) belongs to the first subset of pixels (51) and the second subset of pixels (51).
11. The sensor device (10) according to claim 1, wherein: The pixels (51) in the first subset of the pixels (51) are different from the pixels in the second subset of the pixels (51).
12. The sensor device (10) according to claim 1, wherein: The pixel signals of the second subset of the pixels (51) form a continuous series of image frames (F1); as well as The reconstructed intensity values form an image frame (F2) preceding said continuous series of image frames (F1).
13. A method for operating a sensor device (10), the method comprising: Utilizing a plurality of pixels (51) of the sensor device (10) to receive light and perform photoelectric conversion to generate an electrical signal; detecting a change in intensity of the light received by each pixel in the first subset of pixels (51) above a predetermined threshold as event data; generating a pixel signal indicating an intensity value of received light for each pixel (51) in the second subset of pixels (51); Starting detecting event data at a first time point (t1) earlier than a second time point (t2) at which the pixel signal is generated; as well as The intensity values of the second subset of pixels (51) in the time period between the first time point (t1) and the second time point (t2) are reconstructed by using the pixel signal generated after the second time point (t2) and the event data detected before the second time point (t2).
