Sensor device and method for operating a sensor device
By combining event detection and pixel signal generation technology in sensor devices, information in infrared light is extracted, and the problems of real-time three-dimensional model reconstruction and additional information capture in the prior art are solved, and the functions and efficiency of video conferencing and monitoring tasks are improved.
Patent Information
- Application Number
- CN202380029711.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-30
- Filing Date
- 2023-03-01
- Publication Date
- 2024-11-15
AI Technical Summary
The prior art is difficult to capture and process three-dimensional models in real time, so fast three-dimensional virtualization cannot be achieved in applications such as video conferencing, and sensor devices cannot receive additional information other than RGB video streams, limiting the functions of monitoring and monitoring tasks.
Design a sensor device, using multiple pixels and event detection circuit systems, can detect events when the threshold value of infrared light changes, and generate visible light intensity signals through the pixel signal generation circuit system, thereby extracting additional information in infrared light and realizing the Real-time reconstruction of 3D models and capture of additional information.
It realizes the enhancement of extracting additional information from RGB video streams, supports real-time three-dimensional model reconstruction and monitoring tasks, and improves the speed and efficiency of data processing.
Smart Images

Figure CN118975261A_ABST
Abstract
Description
Technical Field
[0001] The present technology relates to a sensor device and a method for operating the sensor device, and in particular, to a sensor device and a method for operating the sensor device that allow capturing a visible light video stream as well as additional information provided via infrared light. Background Art
[0002] Conventional digital cameras are used, for example, as independent sensor devices or integrated into other electronic devices (such as mobile phones) to capture video frames showing intensity values of visible light, for example, by using red R filters, green G filters, blue B filters, etc. For example, such RGB videos are captured to be transmitted to other electronic devices and displayed thereon, for example, for video conferencing, for surveillance, for monitoring of industrial or logistic processes, for computer games, etc.
[0003] The frames captured in this way display a two-dimensional image, i.e., the observed three-dimensional scene projected onto the sensitive surface of the corresponding sensor device. Often, it is desirable to generate a three-dimensional model of the observed scene from the captured two-dimensional image. For example, in video conferencing applications, it may be desirable to correctly subtract the background of the video conference participants, to animate avatars / animated expressions representing the video conference participants, or to correct the viewpoints of the video conference participants, e.g., to make it resemble the impression that the members are looking into each other's eyes. All of these functions can be achieved by reconstructing a three-dimensional model of at least the heads of the video conference participants.
[0004] While it is possible in principle to generate a 3D model of the observed scene from the captured 2D images, such 3D virtualization usually requires too much computing power in practice. Furthermore, virtualization based on RGB image frames is usually too slow to convert a 2D image stream into a 3D model of the observed scene in real time.
[0005] On the other hand, it is known to generate a depth map indicating the distance of each part of the scene to the sensor device, for example by using a structured light method, wherein the imaging distortion of a predetermined and known pattern projected onto the scene allows to infer the distances of visible objects in the scene. Using such a depth map, the desired three-dimensional model of the observed scene can be generated.
[0006] But here the problem arises that computing the depth map from the frame images is time consuming and therefore not suitable for generating real-time 3D virtualization of the scene. Furthermore, in applications such as video conferencing, visible light cannot be used for depth map generation via structured light methods, because the projected light pattern would be visible in the exchanged video stream.
[0007] Therefore, while methods for depth reconstruction can be used in principle, these methods are not directly implementable if the captured RGB images are not degraded.
[0008] Additionally, it would generally be helpful if the imaging capabilities of the sensor device enabled the reception of additional information in addition to the captured RGB video stream, without degradation of the captured RGB image. This could be helpful for monitoring and surveillance tasks where the capture of certain processes or objects would trigger the automatic reception of additional information in the image data.
[0009] It would therefore be desirable to provide a sensor device that allows for the capture of image data from which an interference-free RGB video stream as well as additional data can be retrieved. Summary of the invention
[0010] To this end, a sensor device for observing a scene is provided, comprising: a plurality of pixels, each pixel being configured to receive light from the scene and perform photoelectric conversion to generate an electrical signal; an event detection circuit system, which is configured to detect when an intensity change of event data exceeds a predetermined threshold of infrared light received by each of the pixels of a first subset; a pixel signal generation circuit system, which is configured to generate a pixel signal indicating an intensity value of visible light received by each of the pixels of a second subset; and a control unit, which is configured to extract additional information from the event data detected in the received infrared light, the additional information being different from the two-dimensional intensity information about the observed scene.
[0011] In addition, a method for operating a sensor device for observing a scene is provided, the method comprising: receiving light from the scene and performing photoelectric conversion to generate an electrical signal using each of a plurality of pixels of the sensor device; detecting, using an event detection circuit system of the sensor device, when an intensity change in event data exceeds a predetermined threshold of infrared light received by each of a first subset of pixels; generating, using a pixel signal generation circuit system, a pixel signal indicating an intensity value of visible light received by each of a second subset of pixels; and extracting additional information from the event data detected within the received infrared light, the additional information being different from two-dimensional intensity information about the observed scene.
[0012] Therefore, conventional camera pixels (pixels of the second subset) (also called active pixel sensor APS pixels) that generate pixel signals representing the intensity of received light are complemented with dynamic / event-based vision sensor DVS / EVS pixels (pixels of the first subset) that are sensitive to intensity changes exceeding a certain threshold (which are also called events). While camera pixels generate pixel signals representing received intensity in the visible spectrum, EVS pixels operate on infrared light, i.e., they are sensitive to intensity changes of received infrared light. The sensor device is able to view and extract additional information encoded in infrared light from the detected event stream.
[0013] Thus, in addition to the functionality of inferring the time series of the two-dimensional intensity distribution of the infrared portion of the electromagnetic spectrum, the sensor device is also able to filter out additional information in addition to the two-dimensional intensity distribution. Specifically, infrared structured light can be used to allow reconstruction of a depth map based on detected events. Alternatively, messages sent by infrared emitters visible on the scene can be decoded. Since the infrared spectrum is used to encode this additional information directed to the sensor device, the visible spectrum observed by the APS pixels is not affected by this information.
[0014] Additionally, because the data rate generated by event detection is much lower than the data rate generated for full-intensity frame detection, the EVS pixel ensures that the additional information is available quickly enough to supplement the full-intensity pixel signal with the additional information in real time. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a schematic diagram of the sensor device.
[0016] Figure 2 is a schematic block diagram of the sensor part.
[0017] Figure 3 is a schematic block diagram of the pixel array portion.
[0018] Figure 4 is a schematic circuit diagram of a pixel block.
[0019] Figure 5 is a schematic block diagram showing an event detection part.
[0020] Figure 6 It is a schematic circuit diagram of the current-voltage conversion part.
[0021] Figure 7 It is a schematic circuit diagram of the subtraction part and the quantization part.
[0022] Figure 8 is a schematic timing diagram of an exemplary operation of the sensor portion.
[0023] Fig. 9 It is a schematic diagram of a method for generating frame data based on event data.
[0024] Fig.10 It is a schematic block diagram of another quantization part.
[0025] Fig.11 is a schematic diagram of another event detection part.
[0026] Fig.12 is a schematic block diagram of another pixel array portion.
[0027] Fig.13 is a schematic circuit diagram of another pixel block.
[0028] Fig.14 is a schematic block diagram of a scanning type sensor device.
[0029] Fig.15 is a schematic block diagram of a sensor device.
[0030] Fig.16 is a schematic illustration of a signal encoded in infrared light received by a sensor device.
[0031] Fig.17 is a schematic block diagram of another sensor device.
[0032] Fig.18 yes Fig.17 Schematic illustration of the functionality of a sensor device.
[0033] Fig.19 yes Fig.17 Functional block diagram of the sensor device.
[0034] Fig. 20 is a schematic illustration of the viewpoint adjustment process.
[0035] Fig.21 is a schematic illustration of the structure of a sensor device.
[0036] Fig. 22 is a schematic illustration of the structure of another sensor device.
[0037] Fig.23 is a schematic illustration of the structure of another sensor device.
[0038] Fig.24 is a schematic illustration of the structure of another sensor device.
[0039] Fig.25 The process flow of a method for operating a sensor device is schematically shown.
[0040] Fig.26 is a schematic block diagram of a smartphone. DETAILED DESCRIPTION
[0041] The present disclosure is directed to alleviating a problem that arises when providing information about a captured scene to a sensor device that goes beyond just the two-dimensional intensity distribution observed by the sensor device. Such information may, for example, allow real-time conversion of the two-dimensional intensity distribution into a three-dimensional model of the observed scene, or may consist of messages directed to the sensor device. The solution to this problem discussed below is applicable to all sensor types that allow a scene to be captured by generating an event data stream based on infrared light as well as a full intensity frame sequence based on visible light.
[0042] Here, in order to clarify its function and also to cover important application examples, we first Figures 1 to 14 An event-based vision sensor (EVS) is generally described. This general overview focuses on the possibility of sharing pixels between EVS circuitry and APS circuitry, such as to create a hybrid EVS-APS device. Fig.15 Let's start discussing more specific concepts for solutions to the above problems.
[0043] Of course, it has to be understood that the following description is purely exemplary and that the (hybrid) EVS may also be implemented differently.
[0044] Figure 1 is a schematic diagram showing a configuration example of the sensor device 10. Figure 1 In the example, the sensor device is composed of a sensor chip.
[0045] The sensor device 10 is a single-chip semiconductor chip, and includes a sensor die (substrate) 11 serving as a plurality of dies (substrates) and a stacked logic die 12. Note that the sensor device 10 may also include only a single die or three or more stacked dies.
[0046] exist Figure 1 In the sensor device 10, the sensor die 11 includes a sensor portion 21 (a circuit serving as a sensor portion), and the logic die 12 includes a logic portion 22. Note that the sensor portion 21 may be partially formed on the logic die 12. In addition, the logic portion 22 may be partially formed on the sensor die 11.
[0047] The sensor portion 21 includes pixels configured to perform photoelectric conversion on incident light to generate electrical signals, and generates event data indicating the occurrence of an event as a change in the electrical signals of the pixels. The sensor portion 21 provides the event data to the logic portion 22. That is, similar to a synchronous image sensor, for example, the sensor portion 21 performs imaging by performing photoelectric conversion on incident light in pixels to generate electrical signals. However, in addition to generating image data (frame data) in a frame format, the sensor portion 21 also generates event data indicating the occurrence of an event as a change in the electrical signals of the pixels. The sensor portion 21 outputs the event data obtained by imaging to the logic portion 22.
[0048] Here, the synchronous image sensor is an image sensor configured to perform imaging in synchronization with a vertical synchronization signal and output frame data of image data in a frame format. Compared with the synchronous image sensor, the event detection function of the sensor portion 21 can be regarded as asynchronous (asynchronous image sensor) because, in terms of event detection, the sensor portion 21 does not operate in synchronization with the vertical synchronization signal when outputting event data.
[0049] Note that, similar to the synchronous image sensor, the sensor section 21 generates and outputs frame data in addition to event data. In addition, the sensor section 21 may output an electrical signal of a pixel in which an event has occurred together with the event data as a pixel signal, which is a pixel value of a pixel in the frame data.
[0050] The logic part 22 controls the sensor part 21 as needed. In addition, the logic part 22 performs various types of data processing, such as data processing for generating frame data based on event data from the sensor part 21 and image processing of the frame data from the sensor part 21 or the frame data generated based on the event data from the sensor part 21, and outputs data processing results obtained by performing various types of data processing on the event data and the frame data.
[0051] Figure 2 It shows Figure 1 A block diagram of a configuration example of the sensor portion 21 is shown.
[0052] The sensor section 21 includes a pixel array section 31 , a drive section 32 , an arbiter 33 , an AD (Analog-to-Digital) conversion section 34 , and an output section 35 .
[0053] The pixel array section 31 preferably includes a plurality of pixels 51 ( Figure 3 ). In the case where a change greater than a predetermined threshold value (including a change equal to or greater than the threshold value as required) has occurred in the photocurrent (the voltage corresponding to the photocurrent) as an electrical signal generated by photoelectric conversion in the pixel 51, the pixel array part 31 detects the change in the photocurrent as an event. In the case where an event is detected, the pixel array part 31 outputs a request for requesting output of event data indicating the occurrence of the event to the arbitrator 33. Then, in the case of receiving a response indicating permission for output of event data from the arbitrator 33, the pixel array part 31 outputs the event data to the drive part 32 and the output part 35. In addition, the pixel array part 31 can output the electrical signal of the pixel 51 in which the event has been detected to the AD conversion part 34 as a pixel signal.
[0054] The driving section 32 supplies a control signal to the pixel array section 31 to drive the pixel array section 31. For example, the driving section 32 drives the pixel array section 31 to have the pixel 51 that outputs event data so that the pixel 51 in question supplies (outputs) a pixel signal to the AD conversion section 34. However, the driving section 32 may also drive the pixel 51 in a synchronous manner to generate frame data.
[0055] The arbitrator 33 arbitrates a request from the pixel array section 31 for outputting event data, and returns a response indicating event data output permission or prohibition to the pixel array section 31 .
[0056] For example, the AD conversion section 34 converts the pixel block 41 ( Figure 3 ) includes, for example, a single slope ADC (AD converter) (not shown) in each column. The AD conversion section 34 performs AD conversion on the pixel signal of the pixel 51 of the pixel block 41 in the column using the ADC in the column, and provides the result to the output section 35. Note that the AD conversion section 34 can perform CDS (Correlated Double Sampling) together with the pixel signal AD conversion.
[0057] The output section 35 performs necessary processing on the pixel signal from the AD conversion section 34 and the event data from the pixel array section 31, and outputs the pixel signal to the logic section 22 ( Figure 1 ) to provide the results.
[0058] Here, the change in the photocurrent generated in the pixel 51 can be identified as a change in the amount of light entering the pixel 51, so that it can also be said that the event is a change in the amount of light in the pixel 51 (a change in the amount of light greater than a threshold).
[0059] Event data indicating event occurrence includes at least position information (coordinates, etc.) indicating the position of a pixel block where a change in light amount as an event has occurred. In addition, the event data may also include the polarity (positive or negative) of the change in light amount.
[0060] Regarding a series of event data output from the pixel array portion 31 at the moment when an event has occurred, it can be said that as long as the event data interval is the same as the event occurrence interval, the event data implicitly includes time point information indicating the (relative) time point at which the event has occurred. However, for example, when the event data is stored in a memory and the event data interval is no longer the same as the event occurrence interval, the time point information implicitly included in the event data is lost. Therefore, the output portion 35 includes time point information in the event data, which indicates the (relative) time point at which the event has occurred before the event data interval changes from the event occurrence interval, such as a timestamp. The processing of including time point information in the event data can be performed in any block other than the output portion 35, as long as the processing is performed before the time point information implicitly included in the event data is lost.
[0061] Figure 3 It shows Figure 2 A block diagram of a configuration example of the pixel array section 31 is shown.
[0062] The pixel array section 31 may include a plurality of pixel blocks 41. Each pixel block 41 may include I×J pixels 51, which are one or more pixels arranged in I rows and J columns (I and J are integers), an event detection section 52, and a pixel signal generation section 53. One or more pixels 51 in the pixel block 41 share the event detection section 52 and the pixel signal generation section 53. In addition, in each column of the pixel block 41, a VSL (vertical signal line) for connecting the pixel block 41 to the ADC of the AD conversion section 34 is wired.
[0063] The pixel 51 receives light incident from an object and performs photoelectric conversion to generate a photocurrent serving as an electric signal. The pixel 51 supplies the photocurrent to the event detection section 52 under the control of the drive section 32 .
[0064] The event detection section 52 detects a change in the photocurrent from each of the pixels 51 that is greater than a predetermined threshold as an event under the control of the drive section 32. When an event is detected, the event detection section 52 may report to the arbitrator 33 ( Figure 2 ) provides a request for requesting output of event data indicating the occurrence of an event. Then, when receiving a response indicating permission of output of the requested event data from the arbitrator 33, the event detection section 52 outputs the event data to the drive section 32 and the output section 35.
[0065] In the case where the event detection section 52 has detected an event, the pixel signal generation section 53 generates a voltage corresponding to the photocurrent from the pixel 51 as a pixel signal, and supplies the voltage to the AD conversion section 34 through VSL under the control of the drive section 32 .
[0066] Here, detecting a change in the (log) photocurrent greater than a predetermined threshold as an event may also be identified as detecting the absence of a change in the photocurrent greater than a predetermined threshold as an event. The pixel signal generating portion 53 may generate a pixel signal in the case where the absence of a change in the photocurrent greater than a predetermined threshold has been detected as an event and in the case where a change in the photocurrent greater than a predetermined threshold has been detected as an event.
[0067] In addition, the event detection section 52 and the pixel signal generation section 53 can operate independently of each other. This means that while event data is generated by the event detection section 52, a pixel signal is generated by the pixel signal generation section 53 in a synchronized manner to generate frame data. The event detection section 52 and the pixel signal generation section 53 can also operate in a time multiplexed manner so that each of the sections receives a photocurrent from the pixel 51 only during different time intervals.
[0068] Figure 4 is a circuit diagram showing a configuration example of the pixel block 41 . Figure 4The configuration can be called a "shared pixel configuration".
[0069] As reference Figure 3 As described, the pixel block 41 includes pixels 51 , an event detection section 52 , and a pixel signal generation section 53 .
[0070] The pixel 51 includes a photoelectric conversion element 61 and transfer transistors 62 and 63 .
[0071] The photoelectric conversion element 61 includes, for example, a PD (Photo Diode). The photoelectric conversion element 61 receives incident light and performs photoelectric conversion to generate electric charges.
[0072] The transfer transistor 62 includes, for example, an N (negative) type MOS (metal oxide semiconductor) FET (field effect transistor). The transfer transistor 62 of the n-th pixel 51 among the I×J pixels 51 in the pixel block 41 responds to the transmission from the drive section 32 ( Figure 2 ) is turned on or off by a control signal OFGn supplied from the photoelectric conversion element 61. When the transfer transistor 62 is turned on, the charge generated in the photoelectric conversion element 61 is transferred (supplied) to the event detection section 52 as a photocurrent.
[0073] The transfer transistor 63 includes, for example, an N-type MOSFET. The transfer transistor 63 of the n-th pixel 51 among the I×J pixels 51 in the pixel block 41 is turned on or off in response to the control signal TRGn supplied from the driving section 32. When the transfer transistor 63 is turned on, the charge generated in the photoelectric conversion element 61 is transferred to the FD 74 of the pixel signal generating section 53.
[0074] The I×J pixels 51 in the pixel block 41 are connected to the event detection section 52 of the pixel block 41 through the node 60. Therefore, the photocurrent generated in (the photoelectric conversion element 61 of) the pixel 51 is supplied to the event detection section 52 through the node 60. Therefore, the event detection section 52 receives the sum of the photocurrents from all the pixels 51 in the pixel block 41. Therefore, the event detection section 52 detects a change in the sum of the photocurrents supplied from the I×J pixels 51 in the pixel block 41 as an event.
[0075] The pixel signal generating section 53 includes a reset transistor 71 , an amplifying transistor 72 , a selecting transistor 73 , and an FD (Floating Diffusion) 74 .
[0076] The reset transistor 71 , the amplifying transistor 72 , and the selecting transistor 73 include, for example, N-type MOSFETs.
[0077] The reset transistor 71 responds to the slave driving section 32 ( Figure 2) is turned on or off by a control signal RST provided by . When the reset transistor 71 is turned on, the FD 74 is connected to the power supply VDD, and thus the charge accumulated in the FD 74 is discharged to the power supply VDD. Thus, the FD 74 is reset.
[0078] The amplifier transistor 72 has a gate connected to the FD 74, a drain connected to the power supply VDD, and a source connected to VSL through the selection transistor 73. The amplifier transistor 72 is a source follower and outputs a voltage (electric signal) corresponding to the voltage of the FD 74 supplied to the gate to VSL through the selection transistor 73.
[0079] The selection transistor 73 is turned on or off in response to a control signal SEL supplied from the driving section 32. When the selection transistor 73 is turned on, a voltage corresponding to the voltage of the FD 74 from the amplification transistor 72 is output to VSL.
[0080] The FD 74 accumulates the charge transferred from the photoelectric conversion element 61 of the pixel 51 through the transfer transistor 63 , and converts the charge into a voltage.
[0081] With respect to the pixel 51 and the pixel signal generating section 53 configured as described above, the driving section 32 turns on the transfer transistor 62 using the control signal OFGn so that the transfer transistor 62 supplies the event detecting section 52 with a photocurrent based on the charge generated in the photoelectric conversion element 61 of the pixel 51. Thus, the event detecting section 52 receives a current which is the sum of the photocurrents from all the pixels 51 in the pixel block 41, which may also be only a single pixel.
[0082] When the event detection section 52 detects a change in the photocurrent (the sum of the photocurrents) in the pixel block 41 as an event, the drive section 32 may turn off the transfer transistors 62 of all the pixels 51 in the pixel block 41, thereby stopping the supply of the photocurrent to the event detection section 52. Then, the drive section 32 may sequentially turn on the transfer transistors 63 of the pixels 51 in the pixel block 41 in which the event has been detected using the control signal TRGn, so that the transfer transistors 63 transfer the charge generated in the photoelectric conversion element 61 to the FD 74. The FD 74 accumulates the charge transferred from (the photoelectric conversion element 61 of) the pixel 51. A voltage corresponding to the charge accumulated in the FD 74 is output to VSL as a pixel signal of the pixel 51 through the amplification transistor 72 and the selection transistor 73.
[0083] As described above, in the sensor section 21 ( Figure 2 ), in this way, only the pixel signals of the pixels 51 in which an event has been detected in the pixel block 41 are sequentially output to the VSL. The pixel signals output to the VSL are supplied to the AD conversion section 34 to be AD converted.
[0084] Here, in the pixels 51 in the pixel block 41, the transfer transistors 63 may be turned on not sequentially but simultaneously. In this case, the sum of the pixel signals of all the pixels 51 in the pixel block 41 may be output.
[0085] In addition, the pixel generation portion 53 can also be operated independently of event detection by controlling the transfer transistor 63 without regard to event detection or by completely omitting the transfer transistor 63. In this case, frame data can be generated by the pixel generation portion 53 in a conventional synchronous manner, while event detection can be performed simultaneously. If desired, the photocurrent transmitted to the event detection portion 52 and the pixel signal generation portion 53 can be time-multiplexed so that only one of the two transfer transistors 62, 63 of each pixel 51 is turned on at a given point in time.
[0086] exist Figure 3 In the pixel array section 31, the pixel block 41 includes one or more pixels 51, and the one or more pixels 51 share the event detection section 52 and the pixel signal generation section 53. Therefore, in the case where the pixel block 41 includes a plurality of pixels 51, the number of the event detection section 52 and the pixel signal generation section 53 can be reduced compared to the case where the event detection section 52 and the pixel signal generation section 53 are provided for each of the pixels 51, with the result that the scale of the pixel array section 31 can be reduced.
[0087] Note that, in the case where the pixel block 41 includes a plurality of pixels 51, the event detection section 52 may be provided for each of the pixels 51. In the case where a plurality of pixels 51 in the pixel block 41 share the event detection section 52, an event is detected in units of the pixel block 41. However, in the case where the event detection section 52 is provided for each of the pixels 51, an event may be detected in units of the pixel 51.
[0088] However, even in the case where a plurality of pixels 51 in the pixel block 41 share a single event detection section 52 , an event can be detected in units of pixels 51 when the transfer transistors 62 of the plurality of pixels 51 are temporarily turned on in a time-division manner.
[0089] Furthermore, in the case where it is not necessary to output a pixel signal, the pixel block 41 may be formed without the pixel signal generating section 53. In the case where the pixel block 41 is formed without the pixel signal generating section 53, the sensor section 21 may be formed without the AD conversion section 34 and the transfer transistor 63. In this case, it is possible to reduce the scale of the sensor section 21. The sensor will then output the address of the pixel (block) in which the event occurs, and will also output a time stamp if necessary.
[0090] In addition, additional pixels 51 connected to the pixel signal generating portion 53 but not connected to the event detecting portion 52 may be provided on the sensor die 11. These pixels 51 may be interleaved with the pixels 51 connected to the event detecting portion 52, but may also form a separate pixel array for generating frame data. The separation of the event detection function and the pixel signal generating function may even be achieved by implementing them as two pixel arrays on different dies, the two pixel arrays being arranged to observe the same scene. Thus, essentially any pixel arrangement / pixel circuit system may be used that allows event data to be obtained using a first subset of pixels 51 and pixel signals to be generated using a second subset of pixels 51.
[0091] Figure 5 It shows Figure 3 A block diagram of a configuration example of the event detection section 52.
[0092] The event detection section 52 includes a current-voltage conversion section 81 , a buffer 82 , a subtraction section 83 , a quantization section 84 , and a transmission section 85 .
[0093] The current-voltage conversion section 81 converts (the sum of) the photocurrents from the pixels 51 into a voltage corresponding to the logarithm of the photocurrent (hereinafter also referred to as “photovoltage”), and supplies the voltage to the buffer 82 .
[0094] The buffer 82 buffers the photovoltage from the current-voltage conversion section 81 and supplies the result to the subtraction section 83 .
[0095] The subtraction section 83 calculates the difference between the current photovoltage and the photovoltage at a timing slightly shifted from the current time at a timing indicated by a row drive signal as a control signal from the drive section 32 , and supplies a difference signal corresponding to the difference to the quantization section 84 .
[0096] The quantization section 84 quantizes the difference signal from the subtraction section 83 into a digital signal, and supplies the quantized value of the difference signal to the transmission section 85 as event data.
[0097] The transmission section 85 transmits (outputs) the event data to the output section 35 based on the event data from the quantization section 84. That is, the transmission section 85 provides a request for requesting output of the event data to the arbitrator 33. Then, when receiving a response indicating output permission of the requested event data from the arbitrator 33, the transmission section 85 outputs the event data to the output section 35.
[0098] Figure 6 It shows Figure 5 A circuit diagram of a configuration example of the current-voltage conversion portion 81.
[0099] The current-voltage conversion section 81 includes transistors 91 to 93. For example, N-type MOSFETs may be used as the transistors 91 and 93. For example, P-type MOSFETs may be used as the transistor 92.
[0100] The transistor 91 has a source connected to the gate of the transistor 93, and a photocurrent is supplied from the pixel 51 to a connection point between the source of the transistor 91 and the gate of the transistor 93. The transistor 91 has a drain connected to the power supply VDD, and a gate connected to the drain of the transistor 93.
[0101] The transistor 92 has a source connected to the power supply VDD, and a drain connected to a connection point between the gate of the transistor 91 and the drain of the transistor 93. A predetermined bias voltage Vbias is applied to the gate of the transistor 92. With the bias voltage Vbias, the transistor 92 is turned on or off, and the operation of the current-voltage conversion section 81 is turned on or off depending on whether the transistor 92 is turned on or off.
[0102] The source of the transistor 93 is grounded.
[0103] In the current-voltage conversion section 81, the transistor 91 has a drain connected to the power supply VDD side. The source of the transistor 91 is connected to the pixel 51 ( Figure 4 ), so that a photocurrent based on the charge generated in the photoelectric conversion element 61 of the pixel 51 flows through the transistor 91 (from the drain to the source). The transistor 91 operates in the subthreshold region, and at the gate of the transistor 91, a photovoltage corresponding to the logarithm of the photocurrent flowing through the transistor 91 is generated. As described above, in the current-voltage conversion portion 81, the transistor 91 converts the photocurrent from the pixel 51 into a photovoltage corresponding to the logarithm of the photocurrent.
[0104] In the current-voltage conversion section 81, the transistor 91 has a gate connected to a connection point between the drain of the transistor 92 and the drain of the transistor 93, and a photovoltage is output from the connection point in question.
[0105] Figure 7 It shows Figure 5 A circuit diagram of a configuration example of the subtraction section 83 and the quantization section 84.
[0106] The subtracting section 83 includes a capacitor 101, an operational amplifier 102, a capacitor 103, and a switch 104. The quantizing section 84 includes a comparator 111.
[0107] Capacitor 101 has a capacitor connected to buffer 82 ( Figure 5) and the other end is connected to the input terminal (inverting input terminal) of the operational amplifier 102. Therefore, the photovoltage is input to the input terminal of the operational amplifier 102 through the capacitor 101.
[0108] The operational amplifier 102 has an output terminal connected to the non-inverting input terminal (+) of the comparator 111 .
[0109] The capacitor 103 has one end connected to the input terminal of the operational amplifier 102 , and the other end connected to the output terminal of the operational amplifier 102 .
[0110] The switch 104 is connected to the capacitor 103 to switch the connection between the ends of the capacitor 103. The switch 104 is turned on or off in response to a row driving signal as a control signal from the driving section 32, thereby switching the connection between the ends of the capacitor 103.
[0111] When the switch 104 is turned on, the buffer 82 ( Figure 5 ) side is represented by Vinit, and the capacitance (electrostatic capacitance) of the capacitor 101 is represented by C1. The input terminal of the operational amplifier 102 serves as a virtual ground terminal, and the charge Qinit accumulated in the capacitor 101 when the switch 104 is turned on is represented by expression (1).
[0112] Qinit=C1×Vinit (1)
[0113] Furthermore, in the case where the switch 104 is turned on, the connection between the ends of the capacitor 103 is cut off (short-circuited), so that no charge is accumulated in the capacitor 103 .
[0114] When the switch 104 is turned off, the buffer 82 ( Figure 5 ) side is represented by Vafter, the charge Qafter accumulated in the capacitor 101 when the switch 104 is turned off is represented by Expression (2).
[0115] Qafter=C1×Vafter (2)
[0116] When the capacitance of the capacitor 103 is represented by C2 and the output voltage of the operational amplifier 102 is represented by Vout, the charge Q2 accumulated in the capacitor 103 is represented by Expression (3).
[0117] Q2=-C2×Vout (3)
[0118] Since the total amount of charges in the capacitors 101 and 103 does not change before and after the switch 104 is turned off, expression (4) is established.
[0119] Qinit=Qafter+Q2 (4)
[0120] When Expression (4) is replaced by Expression (1) to Expression (3), Expression (5) is obtained.
[0121] Vout=-(C1 / C2)×(Vafter-Vinit) (5)
[0122] By expression (5), the subtraction part 83 subtracts the photovoltage Vinit from the photovoltage Vafter, that is, calculates the difference signal (Vout) corresponding to the difference Vafter-Vinit between the photovoltage Vafter and Vinit. By expression (5), the subtraction gain of the subtraction part 83 is C1 / C2. Since the maximum gain is usually expected, C1 is preferably set to a large value, and C2 is preferably set to a small value. At the same time, when C2 is too small, the kTC noise increases, resulting in the risk of noise characteristic degradation. Therefore, the capacitor C2 can only be reduced within the range of achieving acceptable noise. In addition, since the pixel blocks 41 each have an event detection part 52 including the subtraction part 83 installed therein, the capacitors C1 and C2 have space limitations. Taking these issues into consideration, the values of the capacitors C1 and C2 are determined.
[0123] The comparator 111 compares the difference signal from the subtracting section 83 with a predetermined threshold value (voltage) Vth (>0) applied to the inverting input terminal (-), thereby quantizing the difference signal. The comparator 111 outputs the quantized value obtained by quantization to the transmission section 85 as event data.
[0124] For example, when the difference signal is greater than the threshold value Vth, the comparator 111 outputs an H (high) level indicating 1 as event data indicating that an event has occurred. When the difference signal is not greater than the threshold value Vth, the comparator 111 outputs an L (low) level indicating 0 as event data indicating that an event has not occurred.
[0125] In the case where the occurrence of a light amount change as an event is confirmed based on the event data from the quantization section 84, that is, in the case where the difference signal (Vout) is greater than the threshold value Vth, the transmission section 85 provides a request to the arbiter 33. When receiving a response indicating event data output permission, the transmission section 85 outputs event data indicating the occurrence of the event (for example, H level) to the output section 35.
[0126] The output section 35 includes, among the event data from the transmission section 85, position / address information about the pixel 51 (including the pixel block 41 of the pixel) in which the event indicated by the event data has occurred and time point information indicating the time point of the event occurrence, and further, as required, the polarity of the change in the amount of light as the event, that is, whether the intensity increases or decreases. The output section 35 outputs the event data.
[0127] As a data format that includes position information about the pixel 51 in which an event has occurred, time point information indicating the time point when the event occurred, and event data of the polarity of the change in the amount of light as the event, for example, a data format called "AER (Address Event Representation)" can be adopted.
[0128] Note that the gain A of the entire event detection section 52 is represented by the following expression, where the gain of the current-voltage conversion section 81 is given by CG log Indicates that, and the gain of buffer 82 is 1.
[0129] A=CG log C1 / C2(∑i photo _n) (6)
[0130] Here, I Photo _n represents the photocurrent of the nth pixel 51 among the I×J pixels 51 in the pixel block 41. In Expression (6), Σ represents the sum of n, and n takes an integer ranging from 1 to I×J.
[0131] Note that the pixel 51 may receive any light as incident light through an optical filter (such as a color filter) through which predetermined light passes. For example, in the case where the pixel 51 receives visible light as incident light, the event data indicates the occurrence of a change in the pixel value in the image including the visible object. In addition, for example, in the case where the pixel 51 receives infrared light, millimeter waves, etc. as incident light for ranging, the event data indicates the occurrence of a change in the distance to the object. In addition, for example, in the case where the pixel 51 receives infrared light for temperature measurement as incident light, the event data indicates the occurrence of a temperature change of the object.
[0132] Figure 8 It shows Figure 2 A timing diagram of an example of the operation of the sensor portion 21.
[0133] At the timing T0, the driving section 32 changes all the control signals OFGn from the L level to the H level, thereby turning on the transfer transistors 62 of all the pixels 51 in the pixel block 41. Thus, the sum of the photocurrents from all the pixels 51 in the pixel block 41 is supplied to the event detection section 52. Here, the control signals TRGn are all at the L level, and thus the transfer transistors 63 of all the pixels 51 are turned off.
[0134] For example, at the timing T1 , when an event is detected, the event detection section 52 outputs event data at an H level in response to the detection of the event.
[0135] At the timing T2, the driving section 32 sets all the control signals OFGn to the L level based on the event data at the H level to stop the supply of the photocurrent from the pixel 51 to the event detecting section 52. In addition, the driving section 32 sets the control signal SEL to the H level and sets the control signal RST to the H level for a certain period of time to control the FD 74 to discharge the electric charge to the power supply VDD, thereby resetting the FD 74. When the FD 74 has been reset, the pixel signal generating section 53 outputs a pixel signal corresponding to the voltage of the FD 74 as a reset level, and the AD conversion section 34 performs AD conversion on the reset level.
[0136] At the timing T3 after the reset level AD conversion, the driving section 32 sets the control signal TRG1 to the H level for a certain period to control the first pixel 51 in the pixel block 41 in which an event has been detected (or which is triggered by other reasons, such as time multiplexing output and / or synchronous readout of frame data), to transfer the charge generated by the photoelectric conversion in (the photoelectric conversion element 61 of) the first pixel 51 to the FD 74. The pixel signal generating section 53 outputs a pixel signal corresponding to the voltage of the FD 74 to which the charge has been transferred from the pixel 51 as a signal level, and the AD conversion section 34 performs AD conversion on the signal level.
[0137] The AD conversion section 34 outputs the difference between the signal level obtained after the AD conversion and the reset level to the output section 35 as a pixel signal serving as a pixel value (frame data) of an image.
[0138] Here, the process of obtaining the difference between the signal level and the reset level as a pixel signal used as a pixel value of an image is referred to as "CDS". CDS may be performed after AD conversion of the signal level and the reset level, or may be performed simultaneously with AD conversion of the signal level and the reset level in the case where the AD conversion section 34 performs single slope AD conversion. In the latter case, AD conversion is performed on the signal level by using the AD conversion result of the reset level as an initial value.
[0139] At timing T4 after AD conversion of the pixel signal of the first pixel 51 in the pixel block 41, the drive section 32 sets the control signal TRG2 to H level for a certain period of time to control the second pixel 51 in the pixel block 41 in which the event has been detected to output the pixel signal.
[0140] In the sensor section 21 , similar processing is performed thereafter so that pixel signals of the pixels 51 in the pixel block 41 that have detected an event are sequentially output.
[0141] When outputting pixel signals of all the pixels 51 in the pixel block 41 , the driving section 32 sets all the control signals OFGn to the H level to turn on the transfer transistors 62 of all the pixels 51 in the pixel block 41 .
[0142] Fig. 9 is a schematic diagram showing an example of a frame data generation method based on event data.
[0143] For example, the logic part 22 sets the frame interval and the frame width based on the external input command. Here, the frame interval represents the frame interval of the frame data generated based on the event data. The frame width represents the time width of the event data used to generate the frame data on a single frame. The frame interval and the frame width set by the logic part 22 are also referred to as "setting the frame interval" and "setting the frame width", respectively.
[0144] The logic section 22 generates frame data as image data in a frame format based on the set frame interval, the set frame width, and the event data from the sensor section 21, thereby converting the event data into the frame data.
[0145] That is, the logic section 22 generates frame data in each set frame interval based on event data in a set frame width starting from the set frame interval.
[0146] Here, it is assumed that the event data includes time point information f indicating a time point at which the event has occurred (hereinafter also referred to as the "event time point") and coordinates (x, y) serving as position information about a pixel 51 (including a pixel block 41 of pixels) at which the event has occurred (hereinafter also referred to as the "event position").
[0147] exist Fig. 9 In the three-dimensional space (time and space) having an x-axis, a y-axis and a time axis t, points representing event data are plotted based on the event time point t and the event position (coordinates) (x, y) included in the event data.
[0148] That is, when the position (x, y, t) on the three-dimensional space indicated by the event time point t and the event position (x, y) included in the event data is regarded as the spatiotemporal position of the event, Fig. 9 In , points representing event data are plotted at the spatiotemporal location (x, y, t) of the event.
[0149] The logic part 22 starts generating frame data based on event data by using a predetermined time point (eg, a time point when frame data generation is externally instructed or a time point when the sensor device 10 is powered on) as a generation start time point at which frame data generation starts.
[0150] Here, within the set frame interval appearing from the generation start time point, each rectangular parallelepiped having a set frame width in the direction of the time axis t is referred to as a "frame volume." For example, the size of the frame volume in the x-axis direction or the y-axis direction is equal to the number of pixel blocks 41 or pixels 51 in the x-axis direction and the y-axis direction.
[0151] The logic section 22 generates frame data on a single frame in each set frame interval based on event data in a frame volume having a set frame width starting from the set frame interval.
[0152] For example, frame data may be generated by setting pixels (pixel values) in a frame at an event position (x, y) included in the event data to white, and setting pixels at other positions in the frame to a predetermined color such as gray.
[0153] In addition, when the event data includes the polarity of the change in the amount of light as an event, the frame data can be generated taking into account the polarity included in the event data. For example, in the case of positive polarity, the pixel can be set to white, and in the case of negative polarity, the pixel can be set to black.
[0154] In addition, in reference Figure 3 and Figure 4 In the case where the pixel signal of the pixel 51 is also output when the event data is output, the frame data can be generated based on the event data by using the pixel signal of the pixel 51. That is, the frame data can be generated by setting the pixel at the event position (x, y) included in the event data (in the block corresponding to the pixel block 41) as the pixel signal of the pixel 51 at the position (x, y) in the frame, and setting a predetermined color such as gray to the pixels at other positions.
[0155] Note that in the frame volume, in some cases, there are multiple pieces of event data that are different at event time points t but at the same event position (x, y). For example, in this case, the event data at the latest or oldest event time point t may be given priority. In addition, in the case where the event data includes polarity, the polarities of the multiple pieces of event data that are different at event time points t but at the same event position (x, y) may be added together, and a pixel value based on the added value obtained by the addition may be set to the pixel at the event position (x, y).
[0156] Here, in the case where the frame width and the frame interval are the same, the frame volumes are adjacent to each other without any gap. In addition, in the case where the frame interval is larger than the frame width, the frame volumes are arranged with gaps. In the case where the frame width is larger than the frame interval, the frame volumes are arranged to partially overlap each other.
[0157] As described above, the pixel signal generating section 53 may also generate frame data in a conventional synchronous manner.
[0158] Fig.10 It shows Figure 5 A block diagram of another configuration example of the quantization section 84 is shown.
[0159] Note that Fig.10 In, with Figure 7 The corresponding parts are denoted by the same figure marks, and their descriptions are appropriately omitted below.
[0160] exist Fig.10 , the quantization section 84 includes comparators 111 and 112 and an output section 113 .
[0161] therefore, Fig.10 The quantization section 84 is similar to the Figure 7 However, Fig.10 The quantization part 84 and Figure 7 The case is different in that a comparator 112 and an output portion 113 are newly included.
[0162] In addition to events, including Fig.10 The event detection section 52 ( Figure 5 ) also detects the polarity of the change in the amount of light as an event.
[0163] exist Fig.10 In the quantization section 84, when the difference signal is greater than the threshold value Vth, the comparator 111 outputs an H level indicating 1 as event data indicating that an event with a positive polarity has occurred. When the difference signal is not greater than the threshold value Vth, the comparator 111 outputs an L level indicating 0 as event data indicating that an event with a positive polarity has not occurred.
[0164] In addition, Fig.10 In the quantization section 84, the threshold value Vth' (<Vth) is supplied to the non-inverting input terminal (+) of the comparator 112, and the difference signal is supplied from the subtraction section 83 to the inverting input terminal (-) of the comparator 112. Here, for example, for the sake of simple description, it is assumed that the threshold value Vth' is equal to -Vth, but this is not necessarily the case.
[0165] The comparator 112 compares the difference signal from the subtraction section 83 with the threshold value Vth' applied to the inverting input terminal (-), thereby quantizing the difference signal. The comparator 112 outputs a quantized value obtained by the quantization as event data.
[0166] For example, in the case where the difference signal is less than the threshold value Vth' (the absolute value of the difference signal having a negative value is greater than the threshold value Vth), the comparator 112 outputs an H level indicating 1 as event data indicating that an event having a negative polarity has occurred. In addition, in the case where the difference signal is not less than the threshold value Vth' (the absolute value of the difference signal having a negative value is not greater than the threshold value Vth), the comparator 112 outputs an L level indicating 0 as event data indicating that an event having a negative polarity has not occurred.
[0167] The output section 113 outputs event data indicating occurrence of an event with positive polarity, event data indicating occurrence of an event with negative polarity, or event data indicating no occurrence of an event to the transmission section 85 based on the event data output from the comparators 111 and 112 .
[0168] For example, in the case where the event data from the comparator 111 is an H level indicating 1, the output section 113 outputs +V volts indicating +1 as event data indicating the occurrence of an event with a positive polarity to the transmission section 85. In addition, in the case where the event data from the comparator 112 is an H level indicating 1, the output section 113 outputs -V volts indicating -1 as event data indicating the occurrence of an event with a positive polarity to the transmission section 85. In addition, in the case where each of the event data from the comparators 111 and 112 is an L level indicating 0, the output section 113 transmits 0 volts (GND level) indicating 0 to the transmission section 85 as event data indicating that no event has occurred.
[0169] In the case where it is confirmed that a change in the amount of light as an event with a positive polarity or a negative polarity has occurred based on the event data from the output portion 113 of the quantization portion 84, the transmission portion 85 provides a request to the arbitrator 33. After receiving a response indicating event data output permission, the transmission portion 85 outputs event data indicating the occurrence of an event with a positive polarity or a negative polarity (+V volts indicating 1 or -V volts indicating -1) to the output portion 35.
[0170] Preferably, the quantization section 84 has Fig.10 Configuration shown.
[0171] Fig.11 is a schematic diagram showing another configuration example of the event detection section 52 .
[0172] exist Fig.11, the event detecting section 52 includes a subtractor 430, a quantizer 440, a memory 451, and a controller 452. The subtractor 430 and the quantizer 440 correspond to the subtracting section 83 and the quantizing section 84, respectively.
[0173] Note that Fig.11 In the embodiment, the event detection section 52 further includes blocks corresponding to the current-voltage conversion section 81 and the buffer 82, but Fig.11 Illustration of these blocks is omitted.
[0174] Subtractor 430 includes capacitor 431, operational amplifier 432, capacitor 433, and switch 434. Capacitor 431, operational amplifier 432, capacitor 433, and switch 434 correspond to capacitor 101, operational amplifier 102, capacitor 103, and switch 104, respectively.
[0175] The quantizer 440 includes a comparator 441. The comparator 441 corresponds to the comparator 111.
[0176] The comparator 441 compares the voltage signal (difference signal) from the subtractor 430 with a predetermined threshold voltage Vth applied to the inverting input terminal (-). The comparator 441 outputs a signal indicating the comparison result as a detection signal (quantized value).
[0177] The voltage signal from the subtractor 430 may be input to an input terminal (−) of the comparator 441 , and a predetermined threshold voltage Vth may be input to an input terminal (+) of the comparator 441 .
[0178] The controller 452 provides a predetermined threshold voltage Vth applied to the inverting input terminal (-) of the comparator 441. The provided threshold voltage Vth can be changed in a time-division manner. For example, the controller 452 provides a threshold voltage Vth1 corresponding to an ON event (e.g., a positive change in photocurrent) and a threshold voltage Vth2 corresponding to an OFF event (e.g., a negative change in photocurrent) at different timings to allow a single comparator to detect multiple types of address events (events).
[0179] The memory 451 accumulates the output of the comparator 441 based on the sampling signal provided from the controller 452. The memory 451 may be a sampling circuit such as a switch, plastic or capacitor, or a digital memory circuit such as a latch or a flip-flop. For example, in a period in which a threshold voltage Vth2 corresponding to an OFF event is provided to the inverting input terminal (-) of the comparator 441, the memory 451 may store the result of comparison by the comparator 441 using the threshold voltage Vth1 corresponding to an ON event. Note that the memory 451 may be omitted, may be provided inside the pixel (pixel block 41), or may be provided outside the pixel.
[0180] Fig.12 It shows Figure 2 A block diagram of another configuration example of the pixel array section 31 is shown.
[0181] Note that Fig.12 In, with Figure 3 The corresponding parts are denoted by the same figure marks, and their descriptions are appropriately omitted below.
[0182] exist Fig.12 , the pixel array section 31 includes a plurality of pixel blocks 41. The pixel block 41 includes I×J pixels 51 as one or more pixels and an event detection section 52.
[0183] therefore, Fig.12 The pixel array portion 31 and Figure 3 The case of is similar because the pixel array section 31 includes a plurality of pixel blocks 41, and the pixel block 41 includes one or more pixels 51 and an event detection section 52. However, Fig.12 The pixel array portion 31 and Figure 3 The case is different from that of FIG. 4 in that the pixel block 41 does not include the pixel signal generating section 53 .
[0184] As mentioned above, in Fig.12 In the pixel array section 31 of FIG. 5 , the pixel block 41 does not include the pixel signal generating section 53, so that the sensor section 21 ( FIG. 5 ) can be formed without the AD conversion section 34. Figure 2 As further described above, the pixel signal generating portion 53 and the pixels 51 fed thereto may be placed on different parts of the die 11, or may even be placed on another die or sensor chip.
[0185] Fig.13 It shows Fig.12 A circuit diagram of a configuration example of a pixel block 41.
[0186] As reference Fig.12 As described, the pixel block 41 includes the pixels 51 and the event detection section 52 , but does not include the pixel signal generation section 53 .
[0187] In this case, the pixel 51 may include only the photoelectric conversion element 61 without including the transfer transistors 62 and 63 .
[0188] Note that pixel 51 has Fig.13 In the case of the configuration shown in , the event detection section 52 can output a voltage corresponding to the photocurrent from the pixel 51 as a pixel signal.
[0189] In the above, the sensor device 10 is described as an asynchronous imaging device configured to read out events by an asynchronous readout system. However, the event readout system is not limited to the asynchronous readout system, and may be a synchronous readout system. The imaging device to which the synchronous readout system is applied is a scanning type imaging device, which is the same as a general imaging device configured to perform imaging at a predetermined frame rate. In addition, event data detection may be performed asynchronously, while pixel signal generation may be performed synchronously.
[0190] Fig.14 is a block diagram showing a configuration example of a scanning type image forming apparatus.
[0191] like Fig.14 As shown, the imaging device 510 includes a pixel array section 521 , a driving section 522 , a signal processing section 525 , a readout region selection section 527 , and a signal generation section 528 .
[0192] The pixel array section 521 includes a plurality of pixels 530. Each of the plurality of pixels 530 outputs an output signal in response to a selection signal from the readout region selection section 527. For example, Fig.11 As shown, the plurality of pixels 530 may each include an intra-pixel quantizer. The plurality of pixels 530 output an output signal corresponding to the amount of change in light intensity. Fig.14 As shown, the plurality of pixels 530 may be two-dimensionally arranged in a matrix.
[0193] The driving part 522 drives the plurality of pixels 530 so that the pixels 530 output the pixel signals generated in the pixels 530 to the signal processing part 525 through the output line 514. Note that the driving part 522 and the signal processing part 525 are circuit parts for acquiring grayscale information. Therefore, in the case of acquiring only event information (event data), the driving part 522 and the signal processing part 525 can be omitted. On the other hand, when event detection is not required, all parts not required for acquiring grayscale information can be omitted.
[0194] The readout area selection section 527 selects some of the plurality of pixels 530 included in the pixel array section 521. For example, the readout area selection section 527 selects one or more rows included in a two-dimensional matrix structure corresponding to the pixel array section 521. The readout area selection section 527 sequentially selects one or more rows based on a preset cycle. In addition, the readout area selection section 527 can determine the selection area based on a request from the pixel 530 in the pixel array section 521.
[0195] The signal generating section 528 generates an event signal corresponding to an active pixel in which an event has been detected in the selected pixel 530, based on the output signal of the pixel 530 selected by the readout area selecting section 527. An event means an event in which light intensity changes. An active pixel means a pixel 530 in which the amount of change in light intensity corresponding to the output signal exceeds or falls below a preset threshold. For example, the signal generating section 528 compares the output signal from the pixel 530 with a reference signal, and detects a pixel that outputs an output signal greater than or less than the reference signal as an active pixel. The signal generating section 528 generates an event signal (event data) corresponding to the active pixel.
[0196] The signal generation portion 528 may include, for example, a column selection circuit configured to arbitrate signals input to the signal generation portion 528. In addition, the signal generation portion 528 may output not only information about active pixels in which an event has been detected, but also information about inactive pixels in which no event has been detected (i.e., it may operate as a conventional synchronous image sensor that generates a series of consecutive image frames).
[0197] The signal generating section 528 outputs address information and time stamp information (e.g., (X, Y, T)) about active pixels that have detected an event through the output line 515. However, the data output from the signal generating section 528 may be not only address information and time stamp information but also information in a frame format (e.g., (0, 0, 1, 0, ...)).
[0198] In the above description, a sensor device 10 has been described in which event data generation and pixel signal generation can be dependent on each other, or can be independent of each other. In addition, the pixel 51 can be shared between the event detection circuit system and the pixel signal generation circuit system through a corresponding circuit system or through time multiplexing. However, the pixel 51 can also be divided so that there are event detection pixels and pixel signal generation pixels. These pixels 51 can be interwoven in the same pixel array, or can be part of different pixel arrays on the same tube core, or even arranged on different tube cores.
[0199] In all of these configurations, the event detection circuit system operates on the pixels 51 of the first subset and is configured to detect when the event data intensity changes by more than a predetermined threshold of the light received by each of the pixels 51 of the first subset, and the pixel signal generation circuit system operates on the pixels 51 of the second subset and is configured to generate pixel signals indicating the intensity value of the received light for each pixel in the second subset of pixels 51.
[0200] In the general description above, it is assumed that both the pixels 51 of the first subset of pixels 51 (i.e., EVS pixels) and the pixels 51 of the second subset of pixels (i.e., APS pixels) can in principle operate based on light having any wavelength. However, in the following, the more specific description of the pixels 51 in the first subset operates on infrared light, i.e., on light having a wavelength in the range of 750nm to 1000μm (preferably between 780nm and 3μm). The pixels 51 in the second subset operate on visible light, i.e., on light having a wavelength between 400nm and 700nm.
[0201] In order to allow collection or transmission of information beyond the observable scene (ie, beyond the two-dimensional intensity distribution observable by the sensor device), a sensor device 10 is provided that uses EVS pixels sensitive to infrared light to detect such information. Fig.15 Such a sensor device 10 is schematically shown.
[0202] like Fig.15 As shown, the sensor device 10 for observing a scene S includes a plurality of pixels 51, each pixel being configured to receive light from the scene and perform photoelectric conversion to generate an electrical signal. As described below, the pixels can be arranged in a two-dimensional pattern or pixel array 51a on a single die or wafer, but can also be arranged on different stacked wafers. In addition, hereinafter, "pixel" shall also include a configuration in which multiple photoelectric conversion elements (such as photodiodes) are read out together to produce a single electrical signal. Referring to such a configuration as a single "pixel" should not be considered a limitation, but is merely used to simplify the description.
[0203] As described above, different pixels 51 may be particularly sensitive to specific wavelength ranges. Therefore, photoelectric conversion elements are used in pixels 51 designed to have high sensitivity in the corresponding wavelength ranges. Specifically, photoelectric conversion elements sensitive to blue light (380nm or 430nm to 490nm), green light (490nm to 570nm or 600nm or 640nm), red light (570nm or 600nm or 640nm to 780nm) or infrared light (780nm to 3000nm or 50000nm or 1mm) can be used. Alternatively or additionally, wavelength sensitivity can be supported or generated by corresponding color filters.
[0204] like Fig.15 As schematically shown in FIG. 5 , the pixels 51 are divided into at least two subsets S1 and S2 . The pixels 51 of the first subset S1 are sensitive to infrared light, while the pixels 51 of the second subset S2 are sensitive to at least visible light.
[0205] The sensor device 10 includes an event detection circuit system 20 that is configured to detect when the event data intensity changes by more than a predetermined threshold of infrared light received by each pixel 51 of the first subset S1. Thus, the pixels 51 of the first subset S1 operate as EVS pixels via the event detection circuit system 20, for example, as described above with respect to Figures 5 to 7 as described.
[0206] The sensor device 10 also includes a pixel signal generation circuit system 30, which is configured to generate a pixel signal indicating an intensity value of visible light received by each pixel 51 of the second subset S2. Therefore, the pixels 51 of the second subset S2 are operated by the pixel signal generation circuit system 30 as APS pixels, which generate, for example, an RGB image with a given frame rate. Here, it should be noted that the pixel signal generation circuit system 30 can also operate on the signals provided from the pixels 51 of the first subset S1, for example in a time-multiplexed manner.
[0207] The control unit 40 of the sensor device 10 is then configured to extract additional information from the event data detected within the received infrared light, which additional information is different from the two-dimensional intensity information about the observed scene S. The control unit 40 can be any kind of processor, circuit system, hardware or software capable of performing the functions of the control unit 40 described herein. The control unit 40 can be located on the same wafer / chip / die as the pixel array 51a and / or the readout circuit system 20, 30, or can be arranged separately, or can even be arranged externally.
[0208] Control unit 40 receives event data generated by event detection circuitry 20 based on received infrared light. In the infrared light, specific information may be encoded that goes beyond the infrared intensity of visible objects in the scene, ie refers to something other than the obvious arrangement of objects O on scene S.
[0209] For example, the sensor device 10 captures an RGB image of the scene S using the pixels 51 of the second subset S2. In addition, the sensor device 10 captures event data, such as positive polarity events P and negative polarity events N. These events can form a specific space, time or spatiotemporal pattern, which represents the information encoded in the received infrared light. The control unit 40 is then able to track, identify and decode the information from the detected events. Therefore, the function of the control unit 40 is not just to align or detect a specific pattern of infrared light that has been received. Instead, the infrared (event) signal is understood as a code that carries additional information and is decoded accordingly to obtain the additional information.
[0210] exist Fig.15In the lower part of FIG. 1 , an example of an image of an observed scene S at a given point in time is shown. The image consists of RGB frames showing an object O within the scene S. The positions of positive polarity events P and negative polarity events N overlap. Positive polarity events P are generated by the appearance of vertical infrared lines illuminating objects in the scene, while negative polarity events N are generated by the disappearance of such lines.
[0211] In conventional systems, the image obtained will simply show the RGB image of the scene with infrared superimposed, such as Fig.15 In contrast, the control unit 40 is able to recognize that the specific pattern of projection of appearing and disappearing infrared light indicates the three-dimensional shape of the object O in the scene S. In this sense, the control unit 40 is configured to decode additional information different from the merely two-dimensional intensity information on the observed scene S.
[0212] In a similar manner, the control unit 40 may also be configured to decode the narrow sense signal X encoded in the received infrared light into additional information. Fig.16 It is shown as an example in FIG.
[0213] Fig.16 A series of RGB frames captured via the pixels 51 of the second subset S2 are shown. The captured scene includes a top view of a forked gripper F with an infrared light emitter E on top. A series of positive polarity events P and negative polarity events N are shown above the RGB frames, which are caused by changes in the intensity of the emitted infrared light. Fig.16 As shown at the bottom of , the sequence of negative polarity events N, positive polarity events P and absence of events forms a signal X in time, which can be used to provide additional information to the sensor device 10 in addition to the indication of the light intensity emitted by the infrared light emitter E.
[0214] Furthermore, it must be noted that event detection is performed with much less latency (i.e. much higher frame rate) than capturing RGB frames. Fig.16 The illustration is a simplification because many events can be detected between RGB frames, which greatly increases the information density received via the sensor device 10 compared to the situation where signal changes can only be observed through the frame rate of the pixel signal generation circuit system 30.
[0215] In particular, the signal X encoded in the received infrared light may be a communication signal preferably generated by modulating the infrared light at a high frequency. Fig.16As shown, one possible application of such an additional IR communication channel may be to monitor objects such as forked grippers or robots. In this way, the RGB video stream can be supplemented with additional information, such as the ID of the monitored object, the operating status (idle vs. occupied), the currently executed task, the operator, etc., without the need to provide additional communication equipment. The RGB information will be automatically synchronized with the additional information and can be combined into more in-depth monitoring data.
[0216] A further example of application of the signal coding for the sensor device lies in the field of computer games, in particular in virtual reality applications. Here, an infrared light emitter can be arranged on equipment worn by the user, such as virtual reality glasses / mask, and provide a signal which allows the position of the user to be determined, and conveys additional information about the user, such as, for example, the ID of the equipment worn (in particular if several users are in the same room), or via identification of the relative movement of the user with respect to the Doppler shift of the sensor device 10. Thus, also in this case, additional information can be transmitted via the infrared signal, while the RGB video channel remains open for video calls or video streaming.
[0217] As discussed above, the sensor device 10 can be used to detect infrared light patterns to infer the three-dimensional shape of an object O within a scene S. Fig.17 As shown, for this purpose, the sensor device 10 may further include an infrared light emitter 50, which is configured to emit a predetermined time-varying infrared light pattern P onto the scene S. Then, the control unit 40 is configured to generate additional information depth information indicating the distance of the object O in the scene S to the sensor device 10 based on the reflection R of the predetermined time-varying infrared light pattern P on the scene S.
[0218] Therefore, in addition to capturing an RGB image of the scene S, the sensor device 10 comprises a structured light arrangement allowing reconstruction of the distance between the observed part of the scene S and the sensor device 10. As the principles of structured light are known to those skilled in the art, they will only be briefly repeated here.
[0219] like Fig.17 and Fig.18As shown in the schematic diagram of , an infrared light emitter 50 (e.g., an infrared vertical cavity surface emitting laser VCSEL, an infrared photodiode array, etc.) projects a known but time-varying pattern P onto a scene S (e.g., one or more vertical lines whose positions change over time). The pattern P will be reflected by an object O in the scene. The reflection R forms a corresponding signal on the pixel array 51a of the sensor device 10. The change in position of the reflection R and the deviation from the shape of the original pattern P allow the distance to all visible parts of the scene S to be determined, for example, by triangulation when the baseline between the infrared light emitter 50 and the pixel array 51a is known. If the infrared light emitter 50 and the pixel array 51a are located in different elements of the sensor device 10, the baseline can be determined via calibration during installation. Alternatively, if both parts have a fixed spatial relationship (e.g., in a common housing), it will be fixed.
[0220] In this way, a three-dimensional representation of the scene S can be obtained. Since the time constant of event detection is much shorter than that of intensity frame detection, multiple scans of the infrared light pattern across the scene can be performed within a single frame capture period of the APS pixels of the second subset S2. Thus, a conventional video stream with a three-dimensional representation of the observed scene S can be provided in real time.
[0221] like Fig.18 As shown, the sensor device 10 may include a preferably wired communication connection 45 between the infrared light emitter 50 and the control unit 40, which allows synchronization of the light projected from the infrared light emitter 50 with the event data detected due to the reception of the infrared light. This allows the control unit 40 to know which specific infrared light pattern P is projected onto the scene S at which time, and how this is distorted by reflections from the scene S. In this way, the accuracy of the depth reconstruction and the accuracy of the three-dimensional model of the scene S generated thereby will be improved. Providing a wired communication connection 45 makes the synchronization fail-safe. On the other hand, if the infrared light emitter 50 and the pixel array 51a are arranged in separate elements, wireless communication may be preferred.
[0222] Fig.19 A schematic and exemplary functional block diagram of a sensor device 10 is shown, which includes, in addition to a pixel array 51a, an infrared light emitter 50. As explained above, the infrared light emitter 50 and the pixel array 51a can operate synchronously based on a communication connection 45. The pixel array transmits EVS data obtained by the event detection circuit system 20 and RGB data obtained by the pixel signal generation circuit system 30 to the control unit 40. The EVS data is fed to a structured light process 40a executed by the control unit 40, which also controls the projection of infrared light patterns via the infrared light emitter 50.
[0223] The depth map created in the structured light process is provided together with the RGB data to a data fusion process 40b performed by the control unit 40, wherein the two data sets are fused. Data fusion can be achieved by simply connecting corresponding data (e.g. by providing a depth value for each RGB pixel value). However, it is also possible to apply more complex data fusion processes, such as building a virtual three-dimensional model of the observed scene, which model can also be used to re-render the observed scene from different angles.
[0224] Based on the fused data, various processes may be performed. For example, the background subtraction process 60a may rely on an improved separation of foreground and background based on a distance threshold, ie only pixel signals having depth values less than the distance threshold are kept.
[0225] Based on the observed three-dimensional model of the person, an avatar of the person (or the person itself) can be rendered and controlled in the avatar process 60b. Similarly, a three-dimensional model of the observed human head can be generated in the viewpoint correction process 60c to allow virtual changes in the viewpoint of the sensor device 10.
[0226] Here, the processes 60a, 60b, 60c performed based on the fused data may be performed by the control unit 40, or as Fig.19 As shown, it is executed by an external device (such as a personal computer).
[0227] Fig. 20 The principle of the viewpoint correction process 60c is schematically explained. Fig. 20 As shown, a human head H is part of the observed scene. The person is using the sensor device 10 to capture his video stream while focusing on the content displayed on the screen 70. For example, the sensor device 10 is used as a camera for video conferencing.
[0228] As shown by the person's line of sight V, the user's line of sight is directed toward the screen 70, not toward the sensor device 10. For this reason, the person does not look into the field of view F of the sensor device 10, but rather looks past it. As a result, other conference participants will have the impression that the person is looking past it, not toward them.
[0229] As described above, the infrared light pattern is projected from the sensor device 10 onto the head H. The control unit 40 is then configured to generate depth information data of the virtual three-dimensional model M of the human head H based on the pixel signal indicating the intensity value of the visible light and the depth information data of the virtual three-dimensional model M of the human head H. Fig. 20 This is illustrated in the lower part of , where a three-dimensional representation M of the head H is shown.
[0230] Based on the data of this three-dimensional representation M, the control unit 40 can be configured, among other things, to generate modified image data that allow the rendering of a human head H on a display with an adjusted viewing direction. Thus, in the three-dimensional representation M of the scene S including the head H, the field of view F of the sensor device 10 is virtually moved so that the viewing direction V lies within the field of view. Adjusted pixel signals can be automatically generated and output so that the sensor device 10 only produces a video of a person looking at the sensor device 10. This allows a more natural video conference, since the participants of the conference can talk to each other like a face-to-face meeting. Alternatively, the control unit can also only provide the data required for viewpoint correction to allow viewpoint correction to be performed on the computer of the data recipient.
[0231] A three-dimensional representation of the captured person as well as viewpoint correction may be generated using a neural network, in particular if occluded parts of the head H need to be reconstructed in order to render the head H with an adjusted line of sight V.
[0232] Although the above possible applications of the sensor device 10 have been described, the following description is directed to the pixel and circuit structure of the sensor device 10. Here, it should be noted that the specific implementation of the sensor material, pixel and / or readout circuit system is in principle known to those skilled in the art and is arbitrary in this sense. Therefore, in the following, the focus will be on the overall arrangement of the different parts of the sensor device 10, but in principle known details are omitted as much as possible.
[0233] like Figure 21 to Figure 24 As shown, in addition to the plurality of pixels 51, the sensor device 10 may further include a plurality of color filters 80 arranged so that at least visible light reaches the pixels 51 in the pixels 51 of the second subset S2, and so that only infrared light reaches the pixels 51 in the pixels 51 of the first subset S1. This ensures that additional information is received at the event detection pixels of the first subset S1 without being interfered by the visible light, and thus ensures good transmission quality.
[0234] like Fig.21As shown exemplarily, the pixels 51 may be arranged into a pixel array 51a having a plurality of 2×2 pixel groups, each 2×2 pixel group including one pixel 51 of the pixels 51 of the first subset S1 and three pixels 51 of the pixels 51 of the second subset S2. The plurality of color filters 80 are correspondingly arranged into a color filter array 80a having a red (R) filter, a green (G) filter, a blue (B) filter, and an infrared (IR) filter, wherein one red (R) filter, one green (G) filter, one blue (B) filter, and one infrared (IR) filter are assigned to each 2×2 pixel group, wherein an infrared filter (IR) is assigned to the pixels 51 of the pixels 51 of the first subset S1. The color filters 80 may be particularly arranged into a modified Bayer pattern, wherein, in each 2×2 color filter block, one green filter is replaced by an IR filter.
[0235] This arrangement ensures that each 2×2 pixel group receives RGB information and is able to detect events in the infrared channel. Therefore, the spatial resolution of the RGB signal and the event data are comparable. In addition, the pixels 51 of the first subset S1 can also be read out by the pixel single generation circuit system 20 in a shared or time-multiplexed manner so as to supplement the RGB channel with the IR channel. Here, the APS pixels in the second subset S2 and the EVS pixels in the first subset S1 can be designed to be particularly sensitive to the wavelength transmitted through the corresponding color filter, such as Fig.21 , as shown by the color filter 80 and the corresponding texture of the pixel 51.
[0236] Figure 22 to Figure 24 Different possible implementations of the arrangement of pixels 51 and color filters 80 are shown in . Here, the pixels 51 are arranged in groups of five pixels 51 on the first wafer W1, wherein one pixel 51 of the first subset S1 is arranged relative to the light incidence direction behind the 2×2 pixels 51 of the second subset S2. Thus, an APS pixel of the second subset S2 is arranged on the light incidence surface of the first wafer W1, while one EVS pixel of the first subset S1 is arranged on the opposite side of the first wafer W1. The area that can be occupied by a pixel 51 of the first subset S1 then corresponds to four times the area that can be occupied by one of the pixels 51 of the second subset S2. This can advantageously compensate for the increased size of the pixels 51 designed for receiving infrared light on the one hand, and increase the sensitivity due to the larger pixel area on the other hand.
[0237] Furthermore, in this case, a plurality of color filters 80 may be arranged into a color filter array 80a having red (R) color filters, green (G) color filters, and blue (B) color filters, preferably arranged in a Bayer pattern, which color filter 80 is infrared transparent. Specifically, no additional infrared color filter is required. Infrared light will pass through the color filter 80 and the pixels 51 of the second subset S2, while visible light will be absorbed by the color filter 80 and the pixels 51 of the second subset S2 before it can reach the pixels 51 of the second subset S2. Therefore, a conventional color filter array 80a may be used in the sensor device 10, which simplifies manufacturing.
[0238] exist Figure 21 to Figure 24 In all examples of FIG. 1 , an additional layer is provided containing event detection circuitry 20 and pixel signal generation circuitry 30. Although these circuits can be provided elsewhere, due to space limitations, they are preferably provided in a stacked manner, such as Figure 21 to Figure 24 The connection between the pixel array 51a and the event detection circuitry 20 or the pixel signal generation circuitry 30 may then be provided by Cu-Cu connections 90, by through silicon vias, TSVs or any other means known in principle to those skilled in the art.
[0239] In all the exemplary cases shown, the sensor device 10 comprises a first wafer W1 on which a plurality of pixels 51 are formed. Here, a color filter array 80a as described above may be present. If the pixels 51 are designed to be sensitive only to a specific wavelength range limited by the color filter 80, the color filter array 80a may be omitted.
[0240] In addition to the first wafer W1, at least one further wafer W2, W3, on which the event detection circuitry 20 and the pixel signal generation circuitry 30 are formed, is provided in a stacked manner.
[0241] Here, the event detection circuit system 20 is formed by a plurality of first circuit blocks 20a, and the pixel signal generation circuit system 30 is formed by a plurality of second circuit blocks 30, wherein the arrangement of the first circuit blocks 20a and the second circuit blocks 30a on at least one further wafer W2, W3 corresponds to the arrangement of the plurality of pixels 51 on the first wafer W1. Therefore, when viewed in the direction of the incident light L, each pixel 51 covers an area in which the corresponding readout circuit system is located behind it. This simplifies the connection of the pixels 51 and the readout circuit systems 20, 30, which not only simplifies the manufacture of the sensor device 10, but also allows a more compact construction.
[0242] For example, all pixels 51 may be formed on the side of the first wafer W1 facing the incident light L, that is, in a backside illumination structure, such as Fig.21As shown. Then, the readout circuit system 20, 30 having the first circuit block 20a and the second circuit block 30a is also formed on the same side of the second wafer W2, for example, also on the light incident side. The first circuit block 20a is formed at a position below / behind the EVS pixel of the first subset S1 (and if present, below / behind the infrared (IR) filter 80). Similarly, the second circuit block 30a forming the pixel signal generation circuit system 30 is formed below / behind the APS pixel of the second subset S2 (and if present, below / behind the RGB filter 80). Therefore, all pixels 51 can be directly connected to the corresponding circuit blocks 20a, 30a, for example via Cu-Cu connection 90.
[0243] according to Fig. 22 In another example shown in , the pixels 51 of the second subset S2 are formed on the side of the first wafer W1 facing the incident light L, while the pixels 51 of the first subset S1 are formed on the opposite side of the first wafer W1. Therefore, as described above, a two-sided implementation is selected, such as an implementation in which a 2×2 APS pixel is located above one (larger) EVS pixel.
[0244] Two further wafers W2, W3 are provided, wherein the wafer adjacent to the first wafer W1 carries the second circuit block 30a, and the other wafer carries the first circuit block 20a. Thus, the second wafer W2 and the third wafer W3 are arranged so that the arrangement of the circuit blocks 20a, 30a can be similar to the arrangement of the corresponding pixels 51. Fig. 22 As shown, the pixels 51 of the second subset S2 and the second circuit block 30a are arranged in a corresponding pattern, so that the first connection 95 and the second connection 96 can be guided from the pixels 51 of the second subset S2 to the second circuit block 30a between the pixels 51 of the first subset S1. Similarly, the pixels 51 of the first subset S1 also have a one-to-one correspondence with the first circuit block 20a. Here, the third connection 97 is guided from the pixels 51 of the first subset S1 to the first circuit block 20a between the second circuit blocks 30a.
[0245] Although this arrangement involves more connection structure, it has the advantage that the event detection circuit system 20 and the pixel signal generation circuit system 30 can be formed on different corrugations, which simplifies manufacturing. In addition, the event detection circuit system 20 can be formed by a larger first circuit block 20a than in the example in which the entire readout circuit system 20, 30 is formed on one side of a single wafer. This allows greater flexibility in designing the event detection circuit system 20.
[0246] Fig.23 The example shown in corresponds to Fig. 22, except that a single second wafer W2 is used for the readout circuitry. Here, the first circuit block 20a and the second circuit block 30a are formed on two opposite sides of the second wafer W2, wherein the side carrying the second circuit block 20a is adjacent to the first wafer W1. All other arrangements are the same as described above with respect to Fig. 22 The arrangement discussed corresponds to . The advantage of providing event detection circuitry 20 and pixel signal generation circuitry 30 on both sides of a single wafer W2 is improved compactness in the stacking direction, however this is at the expense of a more complex manufacturing process.
[0247] Fig.24 Another example shown in provides a Fig. 22 and Fig.23 The same pixel structure discussed above, i.e., the pixels 51 of the second subset S2 are formed on the side of the first wafer W1 facing the incident light L, while the pixels 51 of the first subset S1 are formed on the opposite side of the first wafer W1. However, here, the first circuit block 20a and the second circuit block 30a are formed on the same side of the second wafer W2, so that the first circuit block 20a is surrounded by the second circuit block 30a.
[0248] Therefore, if Fig.24 As shown, in the central area of each 2×2 block of the second circuit block 30a corresponding to the RGB pixel 51 provided above it, a first circuit block 20a is arranged, which is connected to the corresponding EVS pixel above it. This allows the entire readout circuit system 20, 30 to be formed on a single wafer. This simplifies the manufacture and connection of the pixel wafer W1. At the same time, sufficient flexibility is maintained regarding the size of the first circuit block 20a. In this way, a highly functional and compact layout of the sensor device 10 can be obtained.
[0249] It should be understood that all arrangements discussed above are purely exemplary. In these examples, the key lies in the positional correspondence between the APS pixel and its corresponding second circuit block 30a, and between the EVS pixel and its corresponding first circuit block 20a. Specifically, it should be noted that although this correspondence is described in the above examples so that the circuit blocks 20a, 30a are arranged directly below / behind their corresponding pixels 51, there may also be a lateral offset between the corresponding components. Preferably, the offset from the total overlap will still allow a vertical connection between the pixel 51 and the corresponding block 20a, 30a of the readout circuit system 20, 30.
[0250] Various implementations of the basic idea of using infrared light sensitive EVS pixels to receive additional information encoded in infrared light have been discussed above. Fig.25 The basic methods for all these implementations are summarized.
[0251] Here, at S101 , light from the scene S is received, and photoelectric conversion is performed to generate an electric signal using each of the plurality of pixels 51 of the sensor device 10 .
[0252] At S102 , using the event detection circuitry 20 of the sensor device 10 , a detection is performed when the event data intensity changes by more than a predetermined threshold of infrared light received by each of the pixels 51 of the first subset S1 .
[0253] At S103 , a pixel signal is generated using the pixel signal generation circuitry 30 of the sensor device 10 , the pixel signal indicating an intensity value of visible light received by each pixel 51 of the pixels 51 of the second subset S2 .
[0254] At S104 , additional information other than the two-dimensional intensity information about the observed scene S is extracted from the event data detected within the received infrared light.
[0255] In this way, additional information can be communicated, for example about the three-dimensional shape of the observed object, its identity, its mission etc., without the need to provide additional communication equipment. This increases the field in which the sensor device 10 can be used.
[0256] Furthermore, existing devices can be simplified because the functionality of the sensor device 10 can make additional necessary devices redundant. For example, by using the sensor device 10 on a head mounted display for augmented / virtual reality applications, a separate depth sensor can be omitted. While this obviously reduces the complexity and cost of the head mounted display, it also makes the image registration process between the camera and the depth sensor unnecessary, which also reduces the required processing power.
[0257] Fig.26 9 is a block diagram showing an example of a schematic configuration of a smartphone 900, which may constitute or include the sensor device 10 as described above. The smartphone 900 is equipped with a processor 901, a memory 902, a storage device 903, an external connection interface 904, a camera 906, a sensor 907, a microphone 908, an input device 909, a display device 910, a speaker 911, a radio communication interface 912, one or more antenna switches 915, one or more antennas 916, a bus 917, a battery 918, and an auxiliary controller 919.
[0258] The processor 901 may be, for example, a CPU or a system on chip (SoC), and controls functions in the application layer and other layers of the smartphone 900. The memory 902 includes a RAM and a ROM, and stores programs executed by the processor 901 and data. The storage device 903 may include a storage medium such as a semiconductor memory or a hard disk. The external connection interface 904 is an interface for connecting an external attachment device (such as a memory card or a universal serial bus (USB) device) to the smartphone 900. The processor may be used as the control unit 40.
[0259] The camera 906 includes an image sensor as described above. For example, the sensor 907 may include a sensor group such as a positioning sensor, a gyroscope sensor, a geomagnetic sensor, and an acceleration sensor. The microphone 908 converts the audio input to the smartphone 900 into an audio signal. The input device 909 includes a device such as a touch sensor, a keypad, a keyboard, a button, or a switch that detects a touch on the screen of the display device 910, and receives an operation or information input from a user. The display device 910 includes a screen such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display, and displays an output image of the smartphone 900. The speaker 911 converts an audio signal output from the smartphone 900 into audio.
[0260] The radio communication interface 912 supports cellular communication schemes such as LTE or Advanced LTE, and performs radio communication. Typically, the radio communication interface 912 may include a BB processor 913, an RF circuit 914, and the like. For example, the BB processor 913 may perform processes such as encoding / decoding, modulation / demodulation, and multiplexing / demultiplexing, and perform various signal processing for radio communication. Meanwhile, the RF circuit 914 may include components such as a mixer, a color filter, and an amplifier, and send or receive radio signals via an antenna 916. The radio communication interface 912 may also be a single-chip module integrating the BB processor 913 and the RF circuit 914. The radio communication interface 912 may also include multiple BB processors 913 and multiple RF circuits 91. Note that although Fig.26 An example of the radio communication interface 912 including a plurality of BB processors 913 and a plurality of RF circuits 914 is shown, but the radio communication interface 912 may also include a single BB processor 913 or a single RF circuit 914 .
[0261] In addition, in addition to the cellular communication scheme, the radio communication interface 912 can also support other types of radio communication schemes, such as a short-range wireless communication scheme, a near-field wireless communication scheme, or a wireless local area network (LAN) scheme. In this case, a BB processor 913 and an RF circuit 914 can be included for each radio communication scheme.
[0262] Each antenna switch 915 switches the destination of the antenna 916 between a plurality of circuits (eg, circuits for different radio communication schemes) included in the radio communication interface 912 .
[0263] Each antenna 916 includes a single or multiple antenna elements (e.g., multiple antenna elements constituting a MIMO antenna) and is used by the radio communication interface 912 to send and receive radio signals. The smartphone 900 may also include multiple antennas 916, such as Fig.26 Note that although Fig.26 An example of a smartphone 900 including a plurality of antennas 916 is shown, but the smartphone 900 may also include a single antenna 916 .
[0264] In addition, the smartphone 900 may also be equipped with an antenna 916 for each radio communication scheme. In this case, the antenna switch 915 may be omitted from the configuration of the smartphone 900.
[0265] The bus 917 interconnects the processor 901, the memory 902, the storage device 903, the external connection interface 904, the camera 906, the sensor 907, the microphone 908, the input device 909, the display device 910, the speaker 911, the radio communication interface 912, and the auxiliary controller 919. The battery 918 is connected to the power supply line shown by the dotted line in the figure. Fig.26 900. For example, the auxiliary controller 919 enables the minimum functions of the smartphone 900 to operate in the sleep mode.
[0266] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.
[0267] Furthermore, the effects described herein are merely exemplary and not limiting, and other effects may be provided.
[0268] Note that the present technology can also take the following configurations.
[0269] 1. A sensor device for observing a scene, the sensor device comprising:
[0270] a plurality of pixels, each pixel being configured to receive light from a scene and perform photoelectric conversion to generate an electrical signal;
[0271] event detection circuitry configured to detect when the event data intensity changes by more than a predetermined threshold of infrared light received by each of the pixels of the first subset;
[0272] pixel signal generation circuitry configured to generate a pixel signal indicative of an intensity value of visible light received by each pixel of the second subset of pixels; and
[0273] A control unit is configured to extract additional information from event data detected within the received infrared light, the additional information being different from the two-dimensional intensity information about the observed scene.
[0274] 2. The sensor device according to 1, wherein
[0275] The control unit is configured to decode the signal encoded in the received infrared light into additional information.
[0276] 3. The sensor device according to 2, wherein:
[0277] The signal encoded in the received infrared light is a communication signal preferably generated by modulating the infrared light at a high frequency.
[0278] 4. The sensor device according to any one of 1 to 3, further comprising:
[0279] an infrared light emitter configured to emit a predetermined time-varying infrared light pattern onto a scene; wherein,
[0280] The control unit is configured to generate depth information indicating a distance of an object in the scene to the sensor device as additional information based on reflection of a predetermined time-varying infrared light pattern on the scene.
[0281] 5. The sensor device according to 4, further comprising:
[0282] Preferably a wired communication connection between the infrared light emitter and the control unit allows synchronization of light emission from the infrared light emitter with event data detected due to reception of infrared light.
[0283] 6. A sensor device according to 4 or 5, wherein:
[0284] If a human head is part of the observed scene, the control unit is configured to generate, based on pixel signals indicating intensity values of visible light and depth information data of a virtual three-dimensional model of the human head, which preferably allows rendering the human head on the display with an adjusted viewing direction.
[0285] 7. The sensor device according to any one of 1 to 6, further comprising:
[0286] A plurality of color filters are arranged to allow at least visible light to reach pixels in the second subset of pixels and to allow only infrared light to reach pixels in the first subset of pixels.
[0287] 8. The sensor device according to item 7, wherein:
[0288] The pixels are arranged in a pixel array having a plurality of 2×2 pixel groups, each 2×2 pixel group including one pixel from the first subset of pixels and three pixels from the second subset of pixels; and
[0289] A plurality of color filters are arranged into a color filter array having red filters, green filters, blue filters, and infrared filters, wherein one red filter, one green filter, one blue filter, and one infrared filter are assigned to each 2×2 pixel group, wherein the infrared filter is assigned to the pixels in the first subset of pixels.
[0290] 9. The sensor device according to item 7, wherein:
[0291] The pixels are arranged in groups of five pixels, wherein one pixel of the pixels of the first subset is arranged behind 2×2 pixels of the pixels of the second subset with respect to the light incident direction; and
[0292] A plurality of color filters are arranged in a color filter array having red, green and blue filters, preferably in a Bayer pattern, the color filters being infrared transparent.
[0293] 10. The sensor device according to any one of 1 to 9, further comprising:
[0294] a first wafer having a plurality of pixels formed thereon; and
[0295] at least one additional wafer having event detection circuitry and pixel signal generation circuitry formed thereon; wherein,
[0296] The event detection circuitry is formed of a plurality of first circuit blocks;
[0297] The pixel signal generating circuit system is formed of a plurality of second circuit blocks; and
[0298] The arrangement of the first circuit block and the second circuit block on at least one other wafer corresponds to the arrangement of the plurality of pixels on the first wafer.
[0299] 11. The sensor device according to 10, wherein:
[0300] All pixels are formed on a side of the first wafer facing the incident light; and
[0301] The first circuit block and the second circuit block are formed on the same side of the second wafer.
[0302] 12. The sensor device according to 10, wherein:
[0303] The pixels of the second subset of pixels are formed on a side of the first wafer that faces the incident light, and the pixels of the first subset of pixels are formed on an opposite side of the first wafer; and
[0304] Two further wafers are provided, wherein a wafer adjacent to the first wafer carries the second circuit block and the other wafer carries the first circuit block.
[0305] 13. The sensor device according to item 10, wherein:
[0306] The pixels of the second subset of pixels are formed on a side of the first wafer that faces the incident light, and the pixels of the first subset of pixels are formed on an opposite side of the first wafer; and
[0307] The first circuit block and the second circuit block are formed on two opposite sides of the second wafer, wherein the side carrying the second circuit block is adjacent to the first wafer.
[0308] 14. The sensor device according to item 10, wherein:
[0309] The pixels of the second subset of pixels are formed on a side of the first wafer that faces the incident light, and the pixels of the first subset of pixels are formed on an opposite side of the first wafer; and
[0310] The first circuit block and the second circuit block are formed on the same side of the second wafer such that the first circuit block is surrounded by the second circuit block.
[0311] 15. A method for operating a sensor device for observing a scene, the method comprising:
[0312] receiving light from a scene and performing photoelectric conversion to generate an electrical signal using each of a plurality of pixels of the sensor device;
[0313] detecting, using event detection circuitry of the sensor device, when an intensity change in event data exceeds a predetermined threshold of infrared light received by each of the pixels of the first subset;
[0314] generating, with pixel signal generation circuitry, a pixel signal indicative of an intensity value of visible light received by each pixel of the second subset of pixels; and
[0315] Additional information other than the two-dimensional intensity information about the observed scene is extracted from event data detected within the received infrared light.
Claims
1. A sensor device (10) for observing a scene (S), the sensor device (10) include: a plurality of pixels (51), each pixel being configured to receive light from the scene and perform photoelectric conversion to generate an electrical signal; event detection circuitry (20) configured to detect when an event data intensity change exceeds a predetermined threshold of infrared light received by each of the pixels (51) of the first subset (S1); a pixel signal generating circuit system (30) configured to generate a pixel signal indicating an intensity value of visible light received by each pixel (51) of the pixels (51) of the second subset (S2); as well as A control unit (40) is configured to extract additional information from event data detected within the received infrared light, the additional information being different from the two-dimensional intensity information about the observed scene (S).
2. The sensor device (10) according to claim 1, in, The control unit (40) is configured to decode the signal (X) encoded in the received infrared light into additional information.
3. The sensor device (10) according to claim 2, in, Said signal (X) encoded in the received infrared light is a communication signal preferably generated by modulating said infrared light at a high frequency.
4. The sensor device (10) according to claim 1, further comprising: include: an infrared light emitter (50) configured to emit a predetermined time-varying infrared light pattern (P) onto the scene (S); wherein, The control unit (40) is configured to generate depth information indicating the distance of an object (O) in the scene to the sensor device (10) as additional information based on the reflection (R) of the predetermined time-varying infrared light pattern (P) on the scene (S).
5. The sensor device (10) according to claim 4, further comprising: include: Preferably a wired communication connection (45) between the infrared light emitter (50) and the control unit (40) allows synchronization of light emission from the infrared light emitter (50) with the event data detected due to reception of infrared light.
6. The sensor device (10) according to claim 4, in, If a human head (H) is part of the observed scene (S), the control unit (40) is configured to generate, based on the pixel signals indicating the intensity values of the visible light and depth information data of a virtual three-dimensional model (M) of the human head, the depth information data preferably allowing rendering of the human head on a display with an adjusted viewing direction.
7. The sensor device (10) according to claim 1, further comprising: include: A plurality of color filters (80) are arranged so that at least visible light reaches the pixels (51) in the second subset (S2) of pixels (51) and only infrared light reaches the pixels (51) in the first subset (S1) of pixels.
8. The sensor device (10) according to claim 7, in, The pixels (51) are arranged in a pixel array (51a), the pixel array having a plurality of 2×2 pixel groups, each 2×2 pixel group comprising one pixel (51) of the pixels (51) of the first subset (S1) and three pixels (51) of the pixels (51) of the second subset (S2); and The plurality of color filters (80) are arranged into a color filter array (80a) having red (R) filters, green (G) filters, blue (B) filters and infrared (IR) filters, wherein one red (R) filter, one green (G) filter, one blue (B) filter and one infrared (IR) filter are assigned to each 2×2 pixel group, wherein the infrared filter (IR) is assigned to the pixel (51) in the pixel (51) of the first subset (S1).
9. The sensor device (10) according to claim 7, in, The pixels (51) are arranged in groups of five pixels (51), wherein one pixel (51) of the pixels (51) of the first subset (S1) is arranged behind 2×2 pixels (51) of the pixels (51) of the second subset (S2) with respect to the light incident direction; and The plurality of color filters (80) are arranged in a color filter array (80a) having red (R) color filters, green (G) color filters and blue (B) color filters, preferably arranged in a Bayer pattern, and the plurality of color filters (80) are infrared transparent.
10. The sensor device (10) according to claim 1, further comprising: include: a first wafer (W1) on which the plurality of pixels (51) are formed; as well as at least one additional wafer (W2, W3) on which the event detection circuit system (20) and the pixel signal generation circuit system (30) are formed; wherein, The event detection circuit system (20) is formed by a plurality of first circuit blocks (20a); The pixel signal generating circuit system (30) is formed by a plurality of second circuit blocks (30a); and The arrangement of the first circuit block (20a) and the second circuit block (30a) on the at least one other chip (W2, W3) corresponds to the arrangement of the plurality of pixels (51) on the first chip (W1).
11. The sensor device (10) according to claim 10, in, All pixels (51) are formed on a side of the first wafer (W1) facing the incident light (L); and The first circuit block (20a) and the second circuit block (30a) are formed on the same side of a second wafer (W2).
12. The sensor device (10) according to claim 10, in, The pixels (51) of the second subset (S2) of pixels (51) are formed on the side of the first wafer (W1) facing the incident light (L), while the pixels (51) of the first subset (S1) of pixels (51) are formed on the opposite side of the first wafer (W1); and Two further wafers (W2, W3) are provided, wherein a wafer adjacent to the first wafer (W1) carries the second circuit block (30a) and the other wafer carries the first circuit block (20a).
13. The sensor device (10) according to claim 10, in, The pixels (51) of the second subset (S2) of pixels (51) are formed on the side of the first wafer (W1) facing the incident light (L), while the pixels (51) of the first subset (S1) of pixels (51) are formed on the opposite side of the first wafer (W1); and The first circuit block (20a) and the second circuit block (30a) are formed on two opposite sides of a second wafer (W2), wherein the side carrying the second circuit block (20a) is adjacent to the first wafer (W1).
14. The sensor device (10) according to claim 10, in, The pixels (51) of the second subset (S2) of pixels (51) are formed on the side of the first wafer (W1) facing the incident light (L), while the pixels (51) of the first subset (S1) of pixels (51) are formed on the opposite side of the first wafer (W1); and The first circuit block (20a) and the second circuit block (30a) are formed on the same side of a second wafer (W2) such that the first circuit block (20a) is surrounded by the second circuit block (30a).
15. A method for operating a sensor device (10) for observing a scene (S), the method include: receiving light from the scene (S) and performing photoelectric conversion to generate an electrical signal using each of a plurality of pixels (51) of the sensor device (10); Utilizing event detection circuitry (20) of the sensor device (10), detecting when an event data intensity change exceeds a predetermined threshold of infrared light received by each of the pixels (51) of the first subset (S1); generating, using pixel signal generating circuitry (30), a pixel signal indicative of an intensity value of visible light received by each pixel (51) of the pixels (51) of the second subset (S2); as well as Additional information other than the two-dimensional intensity information about the observed scene (S) is extracted from the event data detected within the received infrared light.
