SENSOR DEVICE AND METHOD FOR OPERATION OF SENSOR DEVICE - Patent application

Abstract

To provide a sensor device that allows the capture of image data from which an undisturbed RGB video stream can be derived along with additional data. [Solution] A sensor device (10) for observing a scene comprises a plurality of pixels (51), each configured to receive light from the scene and perform photoelectric conversion to generate an electrical signal, an event detection circuit (20) configured to detect as an event a data intensity of infrared light received by each of a first subset (S1) of the pixels (51) that exceeds a predetermined threshold, a pixel signal generation circuit (30) configured to generate a pixel signal indicating the intensity value of visible light received by each pixel (51) of a second subset (S2) of the pixels (51), and a control unit (40) configured to extract additional information, different from two-dimensional intensity information regarding the observed scene, from the event data detected in the received infrared light.

Description

[Technical field]

[0001] The present disclosure relates to a sensor device and a method of operating the sensor device, and in particular to a sensor device and a method of operating the sensor device that allows for capturing a visible light video stream with additional information provided via infrared. [Background technology]

[0002] A typical digital video camera may be used, for example, as a stand-alone sensor device or integrated into other electronic devices, such as a mobile phone, to capture video frames indicative of intensity values ​​of visible light, for example using red (R), green (G) and blue (B) color filters, etc. For example, such RGB video may be captured to be transmitted to and displayed on other electronic devices for video conferencing, surveillance, monitoring of industrial or logistics processes, computer games, etc.

[0003] The frames captured in this way display a 2D image that is a projection of the observed 3D scene onto the sensitive surface of the respective sensor device. Often, it is desirable to generate a 3D model of the observed scene from the captured 2D images. For example, in a video conferencing application, it may be desirable to correctly subtract the background of the video conferencing participants, to animate avatars / animojis representing the video conferencing participants, or to correct the viewpoints of the video conferencing participants, for example, to resemble the impression that they are looking at each other. All these functions can be realized by reconstructing a 3D model of at least the heads of the video conferencing participants. Although it is in principle possible to generate a 3D model of the observed scene from captured 2D images, such 3D virtualization often requires computational power that is prohibitively usable in practice, and virtualization based on RGB image frames is usually too slow to convert a 2D image stream into a 3D model of the observed scene in real time.

[0004] On the other hand, it is known to generate a depth map indicating the distance of each part of the scene to the sensor device, for example by using a structured light source, in which case the distance of objects visible in the scene can be inferred from the captured distortion of a certain known pattern projected onto the scene. Such a depth map can be used to generate a desired three-dimensional model of the observed scene.

[0005] However, even here, the problem arises that computing a depth map from frame images is time-consuming and therefore unsuitable for generating a real-time 3D virtualization of a scene. Moreover, in applications like video conferencing, visible light cannot be used to generate a depth map with structured light techniques, since the projected light pattern is visible in the exchanged video stream.

[0006] Thus, although methods for depth reconstruction are available in principle, they cannot be implemented natively without degrading the captured RGB image.

[0007] It is also often useful if the imaging functionality of a sensor device can receive additional information other than the captured RGB video stream without degrading the captured RGB image, which can be useful for surveillance and monitoring tasks where capturing a particular process or object automatically receives additional information in the image data.

[0008] It is therefore desirable to provide a sensor device that allows for the capture of image data from which an undisturbed RGB video stream can be derived along with additional data. Summary of the Invention [Problem to be solved by the invention]

[0009] [Yuhei 1] To this end, a sensor device is provided for observing a scene, the sensor device comprising a plurality of pixels, each configured to receive light from the scene and perform photoelectric conversion to generate an electrical signal; an event detection circuit configured to detect as an event a data intensity of infrared light received by each of a first subset of the pixels that exceeds a predetermined threshold; a pixel signal generation circuit configured to generate a pixel signal indicative of an intensity value of visible light received by each pixel of a second subset of the pixels; and a control unit configured to extract additional information, different from two-dimensional intensity information regarding the observed scene, from the event data detected in the received infrared light.

[0010] Also provided is a method of operating a sensor device for observing a scene, the method comprising: receiving light from the scene by each of a plurality of pixels of the sensor device and performing photoelectric conversion to generate an electrical signal; detecting as an event a data intensity of the infrared light received by each of a first subset of the pixels that exceeds a predetermined threshold; generating by a pixel signal generation circuit of the sensor device a pixel signal indicative of an intensity value of the visible light received by each pixel of a second subset of the pixels; and extracting additional information, different from two-dimensional intensity information about the observed scene, from the event data detected in the received infrared light.

[0011] Thus, conventional video camera pixels (second subset of pixels), also called Active Pixel Sensor (APS) pixels, generate pixel signals representative of received light intensity, in addition to Dynamic / Event-Based Vision Sensor (DVS / EVS) pixels (first subset of pixels) that are sensitive to intensity changes above a certain threshold, called an event. While video camera pixels generate pixel signals representative of received light intensity in the visible spectrum, EVS pixels work in infrared, i.e. they are sensitive to intensity changes of received infrared light. This sensor device is able to search and extract additional information encoded in infrared light from the detected event stream.

[0012] Thus, in addition to the ability to infer the time sequence of a two-dimensional intensity distribution in the infrared part of the electromagnetic spectrum, the sensor device can filter additional information beyond the mere two-dimensional intensity distribution. In particular, it is possible to use infrared structured light to make it possible to reconstruct a depth map from the detected events. Alternatively, it is also possible to decode messages sent by infrared transmitters visible on the scene. The infrared spectrum is used to encode this additional information destined for the sensor device, so that the visible spectrum observed by the APS pixels is not affected by this information.

[0013] Furthermore, because the data rate generated by event detection is much smaller than the data rate generated by full intensity frame detection, EVS pixels ensure that additional information is available fast enough to supplement the full intensity pixel signal with the additional information in real time. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is a schematic diagram of a sensor device. [Diagram 2] FIG. 2 is a schematic block diagram of a sensor unit. [Diagram 3]FIG. 2 is a schematic block diagram of a pixel array unit. [Figure 4] FIG. 2 is a schematic circuit diagram of a pixel block. [Diagram 5] FIG. 2 is a block diagram illustrating an event detection unit. [Figure 6] FIG. 4 is a schematic circuit diagram of a current-voltage converter. [Figure 7] FIG. 2 is a schematic circuit diagram of a subtraction unit and a quantization unit. [Figure 8] 4 is a timing chart illustrating an example of the operation of a sensor unit. [Figure 9] FIG. 2 is a schematic diagram of a method for generating frame data based on event data. [Figure 10] FIG. 13 is a schematic block diagram of another quantization unit. [Figure 11] FIG. 13 is a schematic diagram of another event detection unit. [Figure 12] FIG. 11 is a schematic block diagram of another pixel array unit. [Figure 13] FIG. 11 is a schematic circuit diagram of another pixel block. [Figure 14] FIG. 1 is a schematic block diagram of a scanning sensor device. [Figure 15] FIG. 2 is a schematic block diagram of a sensor device. [Figure 16] FIG. 2 shows a schematic diagram of an infrared encoded signal received by a sensor device; [Figure 17] FIG. 11 is a schematic block diagram of another sensor device. [Figure 18] FIG. 18 is a diagram illustrating the function of the sensor device of FIG. 17. [Figure 19] FIG. 18 is a functional block diagram of the sensor device of FIG. 17. [Figure 20] FIG. 13 is a diagram illustrating a viewpoint adjustment process. [Figure 21] FIG. 2 is a diagram illustrating a schematic structure of a sensor device. [Figure 22] FIG. 13 is a diagram illustrating a schematic structure of another sensor device. [Diagram 23] FIG. 13 is a diagram illustrating a schematic structure of another sensor device. [Figure 24] FIG. 13 is a diagram illustrating a schematic structure of another sensor device. [Diagram 25] FIG. 1 shows a schematic process flow of a method of operating a sensor device. [Figure 26] FIG. 1 is a schematic block diagram of a smartphone. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The present disclosure aims to mitigate the problem that arises when a sensor device is provided with information about the captured scene, which goes beyond just the 2D intensity distribution observed by the sensor device. Such information may, for example, allow for the conversion of the 2D intensity distribution into a 3D model of the observed scene in real time, or may consist of a message addressed to the sensor device. The solution to this problem described below is applicable to all sensor types that are able to capture a scene by generating an infrared-based event data stream together with a visible-based full-intensity frame sequence.

[0016] Here, we first provide a general description of the Event-Based Vision Sensor (EVS) with reference to Figs. 1 to 14 to clarify its functionality and to cover important application examples. This general description mainly focuses on the possibility of sharing pixels between the EVS and APS circuits to create a hybrid EVS-APS device. Then, starting with Fig. 15, we describe more specific concepts aimed at solving the above problems.

[0017] Of course, it should be understood that the following description is purely exemplary and that the (hybrid) EVS can be implemented in different ways.

[0018] FIG. 1 is a diagram showing an example of the configuration of a sensor device 10, which in the example of FIG. 1 is configured by a sensor chip.

[0019] The sensor device 10 is a single-chip type semiconductor chip, and includes a sensor die (substrate) 11 that functions as multiple dies (substrates), and a stacked logic die 12. Note that the sensor device 10 may be only a single die, or may include three or more stacked dies.

[0020] 1, the sensor die 11 includes a sensor unit 21 (including a circuit that functions as the sensor unit 21), and the logic die 12 includes a logic unit 22. Note that a part of the sensor unit 21 may be formed on the logic die 12. Furthermore, a part of the logic unit 22 may be formed on the sensor die 11.

[0021] The sensor unit 21 includes pixels configured to photoelectrically convert incident light to generate electrical signals, and generates event data indicating the occurrence of an event, which is a change in the electrical signals of these pixels. The sensor unit 21 supplies the event data to the logic unit 22. That is, the sensor unit 21 captures an image by photoelectrically converting incident light in the pixels to generate electrical signals, similar to, for example, a synchronous image sensor. However, in addition to generating image data in a frame format (frame data), the sensor unit 21 also generates event data indicating the occurrence of an event, which is a change in the electrical signals of the pixels. The sensor unit 21 outputs the event data obtained by capturing an image to the logic unit 22.

[0022] Here, the synchronous image sensor is an image sensor configured to capture images in synchronization with a vertical synchronization signal and output frame data, which is image data in a frame format. The event detection function of the sensor unit 21 does not operate in synchronization with the vertical synchronization signal when outputting event data with respect to event detection, and therefore can be considered as an asynchronous type (asynchronous image sensor) as opposed to a synchronous image sensor.

[0023] Note that the sensor unit 21 generates and outputs frame data other than event data, similar to a synchronous image sensor. Furthermore, the sensor unit 21 can output, together with the event data, an electrical signal of a pixel in which an event has occurred as a pixel signal that is the pixel value of the pixel in the frame data.

[0024] The logic unit 22 controls the sensor unit 21 as necessary. Furthermore, the logic unit 22 performs various data processing such as data processing to generate frame data based on event data from the sensor unit 21, and image processing on the frame data from the sensor unit 21 or the frame data generated based on the event data from the sensor unit 21, and outputs the data processing results obtained by performing various data processing on the event data and frame data.

[0025] FIG. 2 is a block diagram showing an example of the configuration of the sensor unit 21 in FIG.

[0026] The sensor unit 21 includes a pixel array unit 31, a drive unit 32, an arbitration unit 33, an AD (Analog to Digital) conversion unit 34, and an output unit 35.

[0027] The pixel array unit 31 preferably includes a plurality of pixels 51 (FIG. 3) arranged in a two-dimensional lattice. When a change greater than a predetermined threshold (including a change equal to or greater than the threshold as necessary) occurs in a photocurrent (corresponding voltage), which is an electrical signal generated by photoelectric conversion in the pixel 51, the pixel array unit 31 detects the change in photocurrent as an event. When an event is detected, the pixel array unit 31 outputs a request to the arbitration unit 33 to request output of event data indicating the occurrence of the event. When a response indicating permission to output the event data is received from the arbitration unit 33, the pixel array unit 31 outputs the event data to the drive unit 32 and the output unit 35. Furthermore, the pixel array unit 31 may output an electrical signal, a pixel signal, of the pixel 51 in which the event was detected to the AD conversion unit 34.

[0028] The drive unit 32 supplies a control signal to the pixel array unit 31 to drive the pixel array unit 31. For example, the drive unit 32 drives the pixel 51 to which the pixel array unit 31 outputs event data, so that the pixel 51 supplies (outputs) a pixel signal to the AD conversion unit 34. However, the drive unit 32 may synchronously drive the pixel 51 to generate frame data.

[0029] The arbitration unit 33 arbitrates requests for output of event data from the pixel array unit 31, and returns a response to the pixel array unit 31 indicating whether output of the event data is permitted or prohibited.

[0030] The AD conversion unit 34 includes, for example, a single-slope ADC (AD converter) (not shown) provided for each column of pixel blocks 41 (FIG. 3) described later. The AD conversion unit 34 performs AD conversion on pixel signals of pixels 51 in the pixel blocks 41 in the column using the ADC for each column, and supplies the result to the output unit 35. The AD conversion unit 34 can perform CDS (correlated double sampling) together with the AD conversion of the pixel signals.

[0031] The output unit 35 performs necessary processing on the pixel signal from the AD conversion unit 34 and the event data from the pixel array unit 31, and supplies the results to the logic unit 22 (FIG. 1).

[0032] Here, since a change in the photocurrent generated in pixel 51 can be regarded as a change in the amount of light incident on pixel 51, a change in the amount of light in pixel 51 (a change in the amount of light greater than a threshold) can also be said to be an event.

[0033] Event data indicating the occurrence of an event includes at least position information (coordinates, etc.) indicating the position of the pixel block where the change in light amount, which is the event, occurred. The event data can also include the polarity (positive or negative) of the change in light amount.

[0034] For a series of event data output from the pixel array unit 31 at the timing of an event occurrence, it can be said that time point information indicating the (relative) time point when the event occurred is implicitly included in the event data as long as the event data interval is the same as the event occurrence interval. However, for example, when the event data is stored in a memory and the event data interval is no longer the same as the event occurrence interval, the time point information implicitly included in the event data is lost. Therefore, the output unit 35 includes, in the event data, time point information indicating the (relative) time point when the event occurred, such as a timestamp, before the event data interval is changed from the event occurrence interval. The process of including time point information in the event data can be performed by any block other than the output unit 35, as long as it is performed before the time point information implicitly included in the event data is lost.

[0035] FIG. 3 is a block diagram showing an example of the configuration of the pixel array unit 31 in FIG.

[0036] The pixel array unit 31 may include a plurality of pixel blocks 41. Each pixel block 41 may include I×J pixels 51, which are one or more pixels arranged in I rows and J columns (I and J are integers), an event detection unit 52, and a pixel signal generation unit 53. One or more pixels 51 in the pixel block 41 share the event detection unit 52 and the pixel signal generation unit 53. Furthermore, a VSL (Vertical Signal Line) is wired in each column of the pixel block 41 to connect the pixel block 41 to the ADC of the AD conversion unit 34.

[0037] The pixel 51 receives light incident from an object and performs photoelectric conversion to generate a photocurrent that becomes an electric signal. The pixel 51 supplies the photocurrent to the event detection unit 52 under the control of the drive unit 32.

[0038] Under the control of the driving unit 32, the event detection unit 52 detects, as an event, that the photocurrent from each pixel 51 has changed more than a predetermined threshold. When an event is detected, the event detection unit 52 can supply a request to the arbitration unit 33 (FIG. 2) to request output of event data indicating the occurrence of the event. When the event detection unit 52 receives a response to the request from the arbitration unit 33 indicating permission to output the event data, the event detection unit 52 outputs the event data to the driving unit 32 and the output unit 35.

[0039] When the event detection unit 52 detects an event, the pixel signal generation unit 53 generates a voltage corresponding to the photocurrent from the pixel 51 as a pixel signal, and supplies it to the AD conversion unit 34 via VSL under the control of the drive unit 32.

[0040] Here, detecting a change in (log) photocurrent greater than a predetermined threshold as an event can also be regarded as detecting no change in photocurrent greater than the predetermined threshold as an event. The pixel signal generator 53 can generate a pixel signal both when detecting a change in photocurrent greater than the predetermined threshold as an event and when detecting no change in photocurrent greater than the predetermined threshold as an event.

[0041] Furthermore, the event detection unit 52 and the pixel signal generation unit 53 may operate independently of each other. That is, while the event detection unit 52 generates event data, the pixel signal generation unit 53 synchronously generates pixel signals to generate frame data. The event detection unit 52 and the pixel signal generation unit 53 may operate in a time division multiplexed manner so as to receive photocurrents from the pixels 51 only at different time intervals.

[0042] Fig. 4 is a circuit diagram showing an example of the configuration of the pixel block 41. The configuration in Fig. 4 may be referred to as a "shared pixel configuration."

[0043] As described with reference to FIG. 3, the pixel block 41 includes a pixel 51, an event detection unit 52, and a pixel signal generation unit 53.

[0044] The pixel 51 includes a photoelectric conversion element 61 and transfer transistors 62 and 63 .

[0045] The photoelectric conversion element 61 includes, for example, a photodiode (PD). The photoelectric conversion element 61 receives incident light and performs photoelectric conversion to generate electric charges.

[0046] An example of the transfer transistor 62 is an N (Negative) type MOS (Metal-Oxide-Semiconductor) FET (Field Effect Transistor). The transfer transistor 62 of the n-th pixel 51 among the I×J pixels 51 in the pixel block 41 is turned on and off in response to a control signal OFGn supplied from the drive unit 32 (FIG. 2). When the transfer transistor 62 is turned on, the charge generated in the photoelectric conversion element 61 is transferred (supplied) to the event detection unit 52 as a photocurrent.

[0047] The transfer transistor 63 includes, for example, an N-type MOSFET. The transfer transistor 63 of the n-th pixel 51 among the I×J pixels 51 in the pixel block 41 is turned on and off in response to a control signal TRGn supplied from the drive unit 32. When the transfer transistor 63 is turned on, the charge generated in the photoelectric conversion element 61 is transferred to the FD 74 of the pixel signal generation unit 53.

[0048] The I×J pixels 51 in the pixel block 41 are connected to the event detection unit 52 of the pixel block 41 via the node 60. Therefore, the photocurrent generated in the pixel 51 (the photoelectric conversion element 61) is supplied to the event detection unit 52 via the node 60. As a result, the event detection unit 52 receives the sum of the photocurrents from all the pixels 51 in the pixel block 41. Therefore, the event detection unit 52 detects a change in the sum of the photocurrents supplied from the I×J pixels 51 in the pixel block 41 as an event.

[0049] The pixel signal generation unit 53 includes a reset transistor 71 , an amplification transistor 72 , a selection transistor 73 , and an FD (Floating Diffusion) 74 .

[0050] The reset transistor 71, the amplifying transistor 72, and the selection transistor 73 each include, for example, an N-type MOSFET.

[0051] The reset transistor 71 is turned on / off in response to a control signal RST supplied from the drive unit 32 (FIG. 2). When the reset transistor 71 is turned on, the FD 74 is connected to the power supply VDD, and the charge accumulated in the FD 74 is discharged to the power supply VDD. This resets the FD 74.

[0052] The amplifying transistor 72 has a gate connected to the FD 74, a drain connected to a power supply VDD, and a source connected to VSL via the selection transistor 73. The amplifying transistor 72 is a source follower, and outputs a voltage (electrical signal) corresponding to the voltage of the FD 74 supplied to the gate to VSL via the selection transistor 73.

[0053] The selection transistor 73 is turned on / off in response to a control signal SEL supplied from the drive unit 32. When the selection transistor 73 is turned on, a voltage corresponding to the voltage of the FD 74 from the amplification transistor 72 is output to VSL.

[0054] The FD 74 accumulates the charge transferred from the photoelectric conversion element 61 of the pixel 51 via the transfer transistor 63 and converts it into a voltage.

[0055] For the pixel 51 and pixel signal generation unit 53 configured as above, the drive unit 32 turns on the transfer transistor 62 using a control signal OFGn, and the transfer transistor 62 supplies a photocurrent based on the charge generated in the photoelectric conversion element 61 of the pixel 51 to the event detection unit 52. As a result, the event detection unit 52 receives a current that is the sum of the photocurrents from all the pixels 51 in the pixel block 41 (there may be only one pixel).

[0056] When the event detection unit 52 detects a change in photocurrent (total photocurrent) in the pixel block 41 as an event, the drive unit 32 may stop the supply of photocurrent to the event detection unit 52 by turning off the transfer transistors 62 of all pixels 51 in the pixel block 41. The drive unit 32 may sequentially turn on the transfer transistors 63 of the pixels 51 in the pixel block 41 in which the event is detected using a control signal TRGn, so that the transfer transistors 63 transfer the charge generated in the photoelectric conversion element 61 to the FD 74. The FD 74 accumulates the charge transferred to the pixel 51 (from the photoelectric conversion element 61). A voltage corresponding to the charge accumulated in the FD 74 is output to VSL as a pixel signal of the pixel 51 via the amplification transistor 72 and selection transistor 73.

[0057] As described above, in the sensor unit 21 (FIG. 2), only pixel signals of pixels 51 in pixel blocks 41 in which an event has been detected are sequentially output to the VSL. The pixel signals output to the VSL are supplied to the AD conversion unit 34 and are AD converted.

[0058] Here, in the pixels 51 in the pixel block 41, the transfer transistors 63 can be turned on simultaneously rather than sequentially. In this case, the sum of the pixel signals of all the pixels 51 in the pixel block 41 can be output.

[0059] Furthermore, it is possible to operate the pixel generator 53 independently of event detection, either by controlling the transfer transistor 63 independently of event detection or by omitting the transfer transistor 63 entirely. In this case, frame data can be generated by the pixel generator 53 in a conventional synchronous manner and event detection can occur simultaneously. If desired, the transfer of photocurrent to the event detector 52 and pixel signal generator 53 may be time-division multiplexed so that only one of the two transfer transistors 62, 63 per pixel 51 is open at a given time.

[0060] 3, pixel block 41 includes one or more pixels 51, and one or more pixels 51 share an event detection unit 52 and a pixel signal generation unit 53. In this way, when pixel block 41 includes multiple pixels 51, the number of event detection units 52 and pixel signal generation units 53 can be reduced compared to the case where an event detection unit 52 and a pixel signal generation unit 53 are provided for each pixel 51, and as a result, the size of pixel array unit 31 can be reduced.

[0061] When the pixel block 41 includes multiple pixels 51, an event detection unit 52 can be provided for each pixel 51. When multiple pixels 51 in a pixel block 41 share an event detection unit 52, an event is detected for each pixel block 41. On the other hand, when an event detection unit 52 is provided for each pixel 51, an event can be detected for each pixel 51.

[0062] However, even if multiple pixels 51 in a pixel block 41 share one event detection unit 52, events can be detected on a pixel 51 basis by temporarily turning on the transfer transistors 62 of the multiple pixels 51 in a time-division manner.

[0063] Furthermore, if there is no need to output a pixel signal, the pixel block 41 can be formed without providing the pixel signal generation unit 53. When the pixel block 41 is formed without providing the pixel signal generation unit 53, the sensor unit 21 can be formed without providing the AD conversion unit 34 and the transfer transistor 63. In this case, the scale of the sensor unit 21 can be reduced. The sensor outputs the address of the pixel (block) where an event has occurred together with a timestamp as necessary.

[0064] Also, additional pixels 51 connected to pixel signal generation 53 rather than event detection 52 may be provided on the sensor die 11. These pixels 51 may be interleaved with the pixels 51 connected to event detection 52, but may form a separate pixel array for generating frame data. The separation of the event detection and pixel signal generation functions may also be implemented as two pixel arrays on different dies arranged to observe the same scene. Thus, essentially any pixel arrangement / pixel circuitry capable of acquiring event data on a first subset of pixels 51 and generating pixel signals on a second subset of pixels 51 may be used.

[0065] FIG. 5 is a block diagram showing an example of the configuration of the event detection unit 52 in FIG.

[0066] The event detection unit 52 includes a current-voltage conversion unit 81 , a buffer 82 , a subtraction unit 83 , a quantization unit 84 , and a transfer unit 85 .

[0067] The current-voltage converter 81 converts (the sum of) the photocurrents from the pixels 51 into a voltage (hereinafter also referred to as “photovoltage”) corresponding to the logarithm of the photocurrent, and supplies it to a buffer 82.

[0068] The buffer 82 buffers the photovoltage from the current-voltage conversion unit 81 and supplies the result to a subtraction unit 83 .

[0069] The subtraction unit 83 calculates the difference between the current optical voltage and the optical voltage at a timing slightly shifted from the current time at a timing indicated by the column drive signal, which is a control signal from the drive unit 32, and supplies a difference signal corresponding to the difference to the quantization unit 84.

[0070] The quantization unit 84 quantizes the difference signal from the subtraction unit 83 into a digital signal, and supplies the quantized value of the difference signal to the transfer unit 85 as event data.

[0071] The transfer unit 85 transfers (outputs) the event data to the output unit 35 based on the event data from the quantization unit 84. That is, the transfer unit 85 supplies a request for output of the event data to the arbitration unit 33. When the transfer unit 85 receives a response to the request from the arbitration unit 33 indicating permission to output the event data, the transfer unit 85 outputs the event data to the output unit 35.

[0072] FIG. 6 is a circuit diagram showing an example of the configuration of the current-voltage conversion unit 81 in FIG.

[0073] The current-voltage conversion unit 81 includes transistors 91 to 93. For example, an N-type MOSFET can be used as the transistors 91 and 93. For example, a P-type MOSFET can be used as the transistor 92.

[0074] The source of the transistor 91 is connected to the gate of the transistor 93, and a photocurrent is supplied from the pixel 51 to the connection point between the source of the transistor 91 and the gate of the transistor 93. The drain of the transistor 91 is connected to a power supply VDD, and the gate is connected to the drain of the transistor 93.

[0075] The source of the transistor 92 is connected to a power supply VDD, and the drain is connected to the connection point between the gate of the transistor 91 and the drain of the transistor 93. A predetermined bias voltage Vbias is applied to the gate of the transistor 92. The bias voltage Vbias turns the transistor 92 on or off, and the operation of the current-voltage conversion unit 81 is turned on or off depending on whether the transistor 92 is on or off.

[0076] The source of transistor 93 is grounded.

[0077] In the current-voltage conversion unit 81, the drain of the transistor 91 is connected to the power supply VDD side. The source of the transistor 91 is connected to the pixel 51 (FIG. 4), and a photocurrent based on charges generated in the photoelectric conversion element 61 of the pixel 51 flows (from the drain to the source) via the transistor 91. The transistor 91 operates in the subthreshold region, and a photovoltage corresponding to the logarithm of the photocurrent flowing through the transistor 91 is generated at the gate of the transistor 91. As described above, in the current-voltage conversion unit 81, the transistor 91 converts the photocurrent from the pixel 51 into a photovoltage corresponding to the logarithm of the photocurrent.

[0078] In the current-voltage converter 81, the gate of the transistor 91 is connected to a connection point between the drain of the transistor 92 and the drain of the transistor 93, and a photoelectric conversion voltage is output from the connection point.

[0079] FIG. 7 is a circuit diagram showing an example of the configuration of the subtraction unit 83 and the quantization unit 84 in FIG.

[0080] The subtraction unit 83 includes a capacitor 101, an operational amplifier 102, a capacitor 103, and a switch 104. The quantization unit 84 includes a comparator 111.

[0081] One end of the capacitor 101 is connected to the output terminal of the buffer 82 (FIG. 5), and the other end is connected to the input terminal (inverting input terminal) of the operational amplifier 102. Therefore, the photovoltage is input to the input terminal of the operational amplifier 102 via the capacitor 101.

[0082] The output terminal of the operational amplifier 102 is connected to the non-inverting input terminal (+) of the comparator 111 .

[0083] One end of the capacitor 103 is connected to the input terminal of the operational amplifier 102 , and the other end is connected to the output terminal of the operational amplifier 102 .

[0084] The switch 104 is connected to the capacitor 103 and switches the connection between both ends of the capacitor 103. The switch 104 switches the connection between both ends of the capacitor 103 by being turned on and off in response to a column drive signal, which is a control signal from the drive unit 32.

[0085] The optical voltage on the buffer 82 (FIG. 5) side of the capacitor 101 when the switch 104 is on is represented by Vinit, and the capacitance (electrostatic capacity) of the capacitor 101 is represented by C1. The input terminal of the operational amplifier 102 becomes a virtual ground terminal, and the charge Qinit accumulated in the capacitor 101 when the switch 104 is on is expressed by equation (1). Qinit = C1 × Vinit (1)

[0086] Furthermore, when the switch 104 is on, the connection between both ends of the capacitor 103 is cut (short-circuited), and no charge is stored in the capacitor 103.

[0087] If the photovoltage on the buffer 82 (FIG. 5) side of the capacitor 101 when the switch 104 is subsequently turned off is Vafter, the charge Qafter stored in the capacitor 101 when the switch 104 is off is expressed by equation (2). Qafter=C1×Vafter (2)

[0088] If the capacitance of the capacitor 103 is C2 and the output voltage of the operational amplifier 102 is Vout, then the charge Q2 stored in the capacitor 103 is expressed by equation (3). Q2=-C2×Vout (3)

[0089] Since the total amount of charge in the capacitors 101 and 103 does not change before and after the switch 104 is turned off, the formula (4) holds. Qinit=Qafter+Q2 (4)

[0090] By substituting equations (1) to (3) into equation (4), equation (5) is obtained. Vout=-(C1 / C2)×(Vafter-Vinit) (5)

[0091] According to the formula (5), the subtraction unit 83 subtracts the photovoltage Vinit from the photovoltage Vafter, that is, calculates a difference signal (Vout) corresponding to the difference Vafter-Vinit between the photovoltage Vafter and the photovoltage Vinit. According to the formula (5), the subtraction gain of the subtraction unit 83 is C1 / C2. Since the maximum gain is usually desired, it is preferable to set C1 to a large value and C2 to a small value. On the other hand, if C2 is too small, kTC noise may increase and noise characteristics may deteriorate. Therefore, the capacitance C2 can only be made small to a range where an allowable noise can be realized. Furthermore, since the event detection unit 52 including the subtraction unit 83 is installed in each pixel block 41, there is a space restriction for the capacitances C1 and C2. Taking these matters into consideration, the values ​​of the capacitances C1 and C2 are determined.

[0092] The comparator 111 quantizes the differential signal from the subtraction unit 83 by comparing the differential signal with a predetermined threshold (voltage) Vth (>0) applied to an inverting input terminal (-). The comparator 111 outputs the quantized value obtained by the quantization to the transfer unit 85 as event data.

[0093] For example, when the difference signal is greater than the threshold Vth, the comparator 111 outputs an H (High) level indicating 1 as event data indicating the occurrence of an event. When the difference signal is not greater than the threshold Vth, the comparator 111 outputs an L (Low) level indicating 0 as event data indicating the absence of an event.

[0094] When it is confirmed that a change in the amount of light that constitutes an event has occurred based on the event data from the quantization unit 84, that is, when the differential signal (Vout) is greater than the threshold value Vth, the transfer unit 85 supplies a request to the arbitration unit 33. Upon receiving a response indicating permission to output event data, the transfer unit 85 outputs event data (e.g., H level) indicating the occurrence of an event to the output unit 35.

[0095] The output unit 35 includes, in the event data from the transfer unit 85, position / address information on the pixel 51 (including the pixel block 41) where the event indicated by the event data occurred, time point information indicating the time point when the event occurred, and, if necessary, the polarity of the change in light amount that is the event, i.e., whether the intensity has increased or decreased. The output unit 35 outputs the event data.

[0096] As a data format for the event data that includes position information regarding the pixel 51 where the event occurred, time information indicating the time when the event occurred, and the polarity of the change in light intensity that is the event, for example, a data format called "AER (Address Event Representation)" can be adopted.

[0097] The gain A of the entire event detection unit 52 is calculated by subtracting the gain of the current-voltage conversion unit 81 from the gain A of the current-voltage conversion unit 81. log If the gain of the buffer 82 is 1, then this is expressed by the following equation. A=CG log C1 / C2(Σi photo _n) (6)

[0098] where i photo n represents the photocurrent of the n-th pixel 51 among the I×J pixels 51 in the pixel block 41. In equation (6), Σ represents the sum of n, which is an integer from 1 to I×J.

[0099] Note that the pixel 51 can receive any light as incident light by an optical filter, such as a color filter, through which predetermined light passes. For example, when the pixel 51 receives visible light as incident light, the event data indicates the occurrence of a change in pixel value in an image including a visible object. Also, for example, when the pixel 51 receives infrared light or millimeter waves for distance measurement as incident light, the event data indicates the occurrence of a change in distance to the object. Furthermore, for example, when the pixel 51 receives infrared light for temperature measurement as incident light, the event data indicates the occurrence of a temperature change of the object.

[0100] FIG. 8 is a timing chart showing an example of the operation of the sensor unit 21 in FIG.

[0101] At timing T0, the drive unit 32 changes all of the control signals OFGn from L level to H level, thereby turning on the transfer transistors 62 of all of the pixels 51 in the pixel block 41. As a result, the sum of the photocurrents from all of the pixels 51 in the pixel block 41 is supplied to the event detection unit 52. Here, since the control signals TRGn are all at L level, the transfer transistors 63 of all of the pixels 51 are off.

[0102] For example, when an event is detected at timing T1, the event detection section 52 outputs H-level event data in response to the detection of the event.

[0103] At timing T2, the drive unit 32 sets all the control signals OFGn to the L level based on the H level event data, and stops the supply of photocurrent from the pixels 51 to the event detection unit 52. Furthermore, the drive unit 32 sets the control signal SEL to the H level, and sets the control signal RST to the H level for a certain period of time, thereby controlling the FD 74 to discharge electric charge to the power supply VDD and resetting the FD 74. The pixel signal generation unit 53 outputs a pixel signal corresponding to the voltage of the FD 74 when the FD 74 is reset as a reset level, and the AD conversion unit 34 performs AD conversion on this reset level.

[0104] At timing T3 after AD conversion of the reset level, the drive unit 32 sets the control signal TRG1 to H level for a certain period in order to control the first pixel 51 in the pixel block 41 in which an event has been detected (or triggered for other reasons, such as time-division output and / or synchronous readout of frame data), and transfers the charge generated by photoelectric conversion in (the photoelectric conversion element 61 of) the first pixel 51 to the FD 74. The pixel signal generation unit 53 outputs a pixel signal corresponding to the voltage of the FD 74 to which the charge has been transferred from the pixel 51 as a signal level, and the AD conversion unit 34 performs AD conversion on this signal level.

[0105] The AD conversion section 34 outputs the difference between the signal level obtained after AD conversion and the reset level to the output section 35 as a pixel signal that becomes the pixel value of the image (frame data).

[0106] Here, the process of obtaining the difference between the signal level and the reset level as a pixel signal that becomes the pixel value of the image is called "CDS." CDS can be performed after AD conversion of the signal level and the reset level, or, if the AD conversion unit 34 performs single-slope AD conversion, it can be performed simultaneously with AD conversion of the signal level and the reset level. In the latter case, AD conversion of the signal level is performed with the AD conversion result of the reset level as the initial value.

[0107] At timing T4 after AD conversion of the pixel signal of the first pixel 51 in the pixel block 41, the drive unit 32 sets the control signal TRG2 to an H level for a certain period of time, and controls the second pixel 51 in the pixel block 41 in which the event was detected to output a pixel signal.

[0108] In the sensor unit 21, the same process is subsequently executed, and pixel signals of the pixels 51 in the pixel block 41 in which an event has been detected are sequentially output.

[0109] When the pixel signals of all the pixels 51 in the pixel block 41 have been output, the drive unit 32 sets all the control signals OFGn to the H level to turn on the transfer transistors 62 of all the pixels 51 in the pixel block 41 .

[0110] FIG. 9 is a diagram illustrating an example of a method for generating frame data based on event data.

[0111] The logic unit 22 sets the frame interval and frame width based on, for example, an externally input command. Here, the frame interval represents the frame interval of frame data generated based on event data. The frame width represents the time width of the event data used to generate the frame data of one frame. The frame interval and frame width set by the logic unit 22 are also referred to as the "set frame interval" and "set frame width", respectively.

[0112] The logic unit 22 generates frame data, which is image data in a frame format, based on the set frame section, set frame width, and event data from the sensor unit 21, thereby converting the event data into frame data.

[0113] That is, in each set frame section, the logic section 22 generates frame data based on event data of the set frame width from the beginning of the set frame section.

[0114] Here, the event data includes time point information t that indicates the time point when the event occurred (hereinafter, also referred to as the “event time point”). i and coordinates (x, y) that are position information (hereinafter also referred to as "event position") regarding the pixel 51 (or the pixel block 41 that includes the pixel 51) where the event occurred.

[0115] In FIG. 9, points representing event data are plotted in a three-dimensional space (space-time) having an x-axis, a y-axis, and a time axis t, based on the event time t and event position (coordinates) (x, y) contained in the event data.

[0116] In other words, when the position (x, y, t) in three-dimensional space indicated by the event time t and event position (x, y) contained in the event data is regarded as the spatiotemporal position of the event, in FIG. 9, a point representing this event data is plotted on the spatiotemporal position (x, y, t) of the event.

[0117] The logic unit 22 starts generating frame data based on the event data, using a predetermined point in time, such as the point in time when generation of frame data is instructed from outside or the point in time when the power of the sensor device 10 is turned on, as the start point in time for starting generation of the frame data.

[0118] Here, a rectangular parallelepiped having a set frame width in the time axis t direction in the set frame section that appears from the generation start time point is referred to as a “frame volume.” The size of the frame volume in the x-axis or y-axis direction is equal to, for example, the number of pixel blocks 41 or pixels 51 in the x-axis or y-axis direction.

[0119] The logic unit 22 generates frame data for one frame in each set frame section based on event data in a frame volume having a set frame width from the beginning of the set frame section.

[0120] The frame data can be generated, for example, by setting the pixel (pixel value) in the frame at the event position (x, y) included in the event data to white, and setting pixels at other positions in the frame to a predetermined color such as gray.

[0121] In addition, if the event data includes the polarity of the change in light amount that is the event, the frame data can be generated taking into consideration the polarity included in the event data. For example, if the polarity is positive, white can be set to the pixels, and if the polarity is negative, black can be set to the pixels.

[0122] 3 and 4, if the pixel signal of pixel 51 is also output when event data is output, frame data can be generated based on the event data using the pixel signal of pixel 51. That is, in a frame, the pixel at an event position (x, y) (within a block corresponding to pixel block 41) included in the event data is set to the pixel signal of pixel 51 at that position (x, y), and pixels at other positions are set to a predetermined color such as gray, thereby generating frame data.

[0123] In some cases, a frame volume may contain multiple event data with different event times t but the same event position (x, y). In this case, for example, the event data with the latest or oldest event time t may be prioritized. Furthermore, if the event data includes polarity, the polarities of multiple event data with different event times t but the same event position (x, y) may be added, and a pixel value based on the added value may be set to the pixel at the event position (x, y).

[0124] Here, when the frame width and the frame interval are the same, the frame volumes are arranged adjacent to each other with no gaps. Furthermore, when the frame interval is larger than the frame width, the frame volumes are arranged with gaps. When the frame width is larger than the frame interval, the frame volumes are arranged so as to overlap each other partially.

[0125] As described above, the pixel signal generator 53 can also generate frame data in a conventional synchronous manner.

[0126] FIG. 10 is a block diagram showing another example of the configuration of the quantization unit 84 in FIG.

[0127] In FIG. 10, parts corresponding to those in FIG. 7 are given the same reference numerals, and descriptions thereof will be omitted below as appropriate.

[0128] In FIG. 10, the quantization unit 84 includes comparators 111 and 112 and an output unit 113.

[0129] In this way, the quantization unit 84 in FIG. 10 is the same as that in FIG. 7 in that it includes the comparator 111. However, the quantization unit 84 in FIG. 10 is different from that in FIG. 7 in that it newly includes the comparator 112 and the output unit 113.

[0130] The event detection unit 52 (FIG. 5) including the quantization unit 84 in FIG. 10 detects the polarity of the light amount change that becomes an event in addition to the event.

[0131] In the quantization unit 84 of FIG. 10, when the difference signal is greater than the threshold value Vth, the comparator 111 outputs an H level indicating 1 as event data indicating the occurrence of an event having a positive polarity. When the difference signal is not greater than the threshold value Vth, the comparator 111 outputs an L level indicating 0 as event data indicating that no event having a positive polarity has occurred.

[0132] Furthermore, in the quantization unit 84 of FIG. 10, a threshold value Vth' (<Vth) is supplied to the non-inverting input terminal (+) of the comparator 112, and a difference signal is supplied from the subtraction unit 83 to the inverting input terminal (-) of the comparator 112. Here, for simplicity, it is assumed that the threshold value Vth' is equal to, for example, -Vth, but this is not necessarily the case.

[0133] The comparator 112 quantizes the difference signal by comparing the difference signal from the subtraction unit 83 with the threshold value Vth' applied to the inverting input terminal (-). The comparator 112 outputs the quantization value obtained by this quantization as event data.

[0134] For example, when the differential signal is smaller than the threshold Vth' (when the absolute value of the differential signal having a negative value is larger than the threshold Vth), the comparator 112 outputs an H level indicating 1 as event data indicating the occurrence of an event having negative polarity. Furthermore, when the differential signal is not smaller than the threshold Vth' (when the absolute value of the differential signal having a negative value is not larger than the threshold Vth), the comparator 112 outputs an L level indicating 0 as event data indicating the absence of an event having negative polarity.

[0135] Based on the event data output from comparators 111 and 112, output unit 113 outputs to transfer unit 85 event data indicating the occurrence of an event having positive polarity, event data indicating the occurrence of an event having negative polarity, or event data indicating that no event has occurred.

[0136] For example, when the event data from the comparator 111 is at an H level indicating 1, the output unit 113 outputs +V volts indicating +1 to the transfer unit 85 as event data indicating the occurrence of an event having positive polarity. Furthermore, when the event data from the comparator 112 is at an H level indicating 1, the output unit 113 outputs -V volts indicating -1 to the transfer unit 85 as event data indicating the occurrence of an event having negative polarity. Furthermore, when the event data from the comparators 111 and 112 is at an L level indicating 0, the output unit 113 outputs 0 volts (GND level) indicating 0 to the transfer unit 85 as event data indicating the absence of an event.

[0137] The transfer unit 85 supplies a request to the arbitration unit 33 when it is confirmed that a change in light amount, which is an event having positive polarity or negative polarity, has occurred based on the event data from the output unit 113 of the quantization unit 84. After receiving a response indicating permission to output event data, the transfer unit 85 outputs event data indicating the occurrence of an event having positive polarity or negative polarity (+V volts indicating 1 or -V volts indicating -1) to the output unit 35.

[0138] Preferably, the quantization unit 84 has a configuration as shown in FIG.

[0139] FIG. 11 is a diagram showing another example of the configuration of the event detection unit 52. In FIG.

[0140] 11, the event detection unit 52 includes a subtractor 430, a quantizer 440, a memory 451, and a controller 452. The subtractor 430 and the quantizer 440 correspond to the subtraction unit 83 and the quantization unit 84, respectively.

[0141] In FIG. 11, the event detection unit 52 further includes blocks corresponding to a current-voltage conversion unit 81 and a buffer 82, but these are omitted from the drawing.

[0142] The subtractor 430 includes a capacitor 431, an operational amplifier 432, a capacitor 433, and a switch 434. The capacitor 431, the operational amplifier 432, the capacitor 433, and the switch 434 correspond to the capacitor 101, the operational amplifier 102, the capacitor 103, and the switch 104, respectively.

[0143] The quantizer 440 includes a comparator 441. The comparator 441 corresponds to the comparator 111.

[0144] The comparator 441 compares the voltage signal (difference signal) from the subtractor 430 with a predetermined threshold voltage Vth applied to an inverting input terminal (-). The comparator 441 outputs a signal indicating the comparison result as a detection signal (quantized value).

[0145] The voltage signal from the subtractor 430 may be input to an input terminal (−) of a comparator 441, and a predetermined threshold voltage Vth may be input to an input terminal (+) of the comparator 441.

[0146] The controller 452 supplies a predetermined threshold voltage Vth to be applied to the inverting input terminal (-) of the comparator 441. The supplied threshold voltage Vth may be changed in a time-division manner. For example, the controller 452 supplies a threshold voltage Vth1 corresponding to an ON event (e.g., a positive change in photocurrent) and a threshold voltage Vth2 corresponding to an OFF event (e.g., a negative change in photocurrent) at different timings, so that a single comparator can detect multiple types of address events.

[0147] The memory 451 accumulates the output from the comparator 441 based on a sample signal supplied from the controller 452. The memory 451 may be a sampling circuit such as a switch, a plastic, or a capacitor, or may be a digital memory circuit such as a latch or a flip-flop. For example, the memory 451 may hold a comparison result by the comparator 441 using a threshold voltage Vth1 corresponding to an ON event during a period in which a threshold voltage Vth2 corresponding to an OFF event is supplied to the inverting input terminal (-) of the comparator 441. Note that the memory 451 may be omitted, or may be provided inside the pixel (pixel block 41), or may be provided outside the pixel.

[0148] FIG. 12 is a block diagram showing another example of the configuration of the pixel array unit 31 in FIG.

[0149] In FIG. 12, parts corresponding to those in FIG. 3 are given the same reference numerals, and descriptions thereof will be omitted below as appropriate.

[0150] 12, the pixel array section 31 includes a plurality of pixel blocks 41. The pixel block 41 includes I×J pixels 51, which are one or a plurality of pixels, and an event detection section 52.

[0151] 12 is similar to that in Fig. 3 in that the pixel array section 31 includes a plurality of pixel blocks 41, and each pixel block 41 includes one or a plurality of pixels 51 and an event detection section 52. However, the pixel array section 31 in Fig. 12 is different from that in Fig. 3 in that the pixel block 41 does not include a pixel signal generation section 53.

[0152] As described with reference to FIG. 12, the pixel block 41 includes the pixels 51 and the event detection unit 52 but does not include the pixel signal generation unit 53.

[0153] 12, the pixel block 41 does not include the pixel signal generation unit 53, so the sensor unit 21 (FIG. 2) can be formed without the AD conversion unit 34. Furthermore, as described above, the pixel signal generation unit 53 and the pixel 51 that supplies the signal may be disposed in another part of the die 11, or may be disposed on another die or sensor chip.

[0154] FIG. 13 is a circuit diagram showing an example of the configuration of the pixel block 41 in FIG.

[0155] In this case, the pixel 51 may not include the transfer transistors 62 and 63 and may include only the photoelectric conversion element 61 .

[0156] When the pixel 51 has the configuration shown in FIG. 13, the event detection section 52 can output a voltage corresponding to the photocurrent from the pixel 51 as a pixel signal.

[0157] In the above, the sensor device 10 has been described as an asynchronous imaging element configured to read out events by an asynchronous readout system. However, the event readout system is not limited to an asynchronous readout system, and may be a synchronous readout system. The imaging element to which the synchronous readout system is applied is a scan-type imaging element that is the same as a general imaging element configured to capture images at a predetermined frame rate. Furthermore, the detection of event data may be performed asynchronously, and the generation of pixel signals may be performed synchronously.

[0158] FIG. 14 is a block diagram showing an example of the configuration of a scan-type imaging element.

[0159] As shown in FIG. 14, the imaging element 510 includes a pixel array section 521, a drive section 522, a signal processing section 525, a readout region selection section 527, and a signal generation section 528.

[0160] The pixel array unit 521 includes a plurality of pixels 530. Each of the plurality of pixels 530 outputs an output signal in response to a selection signal from the readout region selection unit 527. Each of the plurality of pixels 530 may include an in-pixel quantizer as shown in FIG. 11, for example. The plurality of pixels 530 outputs an output signal corresponding to the amount of change in light intensity. The plurality of pixels 530 may be two-dimensionally arranged in a matrix as shown in FIG. 14.

[0161] The driving unit 522 drives a plurality of pixels 530, so that the pixels 530 output pixel signals generated by the pixels 530 to the signal processing unit 525 via the output lines 514. The driving unit 522 and the signal processing unit 525 are circuit units for acquiring grayscale information. Therefore, when only event information (event data) is acquired, the driving unit 522 and the signal processing unit 525 can be omitted. On the other hand, when event detection is not required, all parts not required for acquiring grayscale information may be omitted.

[0162] The readout region selection unit 527 selects a portion of the multiple pixels 530 included in the pixel array unit 521. For example, the readout region selection unit 527 selects one or more rows included in a two-dimensional matrix structure corresponding to the pixel array unit 521. The readout region selection unit 527 sequentially selects one or more rows based on a preset period. Furthermore, the readout region selection unit 527 may determine the selection region based on a request from the pixels 530 in the pixel array unit 521.

[0163] The signal generating unit 528 generates an event signal corresponding to an active pixel in which an event of the selected pixel 530 is detected, based on the output signal of the pixel 530 selected by the readout region selecting unit 527. An event means an event in which the intensity of light changes. An active pixel means a pixel 530 in which the amount of change in the intensity of light corresponding to the output signal exceeds or falls below a preset threshold. For example, the signal generating unit 528 compares the output signal from the pixel 530 with a reference signal, and detects a pixel that outputs an output signal that is greater or smaller than the reference signal as an active pixel. The signal generating unit 528 generates an event signal (event data) corresponding to the active pixel.

[0164] The signal generator 528 may, for example, include a column selection circuit configured to arbitrate the signals input to the signal generator 528. Furthermore, the signal generator 528 may output information not only about active pixels where an event is detected, but also about inactive pixels where no event is detected, i.e., it may operate as a conventional synchronous image sensor generating a series of consecutive image frames.

[0165] The signal generator 528 outputs address information and timestamp information (e.g., (X,Y,T)) relating to the active pixel in which an event is detected via output line 515. However, the data output from the signal generator 528 may be not only address information and timestamp information, but also information in a frame format (e.g., (0,0,1,0,......)).

[0166] The above description describes a sensor device 10 in which event data generation and pixel signal generation may be interdependent or independent of each other. Also, pixels 51 may be shared between the event detection circuitry and the pixel signal generation circuitry by the respective circuits or by time division multiplexing. However, the pixels 51 may be divided such that there are event detection pixels and pixel signal generation pixels. These pixels 51 may be interleaved in the same pixel array or may be part of different pixel arrays on the same die or may be located on different dies.

[0167] In all these configurations, the event detection circuit operates on a first subset of pixels 51 and is configured to detect an intensity change in light received by each of the first subset of pixels 51 that exceeds a predetermined threshold as event data, and the pixel signal generation circuit operates on a second subset of pixels 51 and is configured to generate a pixel signal indicative of an intensity value of the light received for each pixel of the second subset of pixels 51.

[0168] In the general description above, it was stated that both the pixels 51 of the first subset of pixels 51, i.e. the EVS pixels, and the pixels 51 of the second subset of pixels 51, i.e. the APS pixels, can in principle operate on light having any wavelength. However, in the more specific description below, the pixels 51 of the first subset operate on infrared light, i.e. light having a wavelength in the range of 750 nm to 1000 μm, preferably in the range of 780 nm to 3 μm. The pixels 51 of the second subset operate on visible light, i.e. light having a wavelength between 400 nm and 700 nm.

[0169] To enable the collection or transmission of information beyond the observable scene (i.e. beyond the two-dimensional intensity distribution observable by the sensor device), a sensor device 10 is provided that detects such information using infrared-sensitive EVS pixels. Figure 15 shows such a sensor device 10 in a schematic manner.

[0170] As shown in FIG. 15, a sensor device 10 for observing a scene S comprises a number of pixels 51, each configured to receive light from the scene and perform a photoelectric conversion to generate an electrical signal. As described below, the pixels may be arranged in a two-dimensional pattern or pixel array 51a on a single die or wafer, but may also be provided on different stacked wafers. Furthermore, in the following, "pixel" also includes an arrangement in which multiple photoelectric conversion elements (such as photodiodes) are read out together to generate a single electrical signal. Referring to such an arrangement as a "pixel" in the singular form is not considered limiting and serves only for ease of explanation.

[0171] As described above, different pixels 51 may be particularly sensitive to a particular wavelength range. Thus, photoelectric conversion elements designed to have high sensitivity in the respective wavelength ranges are used in the pixels 51. In particular, photoelectric conversion elements sensitive to blue light (380 nm or 430 nm to 490 nm), green light (490 nm to 570 nm or 600 nm or 640 nm), red light (570 nm or 600 nm or 640 nm to 780 nm), or infrared light (780 nm to 3000 nm or 50000 nm or 1 mm) may be used. Alternatively or additionally, the wavelength sensitivity can be supported or generated by corresponding color filters.

[0172] 15, the pixels 51 are separated into at least two subsets S1, S2. The pixels 51 of the first subset S1 are sensitive to infrared light, and the pixels 51 of the second subset S2 are sensitive to at least visible light.

[0173] The sensor device 10 comprises an event detection circuit 20 configured to detect as event data intensity changes exceeding a predefined threshold in the infrared light received by each pixel 51 of the first subset S1. The pixels 51 of the first subset S1 are thus operated as EVS pixels via the event detection circuit 20, e.g. as described above with respect to Figures 5 to 7.

[0174] The sensor device 10 also comprises a pixel signal generating circuit 30 configured to generate a pixel signal indicative of an intensity value of visible light received by each pixel 51 of the second subset S2. The pixels 51 of the second subset S2 are thus operated by the pixel signal generating circuit 30 as APS pixels generating, for example, an RGB image at a given frame rate. Note that the pixel signal generating circuit 30 may also operate on signals supplied by the pixels 51 of the first subset S1, for example in a time division multiplexed manner.

[0175] The controller 40 of the sensor device 10 is then configured to extract additional information from the event data detected in the received infrared light, which additional information is different from the two-dimensional intensity information on the observed scene S. The controller 40 may be any type of processor, circuitry, hardware or software capable of performing the functions of the controller 40 described herein. The controller 40 may be located on the same wafer / chip / die as the pixel array 51a and / or readout circuits 20, 30, or may be located separately or externally.

[0176] The control unit 40 receives event data generated by the event detection circuit 20 based on the received infrared radiation. Specific information can be encoded in the infrared radiation that goes beyond the mere infrared intensity of the objects visible in the scene, i.e., referring to something other than the apparent location of the object O on the scene S.

[0177] For example, the sensor device 10 captures an RGB image of the scene S by a second subset S2 of pixels 51. Furthermore, the sensor device 10 captures event data, for example positive polarity events P and negative polarity events N. These events may form a specific spatial, temporal or spatio-temporal pattern that represents information encoded in the received infrared light. The control unit 40 is able to track, identify and decode this information from the detected events. Thus, the function of the control unit 40 goes beyond the mere registration or detection of a specific pattern of the received infrared light. Instead, the infrared (event) signal is understood to be a code carrying additional information and is decoded to obtain this additional information.

[0178] At the bottom of Fig. 15 an example of an image of an observed scene S at a certain point in time is shown. The image consists of RGB frames showing an object O in the scene S. There are overlapping positions of positive polarity events P and negative polarity events N. The positive polarity events P arise due to the appearance of a perpendicular infrared line illuminating an object in the scene, and the negative polarity events N arise due to the disappearance of such a line.

[0179] In conventional systems, the image obtained is simply an infrared overlay on an RGB image of the scene, as visualized in the lower part of Fig. 15. In contrast, the above-mentioned control unit 40 is able to recognize that a particular form of the infrared projection appearing and disappearing indicates the three-dimensional shape of an object O in the scene S. In this sense, the control unit 40 is configured to decode additional information different from the mere two-dimensional intensity information on the observed scene S.

[0180] In a similar manner, the control unit 40 can be configured to decode the narrow signal X encoded in the received infrared light as additional information. This is illustrated in FIG.

[0181] Figure 16 shows a series of RGB frames captured via a second subset S2 of pixels 51. The captured scene consists of a top view of a forklift F with an infrared emitter E on its roof. Overlaid on the RGB frames are a series of positive polarity events P and negative polarity events N resulting from changes in the intensity of the emitted infrared light. As shown in the bottom part of Figure 16, the sequence of negative polarity events N, positive polarity events P, and lack of events forms a signal X over time that can be used to provide the sensor device 10 with additional information beyond just an indication of the intensity of the light emitted from the infrared emitter E.

[0182] It should be noted that event detection is performed with much less latency, i.e., at a much higher frame rate, than RGB frame capture. Thus, the illustration of Figure 16 is simplified in that many events may be detected between RGB frames, greatly increasing the density of information that can be received via the sensor device 10 compared to when signal changes were only observable at the frame rate of the pixel signal generation circuit 30.

[0183] In particular, the signal X encoded on the received infrared light may be a communication signal, preferably generated by modulating the infrared light at a high frequency. As shown in Fig. 16, such an additional infrared communication channel can be used for monitoring objects such as forklifts, robots, etc. In this way, the RGB video stream can be supplemented with additional information, such as for example the identity of the monitored object, its operating state (idle or occupied), the task being performed, the operator, etc., without the need for additional communication equipment. The RGB information can be automatically acquired in synchrony with the additional information and combined into more detailed monitoring data.

[0184] A further application of signal coding by sensor devices is in the field of computer gaming, in particular in virtual reality applications, where infrared emitters can be provided in equipment worn by a user, such as virtual reality glasses / masks, providing a signal that allows for determining the location of the user, in addition to conveying additional information, such as the identity of the worn equipment or the user (especially if several users are in the same room), or via recognition of the Doppler shift relative movement of the user with respect to the sensor device 10. Thus, again, the RGB video channel can remain open for video calling or video streaming, and additional information can be transferred via infrared signaling.

[0185] As mentioned above, the sensor device 10 may be used to detect infrared patterns to infer a three-dimensional shape of an object O in a scene S. As shown in Fig. 17, the sensor device 10 may further comprise an infrared transmitter 50 for this purpose configured to emit a predetermined time-varying infrared pattern P into the scene S. The control unit 40 is configured to additionally generate depth information indicative of a distance of the object O in the scene S relative to the sensor device 10 based on a reflection R of the predetermined time-varying infrared pattern P on the scene S.

[0186] Thus, the sensor device 10, in addition to capturing an RGB image of the scene S, includes a structured light arrangement that makes it possible to reconstruct the distance between the observed part of the scene S and the sensor device 10. The principles of structured light are known to those skilled in the art, so only a brief repetition of these principles will be given here.

[0187] As shown diagrammatically in Figs. 17 and 18, an infrared transmitter 50, e.g. an infrared Vertical Cavity Surface Emitting Laser (VCSEL), an infrared photodiode array, etc., projects a known but time-varying pattern P, e.g. one or more vertical light lines whose position changes with time, onto a scene S. The pattern P is reflected by an object O in the scene. This reflection R forms a corresponding signal on the pixel array 51a of the sensor device 10. The change in position of the reflection R and the deviation from the shape of the original pattern P allows the distance to all visible parts of the scene S to be calculated, e.g. by triangulation, if a reference line between the infrared transmitter 50 and the pixel array 51a is known. This reference line can be determined by calibration at installation, if the infrared transmitter 50 and the pixel array 51a are located on different elements of the sensor device 10. Alternatively, it is fixed, if both parts are mounted in a fixed spatial relationship, e.g. in a common housing.

[0188] In this way, a three-dimensional representation of the scene S can be obtained. Since the event detection operates with a much shorter time constant than the intensity frame detection, multiple sweeps of the infrared pattern across the scene can be performed within one frame capture period of the APS pixels of the second subset S2. It is therefore possible to provide a conventional video stream with a three-dimensional representation of the observed scene S in real time.

[0189] As shown in FIG. 18, the sensor device 10 may have a communication connection 45, preferably cabled, between the infrared transmitter 50 and the control unit 40, which allows synchronization of the light projected by the infrared transmitter 50 and the event data detected by receiving infrared light. This allows the control unit 40 to know which particular infrared pattern P was projected on the scene S at what time and how it was distorted by reflections from the scene S. In this way, the accuracy of the depth reconstruction and thus the generated 3D model of the scene S is improved. Providing a wired connection for the communication connection 45 makes the synchronization fail-safe. On the other hand, if the infrared transmitter 50 and the pixel array 51a are provided in separate elements, wireless communication may be desirable.

[0190] 19 is a schematic and exemplary functional block diagram of a sensor device 10 having an infrared transmitter 50 in addition to a pixel array 51a. As explained above, the infrared transmitter 50 and the pixel array 51a can operate synchronously based on the communication connection 45. The pixel array transfers EVS data obtained by the event detection circuit 20 and RGB data obtained by the pixel signal generation circuit 30 to the control unit 40. The EVS data is supplied to a structured light process 40a executed by the control unit 40, which also controls the projection of an infrared pattern via the infrared transmitter 50.

[0191] The depth map established by the structured light process is provided together with the RGB data to a data fusion process 40b executed by the control unit 40, in which both sets of data are fused. Data fusion may be performed simply by combining corresponding data, for example by assigning a depth value to each RGB pixel value. However, more complex data fusion processes may also be applied, such as the establishment of a virtual three-dimensional model of the observed scene that can be used to re-render the observed scene from different viewpoints.

[0192] Based on the fused data, various processing can be performed. For example, the background subtraction process 60a can rely on an improved separation of foreground and background based on a distance threshold, i.e., keeping only pixel signals with depth values ​​smaller than the distance threshold.

[0193] Based on the 3D model of the observed person, an avatar of the person (or the person itself) may be rendered and controlled in Avatar process 60b. Similarly, a 3D model of the observed human's head may be generated in Viewpoint Correction process 60c to allow for virtually changing the viewpoint of the sensor device 10.

[0194] Here, the processes 60a, 60b, and 60c executed based on the fusion data may be executed by the control unit 40, or may be executed by an external device such as a personal computer as shown in FIG.

[0195] Fig. 20 is a diagram for explaining the principle of the viewpoint correction process 60c. As shown in Fig. 20, a human head H forms part of the observed scene. The person is concentrating on the content displayed on the screen 70 and is capturing its video stream using the sensor device 10. For example, the sensor device 10 is used as a camera for video conferencing.

[0196] As shown by the line of sight V of the person, the user's field of vision is directed toward the screen 70, but not toward the sensor device 10. Therefore, the person is not looking at the field of view F of the sensor device 10, and is looking away from the sensor device 10. Therefore, the other conference participants have the impression that the person is not looking at them, and is looking away.

[0197] From the sensor device 10, an infrared pattern is projected onto the head H as described above. The control unit 40 is configured to generate a virtual three-dimensional model M of the human head H based on pixel signals indicating intensity values ​​of visible light and on the depth information data. This is shown in the lower part of Figure 20, where a three-dimensional representation M of the head H is shown.

[0198] Based on the data of this three-dimensional representation M, the control unit 40 may be configured to generate modified image data that allows in particular to render the human head H on the display with the adjusted viewing direction. Thus, in the three-dimensional representation M of the scene S including the head H, the field of view F of the sensor device 10 is virtually shifted so that the viewing direction V is located within the field of view. The adjusted pixel signals may be automatically generated and output so that the sensor device 10 generates only images in which the person is looking into the sensor device 10. This allows the conference participants to talk to each other as if they were face to face, thus making the video conference more natural. Alternatively, the control unit may provide only the data necessary for the viewpoint correction, and perform the viewpoint correction on the computer of the data recipient.

[0199] In particular, when occluded parts of head H need to be reconstructed to render head H with adjusted gaze V, a 3D representation of the captured person can be generated using a neuronal network, as well as viewpoint correction.

[0200] While above possible applications of the sensor device 10 have been described, the following description is directed to the pixel and circuit structure of the sensor device 10. It should be noted that the specific realization of the sensor material, the pixels and / or the readout circuitry is in principle known to the person skilled in the art and in this sense is arbitrary. Therefore, in the following, the focus will be on the overall arrangement of the different parts of the sensor device 10, but details that are in principle known will be omitted as far as possible.

[0201] 21 to 24, the sensor device 10 may comprise, in addition to the plurality of pixels 51, a plurality of color filters 80 arranged such that at least visible light reaches the pixels 51 of the second subset S2 of pixels 51 and only infrared light reaches the pixels 51 of the first subset S1 of pixels 51. This allows additional information to be received at the event detection pixels of the first subset S1 without interference by visible light and thus ensures good transmission quality.

[0202] 21, the pixels 51 may be arranged in a pixel array 51a having a plurality of 2×2 pixel groups each including one pixel 51 of a first subset S1 of the pixels 51 and three pixels 51 of a second subset S2 of the pixels 51. Thereby, a plurality of color filters 80 are arranged in the color filter array 80a having red (R), green (G), blue (B) and infrared (IR) filters, each 2×2 pixel group being assigned one red (R), one green (G), one blue (B) and one infrared (IR) filter, and an infrared filter (IR) is assigned to the pixels 51 of the first subset S1 of the pixels 51. The color filters 80 may in particular be arranged in a modified Bayer pattern, in which one green filter is replaced by an infrared filter in each 2×2 color filter block.

[0203] This arrangement ensures that each 2x2 pixel group receives RGB information and can detect events in the infrared channel. Thus, the spatial resolution of the RGB signal and the event data is comparable. Also, the pixels 51 of the first subset S1 can be read out in a shared or time-multiplexed manner by a pixel single generation circuit [Yuhei 4][Yuhei 5] 20 to complement the RGB channels with the infrared channel. Here, similar to the EVS pixels of the first subset S1, the APS pixels of the second subset S2 may also be designed to be particularly sensitive to wavelengths transmitted by the corresponding color filters, as shown in FIG. 21 by the color filters 80 and the corresponding textures of the pixels 51.

[0204] Another possible embodiment of the arrangement of the pixels 51 and the color filters 80 is shown in Figs. 22 to 24. Here, the pixels 51 are arranged in groups of five pixels 51 on the first wafer W1, with one pixel 51 of the first subset S1 being arranged with respect to the light incidence direction behind the 2x2 pixel 51 of the second subset S2. Thus, the APS pixel of the second subset S2 is arranged on the light incidence surface of the first wafer W1, and one EVS pixel of the first subset S1 is arranged on the opposite side of the first wafer W1. The area that can be occupied by the pixels 51 of the first subset S1 then corresponds to four times the area that can be occupied by one pixel 51 of the second subset S2. This can be advantageous on the one hand to compensate for the increase in the size of the pixels 51 designed for receiving infrared light, and on the other hand to improve the sensitivity due to the increase in the pixel area.

[0205] Moreover, in this case, it is possible to arrange a number of color filters 80 in a color filter array 80a, preferably arranged in a Bayer pattern, with red filters (R), green filters (G) and blue filters (B), which are infrared transparent. In particular, there is no need to add an infrared filter. Infrared light passes through the color filter 80 as well as the pixels 51 of the second subset S2, but visible light is absorbed by the color filter 80 and the pixels 51 of the second subset S2 before reaching the pixels 51 of the second subset S2. Thus, a conventional color filter array 80a can be used for the sensor device 10, which makes it easier to manufacture.

[0206] In all the examples of Figures 21 to 24 there is provided an additional layer which includes an event detection circuit 20 and a pixel signal generation circuit 30. These circuits may be provided elsewhere but due to space constraints it is preferred to provide them stacked as shown in Figures 21 to 24. The connections between the pixel array 51a and the event detection circuit 20 or the pixel signal generation circuit 30 can be provided by Cu-Cu connections 90, by through silicon vias (TSV) or by any other method known in principle to the person skilled in the art.

[0207] In all of the exemplary illustrated examples, the sensor device 10 includes a first wafer W1 on which a plurality of pixels 51 are formed, where a color filter array 80a may be present as described above, or may be omitted if the pixels 51 are designed to be sensitive only to a particular wavelength range that would otherwise be limited by the color filter 80.

[0208] In addition to the first wafer W1, at least one further wafer W2, W3 is provided in a stack, on which the event detection circuit 20 and the pixel signal generation circuit 30 are formed.

[0209] Here, the event detection circuit 20 is formed from a plurality of first circuit blocks 20a, the pixel signal generation circuit 30 is formed from a plurality of second circuit blocks 30, and the arrangement of the first circuit blocks 20a and the second circuit blocks 30a on at least one further wafer W2, W3 corresponds to the arrangement of the plurality of pixels 51 on the first wafer W1. Thus, when viewed in the direction of the incident light L, each pixel 51 covers an area behind which a corresponding readout circuit is arranged. This facilitates the connection between the pixels 51 and the readout circuits 20, 30, which not only facilitates the manufacture of the sensor device 10 but also allows a more compact structure to be realized.

[0210] For example, all pixels 51 may be formed on the side of the first wafer W1 facing the incident light L, i.e., backside illumination structure, as shown in FIG. 21. The readout circuits 20, 30 having the first circuit block 20a and the second circuit block 30a are also formed on the same side of the second wafer W2, i.e., the light incident side. The first circuit block 20a is formed at a position below / behind the EVS pixels of the first subset S1 (below / behind the infrared (IR) color filter 80, if present). Similarly, the second circuit block 30a forming the pixel signal generation circuit 30 is formed below / behind the APS pixels of the second subset S2 (below / behind the RGB color filter 80, if present). Thus, all pixels 51 can be directly connected to the corresponding circuit block 20a, 30a, for example, via Cu-Cu connections 90.

[0211] According to another example shown in Fig. 22, each pixel 51 of the second subset S2 of pixels 51 is formed on the side of the first wafer W1 facing the incident light L, and each pixel 51 of the first subset S1 of pixels 51 is formed on the opposite side of the first wafer W1. Thus, as mentioned above, a two-sided mounting is chosen, e.g. a mounting in which a 2x2 APS pixel is placed on top of one (larger) EVS pixel.

[0212] Two further wafers W2, W3 are provided, one adjacent to the first wafer W1 carrying the second circuit block 30a and the other carrying the first circuit block 20a. The second wafer W2 and the third wafer W3 are therefore arranged such that the arrangement of the circuit blocks 20a, 30a resembles the arrangement of the corresponding pixels 51. As shown in FIG. 22, the pixels 51 of the second subset S2 and the second circuit block 30a are arranged in a corresponding pattern such that the first and second connections 95, 96 are guided between the pixels 51 of the first subset S1 from the second subset S2 of pixels 51 to the second circuit block 30a. Similarly, the pixels 51 of the first subset S1 also have a one-to-one correspondence with the first circuit block 20a. Here, the third connections 97 are guided between the second circuit block 30a from the pixels 51 of the first subset S1 to the first circuit block 20a.

[0213] This arrangement is more complicated in terms of the connection structure, but has the advantage that the event detection circuit 20 and the pixel signal generation circuit 30 can be formed on different wafers, making manufacturing easier. Furthermore, the event detection circuit 20 can be formed in a first circuit block 20a that is larger than the example in which the entire readout circuits 20, 30 are formed on one side of a single wafer. This increases the degree of freedom in designing the event detection circuit 20.

[0214] The example shown in Fig. 23 corresponds to the example shown in Fig. 22, except that a single second wafer W2 is used for the readout circuit. Here, the first circuit block 20a and the second circuit block 30a are formed on two opposite sides of this second wafer W2, the side carrying the second circuit block 20a being adjacent to the first wafer W1. All other arrangements correspond to those described above with reference to Fig. 22. The advantage of providing the event detection circuit 20 and the pixel signal generation circuit 30 on opposite sides of a single wafer W2 is improved compactness in the stacking direction, but at the expense of more complicated manufacturing steps.

[0215] In another example shown in Figure 24, the same pixel structure is provided as described with respect to Figures 22 and 23. That is, the pixels 51 of the second subset S2 are formed on a side of the first wafer W1 facing the incident light L, and the pixels 51 of the first subset S1 are formed on the opposite side of the first wafer W1. However, now the first circuit block 20a and the second circuit block 30a are formed on the same side of the second wafer W2, such that the first circuit block 20a is surrounded by the second circuit block 30a.

[0216] 24, a first circuit block 20a connected to the corresponding EVS pixel above is disposed in the central region of each 2×2 block of the second circuit block 30a corresponding to the RGB pixel 51 disposed thereon. This allows the entire readout circuit 20, 30 to be formed on one wafer. This facilitates the manufacturing and connection to the pixel wafer W1. At the same time, a sufficient degree of freedom regarding the size of the first circuit block 20a is maintained. In this way, a highly functional and compact layout of the sensor device 10 can be obtained.

[0217] It should be understood that all the above arrangements are purely exemplary. What is essential in these examples is the positional correspondence between the APS pixels and the corresponding second circuit blocks 30a, and between the EVS pixels and the corresponding first circuit blocks 20a. It should be noted that, among other things, in the above examples the correspondence is described such that the circuit blocks 20a, 30a are located directly below / behind the corresponding pixels 51, but a lateral offset between these components may also occur. Preferably, the offset from a complete overlap still allows a vertical connection between the pixels 51 and the corresponding blocks 20a, 30a of the readout circuits 20, 30.

[0218] Various implementations of the basic idea of ​​using infrared-sensitive EVS pixels to receive additional infrared-encoded information have been described above. The basic method underlying all of these implementations is summarized below with reference to FIG.

[0219] Here, in S101, light from a scene S is received and photoelectrically converted to generate an electrical signal by each of a plurality of pixels 51 of the sensor device 10 used to observe the scene.

[0220] In S102, the event detection circuit 20 of the sensor device 10 detects a change in intensity of the infrared light received by each of the first subset S1 of the pixels 51 that exceeds a predetermined threshold as event data.

[0221] In S103, pixel signals indicative of the intensity value of the visible light received by each pixel 51 of the second subset S2 of pixels 51 are generated by the pixel signal generating circuit 30 of the sensor device 10.

[0222] In S104, additional information different from the two-dimensional intensity information of the observed scene S is extracted from the event data detected within the received infrared light.

[0223] In this way, it is possible to transmit additional information, such as the three-dimensional shape of the observed object, its identity, its task, etc., without the need for additional communication equipment. This expands the fields in which the sensor device 10 can be used.

[0224] Furthermore, the functionality of the sensor device 10 can simplify existing devices by making additionally required devices unnecessary. For example, the sensor device 10 can be used in a head-mounted display used for augmented / virtual reality applications to eliminate a separate depth sensor. This obviously reduces the complexity and cost of the head-mounted display, while eliminating the image registration process between the camera and the depth sensor, and thus the required processing power.

[0225] 26 is a block diagram showing an example of a schematic configuration of a smartphone 900 that can constitute the above-mentioned sensor device 10 or can be the sensor device 10. The smartphone 900 includes a processor 901, a memory 902, a storage unit 903, an external connection interface 904, a camera 906, a sensor 907, a microphone 908, an input device 909, a display device 910, a speaker 911, a wireless communication interface 912, one or more antenna switches 915, one or more antennas 916, a bus 917, a battery 918, and an auxiliary controller 919.

[0226] The processor 901 may be, for example, a CPU or a system on chip (SoC), and controls the functions of the application layer and other layers of the smartphone 900. The memory 902 includes a RAM and a ROM, and stores programs executed by the processor 901 together with data. The storage unit 903 may include a storage medium such as a semiconductor memory or a hard disk. The external connection interface 904 is an interface for connecting an external connection device such as a memory card or a USB (Universal Serial Bus) device to the smartphone 900. The processor may function as the control unit 40.

[0227] The camera 906 includes an image sensor as described above. The sensor 907 may include a group of sensors such as a position sensor, a gyro sensor, a geomagnetic sensor, and an acceleration sensor. The microphone 908 converts a voice input to the smartphone 900 into an audio signal. The input device 909 includes a touch sensor that detects a touch on the screen of the display device 910, a keypad, a keyboard, a button, a switch, or other device, and accepts operations or information input from a user. The display device 910 includes a screen such as a liquid crystal display (LCD) or an organic light emitting diode (OLED), and displays an output image of the smartphone 900. The speaker 911 converts an audio signal output from the smartphone 900 into audio.

[0228] The wireless communication interface 912 supports a cellular communication method such as LTE or LTE-Advanced, and performs wireless communication. Typically, the wireless communication interface 912 may include a BB processor 913, an RF circuit 914, and the like. The BB processor 913 may perform various signal processing for wireless communication, such as encoding / decoding, modulation / demodulation, multiplexing / demultiplexing, and the like. On the other hand, the RF circuit 914 may include components such as a mixer, a filter, an amplifier, and the like, and transmits and receives wireless signals via an antenna 916. The wireless communication interface 912 may be a one-chip module in which the BB processor 913 and the RF circuit 914 are integrated. The wireless communication interface 912 may include a plurality of BB processors 913 and a plurality of RF circuits 91. Although FIG. 26 shows an example of the wireless communication interface 912 including a plurality of BB processors 913 and a plurality of RF circuits 914, the wireless communication interface 912 may include a single BB processor 913 or a single RF circuit 914.

[0229] Furthermore, the wireless communication interface 912 may support other types of wireless communication schemes, such as a short-range wireless communication scheme, a near field wireless communication scheme, a wireless local area network (LAN) scheme, etc. In this case, a BB processor 913 and an RF circuit 914 may be included for each wireless communication scheme.

[0230] Each antenna switch 915 switches the destination of the antenna 916 among multiple circuits (eg, circuits for different wireless communication methods) included in the wireless communication interface 912.

[0231] Each antenna 916 includes a single or multiple antenna elements (for example, multiple antenna elements constituting a MIMO antenna) and is used for transmitting and receiving wireless signals via the wireless communication interface 912. The smartphone 900 may also include multiple antennas 916 as shown in Fig. 26. Note that although an example of the smartphone 900 including multiple antennas 916 is illustrated in Fig. 26, the smartphone 900 may also include a single antenna 916.

[0232] Furthermore, the smartphone 900 may include an antenna 916 for each wireless communication system. In this case, the antenna switch 915 may be omitted from the configuration of the smartphone 900.

[0233] The bus 917 interconnects the processor 901, the memory 902, the storage unit 903, the external connection interface 904, the camera 906, the sensor 907, the microphone 908, the input device 909, the display device 910, the speaker 911, the wireless communication interface 912, and the auxiliary controller 919. The battery 918 supplies power to each block of the smartphone 900 illustrated in Fig. 26 via a power supply line partially illustrated by a dashed line in the figure. The auxiliary controller 919 operates the minimum functions of the smartphone 900, for example, during a sleep mode.

[0234] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0235] Additionally, the advantages described herein are exemplary only and not limiting, as other advantages may be provided.

[0236] The present technology can also be configured as follows.

[0237] (1) A sensor device for observing a scene, comprising: a plurality of pixels, each configured to receive light from the scene and perform a photoelectric conversion to generate an electrical signal; an event detection circuit configured to detect an event in a data intensity of infrared light received by each of the first subset of pixels that exceeds a predetermined threshold; a pixel signal generating circuit configured to generate a pixel signal indicative of an intensity value of visible light received by each pixel of the second subset of pixels; a control unit configured to extract additional information from the event data detected within the received infrared light, the additional information being different from two-dimensional intensity information regarding the observed scene; Equipped with Sensor device. (2) The sensor device according to (1), The control unit is configured to decode a signal encoded in the received infrared light as additional information. Sensor device. (3) The sensor device according to (2), The signal encoded on the received infrared light is preferably a communication signal generated by modulating the infrared light at a high frequency. Sensor device. (4) The sensor device according to any one of (1) to (3), further comprising: an infrared transmitter configured to emit a predetermined time-varying infrared pattern into the scene; Equipped with The control unit is configured to generate depth information indicating a distance from an object in the scene to the sensor device based on reflection of the predetermined time-varying infrared pattern in the scene as additional information. Sensor device. (5) The sensor device according to (4), further comprising: A preferably cabled communication connection between the infrared transmitter and the control unit, enabling synchronization of the emission of light from the infrared transmitter with the event data detected by receiving infrared light. Equipped with Sensor device. (6) The sensor device according to (4) or (5), If a human head forms part of the observed scene, the control unit is configured to generate, based on the pixel signals indicative of the intensity values ​​of the visible light and the depth information, data of a virtual three-dimensional model of the human head, preferably enabling the human head to be rendered on a display at an adjusted viewing direction. Sensor device. (7) The sensor device according to any one of (1) to (6), further comprising: a plurality of color filters arranged such that at least visible light reaches the pixels of the second subset of pixels and only infrared light reaches the pixels of the first subset of pixels; Equipped with Sensor device. (8) The sensor device according to (7), the pixels are arranged in a pixel array having a plurality of 2×2 pixel groups each including one pixel of the first subset of pixels and three pixels of the second subset of pixels; The plurality of color filters are arranged in a color filter array having red, green, blue and infrared filters, and each of the 2×2 pixel groups is assigned one red, one green, one blue and one infrared filter with the infrared filters assigned to the pixels of the first subset of pixels. Sensor device. (9) The sensor device according to (7), the pixels are arranged in groups of 5 pixels with one pixel of the first subset of pixels being arranged with respect to a direction of incidence of light behind 2×2 pixels of the second subset of pixels; The plurality of color filters are arranged in a color filter array having red, green and blue filters, preferably arranged in a Bayer pattern, and the color filters are infrared transmissive. Sensor device. (10) The sensor device according to any one of (1) to (9), further comprising: a first wafer on which the plurality of pixels are formed; at least one further wafer on which the event detection circuitry and the pixel signal generation circuitry are formed; Equipped with the event detection circuit is formed from a plurality of first circuit blocks; the pixel signal generation circuit is formed from a plurality of second circuit blocks; An arrangement of the first circuit block and the second circuit block on the at least one further wafer corresponds to an arrangement of the plurality of pixels on the first wafer. Sensor device. (11) The sensor device according to (10), all pixels are formed on a side of the first wafer facing the incident light; The first circuit block and the second circuit block are formed on the same side of a second wafer. Sensor device. (12) The sensor device according to (10), the pixels of the second subset of pixels are formed on a side of the first wafer facing the incident light and the pixels of the first subset of pixels are formed on an opposite side of the first wafer; Two further wafers are provided, one adjacent to the first wafer carrying the second circuit block and the other wafer carrying the first circuit block. Sensor device. (13) The sensor device according to (10), the pixels of the second subset of pixels are formed on a side of the first wafer facing the incident light and the pixels of the first subset of pixels are formed on an opposite side of the first wafer; The first circuit block and the second circuit block are formed on two opposing sides of a second wafer, the side carrying the second circuit block being adjacent to the first wafer. Sensor device. (14) The sensor device according to (10), the pixels of the second subset of pixels are formed on a side of the first wafer facing the incident light and the pixels of the first subset of pixels are formed on an opposite side of the first wafer; The first circuit block and the second circuit block are formed on the same side of a second wafer such that the first circuit block is surrounded by the second circuit block. Sensor device. (15) A method of operating a sensor device to observe a scene, comprising: receiving light from the scene and performing photoelectric conversion on each of a plurality of pixels of the sensor device to generate an electrical signal; detecting, by an event detection circuit of the sensor device, a data intensity of the infrared light received by each of the first subset of pixels that exceeds a predetermined threshold as an event; generating, by a pixel signal generating circuit, pixel signals indicative of intensity values ​​of visible light received by each pixel of the second subset of pixels; Extracting additional information from the event data detected within the received infrared light that is different from two-dimensional intensity information about the observed scene. A method of operation of a sensor device that observes a scene.

Claims

1. A sensor device (10) for observing a scene, comprising: a plurality of pixels (51) each configured to receive light from the scene and perform a photoelectric conversion to generate an electrical signal; an event detection circuit (20) configured to detect, as an event, a data intensity of infrared light received by each of a first subset (S1) of the pixels (51) that exceeds a predetermined threshold; a pixel signal generating circuit (30) configured to generate a pixel signal indicative of an intensity value of visible light received by each pixel (51) of the second subset (S2) of pixels (51); a control unit (40) configured to extract additional information from the event data detected in the received infrared light, different from two-dimensional intensity information about the observed scene; Equipped with A sensor device (10).

2. A sensor device (10) according to claim 1, The control unit (40) is configured to decode a signal (X) encoded in the received infrared light as additional information. A sensor device (10).

3. A sensor device (10) according to claim 2, The signal (X) encoded on the received infrared light is preferably a communication signal generated by modulating the infrared light at a high frequency. A sensor device (10).

4. The sensor device (10) of claim 1, further comprising: an infrared transmitter (50) configured to emit a predetermined time-varying infrared pattern (P) into said scene; Equipped with The control unit (40) is configured to generate depth information indicating a distance from an object (O) in the scene to the sensor device (10) based on the reflection (R) of the predetermined time-varying infrared pattern (P) in the scene as additional information. A sensor device (10).

5. A sensor device (10) according to claim 4, further comprising: A communication connection (45), preferably cabled, between the infrared transmitter (50) and the control unit (40) that allows synchronization of the emission of light from the infrared transmitter (50) and the event data detected by receiving infrared light. Equipped with A sensor device (10).

6. A sensor device (10) according to claim 4, If a human head (H) forms part of the observed scene, the control unit (40) is configured to generate, based on the pixel signals representative of the intensity values ​​of the visible light and the depth information, data of a virtual three-dimensional model (M) of the human head, preferably enabling the human head to be rendered on a display at an adjusted viewing direction. A sensor device (10).

7. The sensor device (10) of claim 1, further comprising: a plurality of color filters (80) arranged such that at least visible light reaches the pixels (51) of the second subset (S2) of the pixels (51) and only infrared light reaches the pixels (51) of the first subset of the pixels (S1); Equipped with A sensor device (10).

8. A sensor device (10) according to claim 7, the pixels (51) are arranged in a pixel array (51a) having a plurality of 2×2 pixel groups each including one pixel (51) of the first subset (S1) of the pixels (51) and three pixels (51) of the second subset (S2) of the pixels (51); The plurality of color filters (80) are arranged in a color filter array (80a) having red (R), green (G), blue (B) and infrared (IR) filters, and each of the 2×2 pixel groups is assigned one red (R), one green (G), one blue (B) and one infrared (IR) filter, with the infrared filter (IR) assigned to the pixels (51) of the first subset (S1) of the pixels (51). A sensor device (10).

9. A sensor device (10) according to claim 7, the [Yuhei 8] pixels (51) are arranged in groups of 5 pixels (51) with one pixel (51) of the first subset (S1) of the pixels (51) being arranged with respect to the direction of incidence of light behind a 2×2 pixel (51) of the second subset (S2) of the pixels (51); The plurality of color filters (80) are arranged in a color filter array (80a) having red filters (R), green filters (G), and blue filters (B), preferably arranged in a Bayer pattern, and the color filters (80) are infrared transparent. A sensor device (10).

10. The sensor device (10) of claim 1, further comprising: a first wafer (W1) on which the plurality of pixels (51) are formed; at least one further wafer (W2, W3) on which the event detection circuit (20) and the pixel signal generation circuit (30) are formed; Equipped with The event detection circuit (20) is formed from a plurality of first circuit blocks (20a); The pixel signal generating circuit (30) is formed from a plurality of second circuit blocks (30a), An arrangement of the first circuit block (20a) and the second circuit block (30a) on the at least one further wafer (W2, W3) corresponds to an arrangement of the plurality of pixels (51) on the first wafer (W1). A sensor device (10).

11. A sensor device (10) according to claim 10, All pixels (51) are formed on the side of the first wafer (W1) facing the incident light (L), The first circuit block (20a) and the second circuit block (30a) are formed on the same side of a second wafer (W2). A sensor device (10).

12. A sensor device (10) according to claim 10, the pixels (51) of the second subset (S2) of pixels (51) are formed on a side of the first wafer (W1) facing the incident light (L) and the pixels (51) of the first subset (S1) of pixels (51) are formed on an opposite side of the first wafer (W1), Two further wafers (W2, W3) are provided, one adjacent to the first wafer (W1) carrying the second circuit block (30a) and the other carrying the first circuit block (20a). A sensor device (10).

13. A sensor device (10) according to claim 10, the pixels (51) of the second subset (S2) of pixels (51) are formed on a side of the first wafer (W1) facing the incident light (L) and the pixels (51) of the first subset (S1) of pixels (51) are formed on an opposite side of the first wafer (W1), The first circuit block (20a) and the second circuit block (30a) are formed on two opposing sides of a second wafer (W2), and the side carrying the second circuit block (20a) is adjacent to the first wafer (W1). A sensor device (10).

14. A sensor device (10) according to claim 10, the pixels (51) of the second subset (S2) of pixels (51) are formed on a side of the first wafer (W1) facing the incident light (L) and the pixels (51) of the first subset (S1) of pixels (51) are formed on an opposite side of the first wafer (W1), The first circuit block (20a) and the second circuit block (30a) are formed on the same side of a second wafer (W2) such that the first circuit block (20a) is surrounded by the second circuit block (30a). A sensor device (10).

15. A method of operating a sensor device (10) for observing a scene, comprising the steps of: receiving light from the scene by each of a plurality of pixels (51) of the sensor device (10) and performing photoelectric conversion to generate an electrical signal; An event detection circuit (20) of the sensor device (10) detects an event when a data intensity of the infrared light received by each of the first subset (S1) of the pixels (51) exceeds a predetermined threshold value; generating, by a pixel signal generating circuit (30), a pixel signal indicative of an intensity value of visible light received by each pixel (51) of a second subset (S2) of said pixels (51); Extracting additional information from the event data detected within the received infrared radiation that is different from two-dimensional intensity information about the observed scene. A method of operating a sensor device (10) for observing a scene.

Images