Image sensor and sensor device for detecting time-dependent image data

Abstract

The invention relates to an image sensor for detecting time-dependent image data, comprising a plurality of photovoltaic converters (1) and a plurality of electronic converters (2) arranged in an array and connected via switching elements (tx1, tx2, ...), whereby each of the photovoltaic converters (1) and one of the electronic converters (2) form a pair, the pair generating digital information dependent on the light intensity on the photovoltaic converters (1), and wherein the switching elements (tx1, tx2, ...) are arranged to selectively connect at least two of the photovoltaic converters (1) to one of the electronic converters (2), and at least two of the electronic converters (2) to one of the photovoltaic converters (1). Furthermore, the invention relates to a sensor device comprising an image sensor and a switching controller for controlling the switching elements, and a method for obtaining image data compensated for gain mismatch.

Description

Technical Field

[0001] The invention relates to an image sensor and a sensor arrangement for detecting time-dependent image data. Background Art

[0002] Modern image sensors usually consist of an array of photovoltaic converters connected to electronic converters. While the photovoltaic converters convert the incident light into corresponding analog electrical signals, the electronic converters further convert these analog signals into digital signals with the help of analog-to-digital converters, also referred to as encoders in the following text.

[0003] In US 7,728,269 B2 a type of image sensor is proposed which encodes the temporal visual contrast of the scene captured by its photoelectric sensor. By encoding the temporal contrast, the temporal redundancy of the image sensor output data is almost eliminated, thereby generating activity-driven sparse data in the format of ON / OFF events. This means that each event consists of the symbol ON / OFF and the pixel coordinates. However, the device proposed in US 7,728,269 B2 cannot capture any temporally static scene information.

[0004] To address the problem of capturing temporally static scene information, one might turn to US9631974 or to asynchronous temporal image sensors (ATIS) (Posch et al., 2010), which describe static scene information captured by an auxiliary dedicated intensity measurement circuit that produces a second output stream that linearly encodes the static scene. This two-stream approach increases the complexity of the sensor and results in two outputs that bear little resemblance to each other: one output encodes the temporal difference in the logarithmic value of the incident light intensity, thereby encoding temporal visual contrast, while the other output encodes the linear absolute light incident intensity.

[0005] Known techniques describe techniques for encoding static scenes logarithmically. For example, according to "Logarithmic response CMOS image sensor with on-chip calibration" (Kavidias et al., 2000), the photocurrent generated by each photodiode is logarithmically converted to a voltage by exploiting the physical properties of subthreshold transistors. The logarithmic voltage is then measured. However, even after calibration, the logarithmic conversion accuracy is affected by transistor mismatch. According to US8363140, logarithmic conversion is achieved by combining a logarithmic digital counter with an in-pixel single slope analog-to-digital converter. However, in this design, the pixels are too complex to be integrated into a time contrast pixel.

[0006] The concept of encoding static scenes by local spatial contrast was proposed in US6828540, which describes an image sensor system that uses mechanically vibrating optics to convert the spatial contrast of the scene into the temporal variance of the incident light to the pixel and encodes this temporal variance using a pulse density modulation based scheme. However, mechanically vibrating optics require additional mechanical parts that are not widely used in today's state-of-the-art image sensor systems. Mechanically vibrating optics require other components to provide precise positioning and timing control. Mechanical vibrations also result in each pixel having a circular scan pattern that does not match the orthogonal pixel arrangement, thus making post-processing of the image less accurate.

[0007] The perception of local spatial contrast can also be achieved by sensors described in US7170043 and "100000fps Vision Sensor with Embedded 535GOPS / W 256x256 SIMD Processor Array" (Carey et al., 2013). However, both of these pixel designs require relatively complex in-pixel circuitry to calculate local spatial contrast and are therefore not suitable for integration into temporal contrast pixels.

[0008] Finally, US2016 / 0093273 A1 describes an image sensor in which a cluster of NxN photosensors, each comprising a photodiode and a transducer, shares a cluster-specific differentiator, which is an analog-to-digital encoding circuit. US2016 / 0093273 A1 aims to increase the photodiode density on the chip by time-multiplexing the connections between different photosensors and encoders without increasing the encoder circuit area. The sensor described in US2016 / 0093273 does not encode static scenes. Summary of the invention

[0009] The object of the present invention is to propose a device that can obtain temporal and spatial visual contrast of a scene while reducing the use of resources, especially the use of chip area. Another object of the present invention is to provide more accurate positioning and timing control to support the removal of gain and offset mismatches in post-processing of image signals.

[0010] According to one aspect of the invention, the image sensor comprises a plurality of photovoltaic converters and a plurality of electronic converters. The photovoltaic converters are arranged in an array, and the electronic converters are arranged in an array. Each of these arrays can be, for example, a one-dimensional array or a two-dimensional array in any form, in particular a square, hexagonal or triangular array. The photovoltaic converters and the electronic converters are linked by a switching element configured to selectively connect the photovoltaic converters to the electronic converters.

[0011] In the following, the fact that the switching element is arranged and configured to selectively connect a specific photovoltaic converter to a specific electronic converter can be expressed as a specific photovoltaic converter. It is connected or can be connected or can be connected to a certain electronic converter through a switching element. Therefore, according to the present invention, each photovoltaic converter is connected to the electronic converter through a switching element, thereby forming a pair, which generates digital information according to the intensity of light incident on the photovoltaic converter. In the image sensor, at least two photovoltaic converters are connected to one of the electronic converters through a switching element. In addition, at least two of the electronic converters are connected to one of the photovoltaic converters through a switching element. Through the switching element, the electronic converter is connected to the photovoltaic converter, or the photovoltaic converter is connected to the electronic converter, which means that depending on the switching state of the switching element, the signal generated by the photovoltaic converter will reach or will not reach the electronic converter. In the former case, it can be said that the photovoltaic converter is connected to the electronic converter through the switching element.

[0012] It should be noted that connection in this context means that there is an electrically conductive connection. Two photovoltaic converters may be connected or connectable to one electronic converter via a switching element, which means that, depending on the configuration of the switching element, the signal from one or the other of the two photovoltaic converters reaches the electronic converter. Similarly, two electronic converters may be connected or connectable to one photovoltaic converter via a switching element, which means that, depending on the configuration of the switching element, the signal from the photovoltaic converter will be directed to one or the other of the two electronic converters.

[0013] As mentioned above, the image sensor described in US 7,728,269 B2 is capable of encoding the temporal contrast of the viewed image. By appropriately driving the switching element, for example by allowing signals from different photovoltaic converters to reach one electronic converter in succession, the spatial contrast can be converted into a temporal contrast, thereby allowing the spatial contrast of the image to be obtained. In other words, depending on the correct control of the switching element, the image sensor can output a temporal image contrast or a spatial image contrast.

[0014] As described above, the image sensor described in US2016 / 0093273 A1 includes a cluster of NxN photoreceptors that share cluster-specific encoding circuits. Therefore, there is an NxN to one mapping between photoreceptors and encoders. The encoder is cluster-specific, which means that the clusters do not overlap in terms of connection to the encoder. At any time, only a small fraction of the total number of photodiodes is active, i.e., encoded. In contrast, in the image sensor according to the present invention, multiple photoelectric converters are connected to multiple electronic converters through switching elements. The purpose is to allow each electronic converter to "observe" or receive signals from different photoelectric converters so that during transitions, they can encode spatial contrast. Although only one of the multiple photovoltaic converters can be actively connected to one electronic converter, the other photovoltaic converters can be connected to another electronic converter. More importantly, over time, the "field of view" of each electronic converter overlaps with each other, so that the spatial contrast of the entire scene can be continuously encoded without gaps. This continuity assumption is also used to identify and compensate for any non-uniformity of the converter pairs, which is due to the inevitable physical limitations of the semiconductor manufacturing process, so that the imaging scene can be better reconstructed.

[0015] A photovoltaic converter is a converter that performs a photon to electron conversion. The photovoltaic converter may be a photodiode, in particular a pinned photodiode (PPD). It may be configured to generate a current, i.e. a photocurrent, that depends, in particular linearly or proportionally, on the intensity of light incident on the photovoltaic converter. In contrast, an electronic converter is so named because it only performs a conversion between different electronic signals or an electron to electron conversion. In the following description, electronic converters may be referred to as pixels instead of photovoltaic converters. In particular, if the electronic converter comprises a signal converter and an encoder, each signal converter / encoder pair may be considered as a pixel of an image sensor.

[0016] The photovoltaic converter may also include another electronic signal converter, which may be a current-current, current-voltage, voltage-current or voltage-voltage converter. The output of the signal converter may depend on the output of its input signal linearly, logarithmically or according to some other function. Thus, in this case, the output of the photovoltaic converter will depend on the light intensity input signal linearly, logarithmically or according to some other function. The function may in particular be a compression function, such as a logarithmic function, so as to allow a larger range of inputs (e.g. corresponding to sixty light intensities) to be compressed into a relatively small electronic signal range (e.g. 100 mV).

[0017] Alternatively or cumulatively, the electronic signal converter or the additional electronic signal converter may be part of the electronic converter. In this embodiment, the output of the electronic signal converter depends linearly, logarithmically or according to some other function on the output of the photovoltaic converter. Therefore, in an advantageous embodiment, the electronic converters of the pair comprise an electronic signal converter, which in combination with the photovoltaic converters of the pair is configured to generate an analog electronic signal that depends on the light intensity on the photovoltaic converter. As an example, the photovoltaic converter performs a conversion of photons to electrons, thereby generating an electric current. This current is converted into a voltage in a logarithmic current-voltage converter, which is part of the electronic converter. Then, using an encoder that is also part of the electronic converter, this analog voltage signal that is logarithmically dependent on the incident light is converted into a digital signal.

[0018] Although it is possible to connect two signal converters in series as part of two photovoltaic converters to a single electronic converter (i.e., an analog-to-digital converter) containing an encoder, this embodiment has the following disadvantages: each signal converter introduces noise. Signal converters are usually not identical and show gain and offset mismatches. Therefore, if different signal converters are connected in series to one encoder, the spatial contrast encoded in the digital encoder signal will contain gain and offset mismatches, which can no longer be eliminated from the digital signal. The advantage of placing the signal converter inside the electronic converter is that the two signals from the two photovoltaic converters are now sent in series to a single electronic converter containing the signal converter and the encoder. Now, although the encoded spatial contrast still contains the gain mismatch of the signal converter, since any two adjacent signal converters have converted a common relative spatial difference, the gain mismatch can be compensated later. In addition, there is no longer an offset mismatch because the relative spatial difference between the two photovoltaic converters originates from the same signal converter.

[0019] According to an advantageous embodiment, the encoder in the electronic converter is configured to convert the analog electronic signal into the digital signal using pulse density modulation. An analog-to-digital encoder based on pulse density modulation has the advantage of being compact enough so that it can be implemented in a pixel-parallel manner so that each pixel contains one such analog-to-digital encoder element.

[0020] According to a preferred embodiment, the image sensor is made of two semiconductor chip dies, which are manufactured separately and connected by an interconnection. In particular, the signal converter, which can be part of a photovoltaic converter or part of an electronic converter, and the encoder, which is part of the electronic converter, can be arranged on two separate semiconductor chip dies during the manufacturing process and then electrically connected by the interconnection between the two semiconductor chip dies.

[0021] Preferably, the switching elements are arranged and configured to time-multiplex the photovoltaic converters to the input of the electronic converter. This means that during a first time interval, the output signal of one photovoltaic converter is sent to the input of the electronic converter, and during a second, consecutive time interval, the output signal of another photovoltaic converter is sent to the input of the electronic converter. If more than two photovoltaic converters are connected to one electronic converter via an appropriate number of switching elements (e.g. three, four or more), the time multiplexing will work by driving the switching elements so that the photovoltaic converters are switched to the electronic converter one after another. The time multiplexing via the switching elements can be performed periodically.

[0022] Preferably, there are a plurality of first switching elements connecting the same number of first photovoltaic converters to the corresponding electronic converters, a plurality of second switching elements connecting the same number of second photovoltaic converters to the electronic converter, etc. In order to allow the output signals from the first photovoltaic converters to reach the corresponding electronic converters, all the first switching elements may be driven simultaneously, then all the second switching elements may be driven simultaneously, etc.

[0023] The driving signal for the switching element may be generated by a switching controller connected to the image sensor. Therefore, in another aspect of the present invention, a sensor device for detecting time-dependent image data is proposed, the sensor device comprising an image sensor as described herein and a switching controller connected to the image sensor and configured to generate a switching signal to control the switching element. The switching controller generating the switching signal may be a microcontroller or a logic block implemented on the same chip as the image sensor. However, the switching controller is located outside the photovoltaic converter array.

[0024] Any features mentioned with reference to the image sensor are equally advantageous in connection with the sensor device. In other words, if it is mentioned that the image sensor or its image component is configured to function in a certain way or generate a certain signal, this may also mean that the switching controller has been appropriately programmed to allow or ensure that this occurs. For example, the feature that the switching element is arranged and configured to time-multiplex the photovoltaic converter to the input of the electronic converter may be equally applicable to the sensor device. When the switching controller is configured to generate appropriate switching signals to drive the switching element to perform this time-multiplexing.

[0025] According to an advantageous embodiment, the electronic converter is configured to generate digital information in dependence on a relative difference or difference between the light intensities at two of the photovoltaic converters connected to the electronic converter by the switching element. In other words, the digital information is dependent on a relative difference between a first intensity and a second intensity, wherein the first intensity is the light intensity at the first photovoltaic converter and the second intensity is the light intensity at the second photovoltaic converter. In this way, the image sensor is able to obtain spatial contrast information of the sensed image. When the input of the electronic converter is switched from the first photovoltaic converter to the second photovoltaic converter, the encoder output will be a digital signal, which is the difference between the signal converter output corresponding to the second intensity and the signal converter output corresponding to the first intensity. As will be further explained below, if the output logarithm of the signal converter depends on the incident light, then, if the intensity difference is small relative to the absolute intensity, the output difference of the signal converter is proportional to the contrast or relative intensity difference.

[0026] Advantageously, at least two photovoltaic converters connected to one of the electronic converters via the switching element are arranged adjacent to each other in the array. This may also apply when more than two photovoltaic converters are connected to one electronic converter via the switching element. In particular, in a two-dimensional square array, four adjacent photovoltaic converters in a square are connected to one electronic converter via the switching element. Thus, in a hexagonal or triangular array, adjacent photovoltaic converters connected to the electronic converter via the switching element may themselves be arranged in a hexagon or triangle.

[0027] Advantageously, a neighboring electronic converter placed adjacent to said electronic converter is configured to generate at least one digital information according to the relative difference or difference between the light intensity on the same two photovoltaic converters, which is connected to said electronic converter and said neighboring electronic converter via a switching element.

[0028] The spatial order of time multiplexing between the photovoltaic converter and the electronic converter via the switching elements can be achieved by different schemes. Two such schemes will be discussed here and in more detail below, one scheme is called the simultaneous dual encoding scheme and the other scheme is called the spatial arrangement scheme. Although the image sensor must be configured to support any such scheme, the switching controller must be configured to generate the appropriate switching signals in order to drive the switching elements of the image sensor accordingly. However, it should be noted that the image sensors can be manufactured and sold separately.

[0029] According to the simultaneous dual encoding scheme, the electronic converter and the adjacent electronic converter are connected to the two photovoltaic converters by two pairs of switching elements, whereby each pair of the switching elements is configured to switch in the same manner. In other words, the relative intensity difference of light between two adjacent photoelectric converters, as well as the relative spatial difference between them, are encoded simultaneously by two adjacent encoders. With this scheme, the influence of motion artifacts can be reduced during post-processing of the encoded digital information. Further, two pairs of switching elements can be arranged and configured to selectively connect the photovoltaic converter and the adjacent photovoltaic converter to the electronic converter and the adjacent electronic converter, so that before a switching event occurs, the photovoltaic converter is connected to the electronic converter and the adjacent photovoltaic converter is connected to the adjacent electronic converter, and after the switching event, the photovoltaic converter is connected to the adjacent electronic converter and the adjacent photovoltaic converter is connected to the electronic converter.

[0030] According to the spatial arrangement scheme, when the first photovoltaic converter is connected to the first electronic converter via the first active switching element and the second photovoltaic converter is connected to the second electronic converter via the second active switching element, the photovoltaic converter array, the electronic converter array and the switching element are configured so that the first photovoltaic converter and the second photovoltaic converter have the same relative distance to the first electronic converter and the second electronic converter at any time. This scheme allows spatially correlated encoding, which means that in all switching modes, each electronic converter or pixel and its connected photoelectric converter have the same relative spatial position. This scheme can support post-processing of image sensor output based on spatial correlation.

[0031] Advantageously, the number of photovoltaic converters is substantially equal to the number of electronic converters. This means that, although two or more photovoltaic converters are connected to one electronic converter via a switching element, each of these two or more photovoltaic converters is in turn connected to the same number of electronic converters. In other words, the field of view of one electronic converter may overlap the field of view of another electronic converter, in particular an adjacent electronic converter. Preferably, at any given time, the output signal of each photovoltaic converter is sent to one corresponding electronic converter. This may not be true only at the edges of the array.

[0032] In this context, the qualifier "substantially" means that the number of photovoltaic converters and the number of electronic converters may vary by a small amount due to geometrical limitations of the image sensor or other reasons. In particular, the difference may be less than 2%, 5% or 10%. The reason for this difference may be that the photovoltaic converters at the edge of the array may not have the same number of electronic converters connected to the photovoltaic converters in the array. In particular, if the number of photovoltaic converters is N 2 , then the difference can be of the order of 2 / N.

[0033] Alternatively, the number of photovoltaic converters may be a multiple of the number of electronic converters. 2 A photovoltaic converter may be associated with a single electronic converter. In this case, in order to be able to implement the switching scheme and gain mismatch compensation described herein, the adjacent electronic converters must share one or more pairs of shared photovoltaic converters. This means that the shared photovoltaic converter is connected to two adjacent electronic converters via a switching element. It can be said that the field of view of the two adjacent electronic converters is overlapped by at least one pair of photovoltaic converters.

[0034] According to another aspect of the present invention, a method for obtaining image data compensated for gain mismatch is provided. The compensation method has the following steps: In a first step, first digital information is obtained from a first electronic converter. The first digital information can in particular be encoded image data from two photovoltaic converters connected to the first electronic converter via a switching element. The first digital information includes first reference information, which depends on the relative difference or difference in light intensity on the two photovoltaic converters. In addition, second digital information is obtained from a second electronic converter. Similar to the first digital information, the second digital information can in particular be encoded image data from the two photovoltaic converters connected to the second electronic converter via a switching element. The second digital information includes second reference information, which also depends on the difference in light intensity on the two photovoltaic converters.

[0035] In the above-mentioned simultaneous dual encoding scheme, the first digital information and the second digital information can be obtained at the same time, while in the spatial arrangement scheme, the second digital information is obtained at a different time compared to the first digital information.

[0036] Although both the first reference information and the second reference information are obtained by encoding the relative difference of light intensity on the same two photovoltaic converters (possibly with different signs), they may be different due to the gain mismatch between the first and second electronic converters. Therefore, one can obtain an adjustment factor from the first reference information and the second reference information and use it to compensate for this gain mismatch in the first digital information and the second digital information. This can be done by taking the first electronic converter as a reference and adjusting the second digital information with the adjustment factor, or by taking the second electronic converter as a reference and adjusting the first digital information with the adjustment factor.

[0037] In particular, the adjustment factor can be taken as the quotient of the first reference information and the second reference information. Figure 7 Describe an example situation of this compensation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In the following description, some examples of embodiments of the present invention will be explained in more detail with reference to the attached schematic diagrams, in which:

[0039] Figure 1 shows a schematic cross-sectional view of a pixel in an image sensor according to a preferred embodiment, the image sensor comprising an electronic converter and a photovoltaic converter;

[0040] Figure 2 shows a schematic diagram of a signal converter connected to four photovoltaic converters via corresponding switching elements according to a preferred embodiment;

[0041] Figure 3 shows a schematic circuit diagram of a signal converter and an encoder according to a preferred embodiment;

[0042] Figure 4 shows a timing diagram of an exemplary encoder output depending on a series of switching signals according to a preferred embodiment;

[0043] Figure 5 shows a schematic layout of an array of photovoltaic converters and an array of signal converters connected to them via switching elements on a semiconductor chip die according to a preferred embodiment;

[0044] Figure 6 shows a schematic layout of an array of electronic converters on another semiconductor chip die, which can be connected to the Figure 5 The chip die shown;

[0045] Figure 7 shows a diagram for illustrating post-processing of a signal generated by an image sensor to compensate for gain mismatch according to a preferred embodiment;

[0046] Figure 8 shows a schematic layout of a photovoltaic converter array arranged and driven on a semiconductor chip according to a simultaneous dual encoding scheme, as well as signal converters and switching elements connecting them;

[0047] Fig. 9 shows a schematic layout of a photovoltaic converter array arranged and driven on a semiconductor die according to a spatially arranged coding scheme, as well as a signal converter and switching elements connecting them; DETAILED DESCRIPTION

[0048] The image sensor according to the embodiments described below comprises a two-dimensional rectangular array of photodiodes and transducer elements, which logarithmically convert the incident light intensity into an analog voltage signal, and a two-dimensional rectangular array of analog-to-digital encoder elements, which encode the analog voltage signal into a digital signal. Thus, each photovoltaic converter is formed as a photodiode, in particular a PPD, and each electronic converter has a logarithmic current-to-voltage signal converter (hereinafter referred to as converter) and an analog-to-digital converter (referred to as encoder) for converting the analog voltage level into digital information.

[0049] The image sensor encodes static scene spatial contrast by electronically time-multiplexing 4 adjacent photodiodes into the input of a transducer, thus converting the static scene spatial contrast into temporal variations of the transducer element input, and thus the converted analog signal also varies in time. The encoder then encodes the temporal variations using a scheme based on pulse density modulation.

[0050] The mathematical relationship between the logarithmic difference in incident light intensity and spatial contrast is explained by the following formula

[0051] If ΔI is small,

[0052] where I is the incident light intensity and k is a constant gain. This formula shows that for small spatial differences in incident light intensity ΔI, the spatial contrast ΔI / I can be approximated by the spatial differences in the logarithmic value of the incident light intensity log(I).

[0053] If the sensor stops the time multiplexing operation between the photodiode and the transducer, the temporal changes in the converted analog signal directly correspond to the temporal contrast of the scene. Thus, without time multiplexing, the sensor encodes only the temporal contrast of the scene.

[0054] Figure 1 An abstract cross-sectional view of a photodiode 1, a transducer 3 and an encoder 4 is shown. It should be noted that even though Figure 1The reference frame marked with "3" also contains a photodiode 1, which is not part of the transducer 3. The sensor uses a pinned photodiode (PPD). The use of a PPD minimizes the difference in the dark signal at the input to the transducer 3 and encoder 4 between different multiplexed time windows. The array of PPDs 1 and transducer 3 are manufactured on the image sensor processing silicon chip. The encoder 4 array is manufactured on a mixed signal processing silicon chip die. Each transducer-encoder pair is connected to the two chip dies by an interconnect 50. As previously mentioned, in the following, a transducer-encoder pair is referred to as a pixel, which does not include the PPD, because different PPDs are connected to the same pixel at different times during the time multiplexing process. Although in the embodiments currently shown and discussed, the interconnect between the two chip dies connects the transducer to the encoder, there are other ways to distribute the circuit to one or two chip dies. As an example, the interconnect can be placed within the circuit for the encoder.

[0055] exist Figure 1 Also indicated are a silicon substrate 51 and a metal layer 52 of the image sensor processing silicon chip die, and a silicon substrate 54 and a metal layer 53 of the mixed signal sensor processing silicon chip die.

[0056] Figure 2 FIG. 1 shows a circuit schematic diagram of a transducer 3 and four PPDs 1 connected to the transducer 3 via switching elements tx1, tx2, tx3, tx4. Each PPD 1 is also connected to the transducer 3 via the switching element (not shown). Figure 2 ) connected to three other electronic converters.

[0057] The output of the transducer 3, the analog voltage signal vsf, is sent to the input of the corresponding encoder on the stacked mixed signal chip die via a one-to-one interconnect 50. Each transducer 3 is connected to four adjacent PPDs 1 through four switching elements, which are implemented as transmission gates controlled by switching signals, and each PPD 1 is connected to four adjacent transducers 3 in the same way. The switching signals must be provided by a switching controller not shown in the figure. These switching signals will be combined below Figure 4 It should be noted that the switching elements with the same reference numerals receive the same switching signal or are driven by the same switching signal. For example, the switching elements labeled "tx1" all receive the same switching signal, which can also be labeled as tx1.

[0058] The encoder then converts the output of the transducer 3, the analog signal vsf, into a digital signal. Considering that the encoder element needs to be compact enough so that it can be implemented in a pixel-parallel manner (i.e., each pixel contains 1 such analog-to-digital encoder element), an analog-to-digital encoder based on pulse density modulation is a suitable choice. Encoders based on pulse density modulation (delta modulation) are implemented in the prior art US 7,728,269B2, US9631974 and ATIS (Posch et al., 2010). Figure 3 An example schematic diagram of an encoder obtained by modifying the device described in US 7,728,269 B2 is shown. Figure 3 The top circuit diagram in is a circuit diagram of a photoelectric array unit taken from US 7,728,269 B2. It comprises a photodiode D, four transistors T1-T4 forming a logarithmic amplifier connected to an encoder via a voltage buffer made of two transistors T5a, T5b. Figure 3 The bottom circuit diagram of FIG. 1 shows that the circuit of US7,728,269 B2 is simplified by retaining the encoder 4, while replacing the photodiode D, the amplifiers T1-T4 and the voltage buffers T5a, T5b (indicated by the dotted box A) with transistors (indicated by the dotted box B), which form a voltage buffer through transistor 6, which is connected to the Figure 2 vfd in transducer 3 in .

[0059] At any time, only one switching element tx1, ..., tx4 is turned on or activated. Via this one switching element tx1, ..., tx4, a pair of transducer-encoder pairs 3, 4 is connected to only one PPD, and one PPD is connected to only one transducer-encoder pair 3, 4. There may be exceptions to the PPDs at the edges of the 2D array, where no sensor / encoder pair may be connected at certain points in time. Therefore, when the switching signals tx1, ..., tx4 remain constant, the image sensor only encodes the temporal contrast of the scene, similar to the device described in US 7,728,269 B2.

[0060] Figure 4A timing diagram of the switching signals tx1, ..., tw4 used to perform time multiplexing is shown. Throughout the time multiplexing process, a one-to-one connection is maintained between the PPD 1 and the sensor-encoder pair 3, 4. During time multiplexing, the output of the encoder 4 is an encoded signal of the change in the logarithmic voltage vsf, which represents the logarithmic difference in photocurrent between adjacent PPDs 1, and therefore represents the spatial contrast. In other words, after a switching event, when the second switching element is off and the first switching element is on, the encoder output corresponds to the difference between the logarithmic signal of the PPD connected through the second switching element and the logarithmic signal of the PPD connected through the first switching element.

[0061] The switching interval used to switch from one switching element to the next is adjustable, which determines the bit depth of the encoded spatial contrast. For example, if a short switching interval is chosen that allows at most one pulse per pixel, the encoded spatial contrast has only one gradient level. On the other hand, if the switching interval allows at most 100 pulses per pixel, the encoded spatial contrast will have 100 possible gradient levels.

[0062] Within a time multiplexing cycle, in which the activation or turn-on pattern of the switching elements tx1, ..., tx4 is shifted four times, the relative spatial difference between every two adjacent photodiode pairs is converted and encoded by two adjacent transducer-encoder pairs, and every two adjacent transducer-encoder pairs have encoded a relative spatial difference between the same two adjacent photodiodes 1.

[0063] Figure 5 A schematic layout of a portion of a PPD 1 array and a transducer 3 array connected to PPD 1 via switching elements tx1, ..., tx4 on a semiconductor chip (i.e., an image sensor processing chip) is shown. The junction leakage current introduced by the transducer 3 circuit remains constant throughout the time multiplexing period. And because the dark current of PPD 1 is low, multiplexing between different PPD 1s introduces minimal offset noise caused by the difference in their dark signals. Therefore, during the time multiplexing process, the temporal variation of each transducer 3 output is able to represent the accurate spatial contrast in the PPD 1 array with minimal offset noise. Because only the spatial contrast of the PPD 1 array is encoded, rather than the absolute signal level, there is also no offset mismatch at the transducer 3 stage.

[0064] Figure 6 The schematic layout of the electronic converter array on another semiconductor chip die (i.e., a mixed signal processing chip die) is shown. The encoder 4 is placed in a 2D array with a pitch matching that of the sensor 5, such as Figure 5 As shown, on the image sensor processing chip die. Figure 5 and Figure 6 The dotted circles in represent the interconnection locations between the two die. Typically, the node size of a mixed signal processing die is much smaller than that of an image sensor processing die, so the same area on a mixed signal processing die can contain more circuitry. The final pixel output from encoder 4 contains fixed pattern noise caused by gain mismatches in the transducer logarithmic conversion, as well as gain mismatches due to encoder quantization threshold variations. These gain mismatches can be eliminated in post-processing.

[0065] Figure 7 A diagram illustrating post-processing of the signal generated by the image sensor to compensate for gain mismatch is shown. This simplified example is based on only two rows and three columns of the PPD 1 and two adjacent pixels (ie, transducer-encoder pairs). Figure 7 and subsequent Figure 8 and 9 The individual PPDs and pixels in are identified by their coordinates (row, column) in the respective arrays. Therefore, for the sake of visual clarity, the reference numerals used so far are omitted.

[0066] After one full time multiplexing cycle, the same spatial contrast between PPD(0,1) and PPD(1,1) is encoded as -1 by pixel (0,0) and 2 by pixel (0,1) (the sign indicates the switching direction). Therefore, the gain mismatch between PPD(0,1) and PPD(1,1) can be deduced to be 1:2. With this information, if pixel (0,0) is used as a reference, the spatial contrast output from pixel (0,1) is scaled by 0.5, so the spatial contrast of the entire array can be calculated with reference to pixel (0,1). If the value of PPD(0,0) is 0, the value of the entire array can be calculated based on the scaled spatial contrast, resulting in a log-compressed image of the scene.

[0067] During the time multiplexing between the PPD and the transducers, there are two possible schemes for arranging the spatial order of the switching elements tx1, ..., tx4: Figure 8 The simultaneous dual encoding scheme shown and Fig. 9 The spatial arrangement encoding scheme shown.

[0068] Figure 8 The schematic layout of the photovoltaic converter array and the signal converters and switching elements connecting them arranged and driven according to the simultaneous dual encoding scheme on the semiconductor dual chip is shown. Figure 5 The scheme shown in is the same as that shown in . Arrows labeled 1st, 2nd, 3rd and 4th are introduced, which indicate the switching order of the conduction mode of the switching element. Figure 8 and 9 In the switching order, the Figure 4 That is, first the switching elements marked tx1 are in the on state, followed by those marked tx2 and so on.

[0069] Simultaneous double encoding means that the relative spatial difference between the same two adjacent PPDs is encoded by two pixels simultaneously in opposite directions. For example, on the fourth switching element of the mode transition (from tx4 to tx1), pixel (x, y) encodes the spatial contrast from PPD (x, y-1) to PPD (x, y), and pixel (x-1, y) encodes the spatial contrast from PPD (x, y) to PPD (x, y-1). This scheme reduces the impact of motion artifacts in post-processing gain mismatch removal.

[0070] from Figure 8 Another way to describe the switching process of the simultaneous dual encoding mode is to look at the two PPDs and the two sensors and the pairs of switching elements that selectively connect them in pairs. For example, one can look at a PPD (PPD(x, y)) and an adjacent PPD (PPD(x+1, y)) connected to a transducer (pixel (x, y)), and an adjacent transducer (pixel (x, y+1)) consisting of two pairs of switching elements (labeled tx1 and tx2). Each pair of these switching elements is represented by an arrow (labeled "1st"), representing the first ("first") switching event, and indicating that before the first switching event, the switching element labeled tx1 was turned on, when the switching element labeled tx2 was turned off, and after the first switching event, the switching element labeled tx2 was turned on and the switching element labeled tx1 was turned on. Thus, before the first switching event, the PPD (PPD (x, y)) is connected to the transducer (pixel (x, y)), and the adjacent PPD (PPD (x+1, y+1)) is connected to the adjacent transducer (pixel (x, y+1)). Then, after the first switching event, the connections are reversed: the PPD (PPD (x, y+1)) is now connected to the adjacent transducer (pixel (x, y+1)), and the adjacent PPD (PPD (x+1, y)) is connected to the transducer (pixel (x, y)).

[0071] Fig. 9A schematic layout of a photovoltaic converter array and signal converters and switching elements connecting them is shown, which are arranged and driven on a semiconductor chip die according to a spatially arranged encoding scheme. The spatially arranged encoding means that each pixel and its connected PPD have the same relative spatial position throughout the time multiplexing process. For example, before the first switching element of the mode switch (from tx1 to tx2), pixel (x-1, y) is connected to PPD (x-1, y), and pixel (x, y) is connected to PPD (x, y), where pixel (x-1, y) is the left neighbor pixel of pixel (x, y), and PPD (x-1, y) is the left neighbor PPD of PPD (x, y). After the first switching element enters mode conversion, pixel (x-1, y) is connected to PPD (x, y), and pixel (x, y) is connected to PPD (x+1, y), where PPD (x, y) is still the left adjacent PPD of PPD (x+1, y) and maintains the same relative spatial position as pixel (x-1, y) and pixel (x, y). Since the relative spatial position of the spatial contrast encoded by the pixel array is constant throughout the time multiplexing, this scheme can support spatial correlation based on post-processing of the pixel array output.

[0072] Reference numerals:

[0073] 1 Photoelectric converter, photodiode, PPD

[0074] 2 Electronic converter

[0075] 3Signal converter, transducer

[0076] 4A / D converter, encoder

[0077] 5 Interconnection

[0078] 6 Output transistors

[0079] 51, 54 Silicon substrate

[0080] 52, 53 Metal layer

[0081] tx1, tx2, tx3, tx4 switching elements, transmission gates, switching signals

Claims

1. An image sensor for detecting time-dependent image data, comprising a plurality of photodiodes (1) and a plurality of electronic converters (2), the plurality of photodiodes (1) and the plurality of electronic converters (2) being arranged in an array and connected via switching elements (tx1, tx2, ...), so that each of the photodiodes (1) and one of the electronic converters (2) forms a pair for generating digital information, the digital information being dependent on digital information of light intensity on the photodiodes (1), and the switching elements (tx1, tx2, ...) being arranged to selectively connect at least two of the photodiodes (1) to one of the electronic converters (2) and at least two of the electronic converters (2) to one of the photodiodes (1), wherein three, four or more photodiodes (1) are connected to one electronic converter (2) via an appropriate number of switching elements (tx1, tx2, ...), the switching elements being driven to perform time multiplexing so that the photodiodes (1) are switched to the electronic converters (2) one by one, wherein each electronic converter (2) comprises an electronic signal converter (3) and an encoder, wherein the output of the electronic signal converter (3) depends on the output of the photodiode (1) linearly, logarithmically or according to some other function, wherein the encoder is configured to generate digital information in dependence on a relative difference or difference between the light intensities on two of the photodiodes (1), wherein two pairs of switching elements (tx1, tx2, ...) are arranged and configured to selectively connect the photodiode (1) and an adjacent photodiode (1) to the electronic converter (2) and an adjacent electronic converter (2), such that before a switching event, the photodiode (1) is connected to the electronic converter (2) and the adjacent photodiode (1) is connected to the adjacent electronic converter (2), and after the switching event, the photodiode (1) is connected to the adjacent electronic converter (2) and the adjacent photodiode (1) is connected to the electronic converter (2), wherein the digital information depends on the relative difference between a first intensity and a second intensity, wherein the first intensity is the light intensity at a photodiode (1) and the second intensity is the light intensity at an adjacent photodiode (1), and wherein when the input of the electronic converter (2) is switched from the photodiode (1) to the adjacent photodiode (1), the output of the encoder is a digital signal corresponding to the difference between the output of the signal converter corresponding to the second intensity and the output of the signal converter corresponding to the first intensity.

2. The image sensor according to claim 1, It is characterized in that The electronic signal converter (3) is configured in combination with the photodiode (1) of the pair to generate an analog electronic signal in dependence on the light intensity on the photodiode (1).

3. The image sensor according to claim 2, It is characterized in that The signal converter (3) is configured so that the generated analog electronic signal is logarithmically dependent on the light intensity on the photodiode (1).

4. The image sensor according to claim 1, It is characterized in that The encoder is an analog-to-digital converter (4) configured to convert an analog electronic signal dependent on the light intensity on the photodiode (1) into a digital signal.

5. The image sensor according to claim 4, It is characterized in that The analog-to-digital converter (4) is configured to convert the analog electronic signal into the digital signal using pulse density modulation.

6. The image sensor according to claim 4, It is characterized in that The signal converter (3) and the analog-to-digital converter (4) are fabricated on two separate semiconductor chip dies and are electrically connected via an interconnect (5) between the two semiconductor chip dies.

7. The image sensor according to any one of claims 1 to 6, It is characterized in that The switching elements (tx1, tx2, ...) are configured to time-multiplex the photodiodes (1) to the input of the electronic converter (2).

8. The image sensor according to claim 7, It is characterized in that The switching elements (tx1, tx2, ...) are configured to periodically time-multiplex the photodiodes (1) to the input of the electronic converter (2).

9. The image sensor according to claim 1, It is characterized in that An adjacent electronic converter (2) placed adjacent to the electronic converter (2) is configured to generate at least one digital information in dependence on a relative difference or difference between light intensities on the same two photodiodes (1), the same two photodiodes (1) being connected to both the electronic converter (2) and the adjacent electronic converter (2) via the switching element (tx1, tx2, ...).

10. The image sensor according to claim 9, It is characterized in that When the first photodiode (1) is connected to the first electronic converter (2) through the first active switching element (tx1, tx2, ...) and the second photodiode (1) is connected to the second electronic converter (2) through the second active switching element (tx1, tx2, ...), the photodiode (1) array, the electronic converter (2) array and the switching elements (tx1, tx2, ...) are configured so that at any time, the relative distance between the first photodiode (1) and the second photodiode (1) is the same as that between the first electronic converter (2) and the second electronic converter (2).

11. The image sensor according to claim 1, It is characterized in that The number of photodiodes (1) is substantially equal to the number of electronic converters (2).

12. A sensor device for detecting time-dependent image data, comprising an image sensor according to any one of claims 1 to 11 and a switching controller connected to the switching element (tx1, tx2, ...) of the image sensor and configured to generate a switching signal to control the switching element (tx1, tx2, ...).

13. A method for controlling an image sensor for detecting time-dependent image data, the image sensor comprising a plurality of photodiodes (1) and a plurality of electronic converters (2), the plurality of photodiodes (1) and the plurality of electronic converters (2) being arranged in an array and connected via switching elements (tx1, tx2, ...), so that each of the photodiodes (1) and one of the electronic converters (2) forms a pair for generating digital information, the digital information being dependent on digital information of light intensity on the photodiode (1), and the switching elements (tx1, tx2, ...) switching at least two of the photodiodes (1) 1) is selectively connected to one of the electronic converters (2) so that a signal from one or another photodiode (1) reaches the electronic converter (2), and the switching element (tx1, tx2, ...) selectively connects at least two of the electronic converters (2) to one of the photodiodes (1), wherein three, four or more photodiodes (1) are connected to one electronic converter (2) through an appropriate number of switching elements (tx1, tx2, ...), and the switching elements are driven to switch the photodiodes (1) to the electronic converter (2) one by one in a multiplexing manner, wherein each electronic converter (2) comprises an electronic signal converter (3) and an encoder, wherein the output of the electronic signal converter (3) depends on the output of the photodiode (1) linearly, logarithmically or according to some other function, wherein the encoder generates digital information by relying on the relative difference or variance between the light intensities on the two photodiodes (1), wherein two pairs of switching elements (tx1, tx2, ...) selectively connect the photodiode (1) and the adjacent photodiode (1) to the electronic converter (2) and the adjacent electronic converter (2), such that before a switching event, the photodiode (1) is connected to the electronic converter (2) and the adjacent photodiode (1) is connected to the adjacent electronic converter (2), and after the switching event, the photodiode (1) is connected to the adjacent electronic converter (2) and the adjacent photodiode (1) is connected to the electronic converter (2), wherein the digital information depends on the relative difference between a first intensity and a second intensity, wherein the first intensity is the light intensity at a photodiode (1) and the second intensity is the light intensity at an adjacent photodiode (1), and wherein when the input of the electronic converter (2) is switched from the photodiode (1) to the adjacent photodiode (1), the output of the encoder is a digital signal corresponding to the difference between the output of the signal converter corresponding to the second intensity and the output of the signal converter corresponding to the first intensity.

14. The method according to claim 13, wherein the method is used to obtain image data compensated for gain mismatch, wherein The following steps are involved: - obtaining first digital information from a first electronic converter (2) of the array of electronic converters (2), the first digital information comprising first reference information, the first reference information being dependent on the relative difference in light intensity on two photodiodes (1) of the array of photodiodes (1), - obtaining second digital information from a second electronic converter (2) in the array of electronic converters (2), the second digital information comprising second reference information, the second reference information being dependent on the relative difference in light intensity on the two photodiodes (1), - In the first digital information and the second digital information, the first digital information and / or the second digital information are adjusted by an adjustment factor obtained from the first reference information and the second reference information to compensate for a gain mismatch between the first electronic converter (2) and the second electronic converter (2).

Images