Dynamic vision sensor

Abstract

The invention provides a dynamic visual sensor. The dynamic visual sensor comprises a sensor; and an image signal processor. The sensor includes a pixel driver, a plurality of pixel groups, a cluster differential analog-to-digital converter, a cluster memory, a sensor data interface, and a controller. Each pixel group comprises a plurality of pixels and floating diffusion points, each pixel comprises a photodiode and a switch, the switch controls whether charges accumulated in the photodiode are transmitted to the floating diffusion points or not, the switch in each pixel is controlled by the pixel driver, and the pixels in the same pixel group are jointly coupled to the floating diffusion points. The controller controls the pixel driver, the cluster differential analog-to-digital converter, the cluster memory and the sensor data interface to be switched between a dynamic sensing mode and a two-dimensional image sensing mode, and the effects of saving operation time, hardware cost and size can be achieved.

Description

Technical Field

[0001] The invention relates to the field of sensors, in particular to a dynamic vision sensor. Background Art

[0002] In general, the dynamic vision sensor (DVS) in the related art can only sense dynamic images (such as the movement of objects) and does not include static two-dimensional (2D) image sensing functions. Therefore, in order to obtain detailed static 2D images, a 2D image sensor must be installed on the device and switch back and forth between dynamic vision sensing and static 2D image sensing, which increases computing time, hardware cost and volume, and is not conducive to application fields such as vehicle safety.

[0003] Therefore, there is an urgent need for a dynamic vision sensor having a dynamic sensing mode and a two-dimensional image sensing mode, and being able to effectively switch between the two modes. There is also an urgent need for a method that enables the dynamic vision sensor to switch between the dynamic sensing mode and the two-dimensional image sensing mode. Summary of the invention

[0004] The main purpose of the present invention is to provide a dynamic vision sensor having a dynamic sensing mode and a two-dimensional image sensing mode, and can effectively switch between the two modes. The present invention also includes a dynamic vision sensor switching method, so that the dynamic vision sensor can switch between the dynamic sensing mode and the two-dimensional image sensing mode.

[0005] In order to achieve the aforementioned objectives, the present invention provides a dynamic vision sensor, including a sensor; and an image signal processor. The sensor includes: a pixel driver, a plurality of pixel groups, a cluster differential analog-to-digital converter having a positive input terminal and a negative input terminal, a cluster memory, a sensor data interface (e.g., a mobile industry processor interface (MIPI)), and a controller. Each pixel group includes a plurality of pixels and a floating diffusion point, each pixel includes a photodiode and a switch, the switch controls whether the charge accumulated in the photodiode is transferred to the floating diffusion point, the switch in each pixel is controlled by the pixel driver, and each pixel in the same pixel group is coupled to the floating diffusion point in common. The image signal processor includes an algorithm. Half of the plurality of pixel groups are coupled to the positive input terminal of the cluster differential analog-to-digital converter, and the rest are coupled to the negative input terminal of the cluster differential analog-to-digital converter. The cluster differential analog-to-digital converter outputs data to the cluster memory, and the cluster memory transmits the data to the image signal processor via the sensor data interface. The image signal processor operates the data using the algorithm and transmits the operation result to the controller. According to the operation result, the controller controls the pixel driver, the cluster differential analog-to-digital converter, the cluster memory and the sensor data interface to switch between the dynamic sensing mode and the two-dimensional image sensing mode.

[0006] In order to achieve the aforementioned objectives, the present invention provides a dynamic vision sensor switching method, comprising: operating the dynamic vision sensor in a dynamic sensing mode; in the dynamic sensing mode, determining whether the movement of an object is detected, and if the movement of the object is detected, switching the dynamic vision sensor to a two-dimensional image sensing mode; and in the two-dimensional image sensing mode, inspecting the details of the object.

[0007] The utility model is that the same set of dynamic vision sensor system can have a dynamic sensing mode and a two-dimensional image sensing mode, thereby saving computing time, hardware cost and volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A and Figure 1B is a system schematic diagram of a dynamic vision sensor according to the present invention;

[0009] Figure 2A is a schematic diagram of a pixel array of a dynamic vision sensor according to the present invention;

[0010] Figure 2B is a schematic diagram of a cluster differential analog-to-digital converter array of a dynamic vision sensor according to the present invention;

[0011] Figure 2Cis a schematic diagram of a cluster memory array of a dynamic vision sensor according to the present invention;

[0012] Figure 2D , Figure 2E and Figure 2F is a schematic diagram of a stack structure of a dynamic vision sensor according to the present invention;

[0013] Figure 3 is a flow chart of a dynamic visual sensor switching method according to the present invention; and

[0014] Figure 4A , Figure 4B and Figure 5 is an example of a timing diagram according to the present invention.

[0015] Description of reference numerals:

[0016] 100: dynamic vision sensor;

[0017] 102: pixel driver;

[0018] 104a, 104b, 104c, 104d: pixel groups;

[0019] tx0, tx1, ..., tx15: control signal;

[0020] SW0, SW1, ..., SW15: switches;

[0021] PD0, PD1, ..., PD15: Photodiode

[0022] Vr0, Vr1: ramp voltage;

[0023] rst0,rst1,rst2,rst3: reset signal;

[0024] VDD: supply voltage;

[0025] SWR0, SWR1, SWR2, SWR3: switch;

[0026] FD0, FD1, FD2, FD3: floating diffusion points;

[0027] Cr0,Cr1,Cr2,Cr3:capacitor;

[0028] SF0, SF1, SF2, SF3: source follower;

[0029] SWC0, SWC1, SWC2, SWC3: switches;

[0030] sel0, sel1, sel2, sel3: select signal;

[0031] 105p: positive input terminal;

[0032] 105n: negative input terminal;

[0033] 106: Cluster differential analog-to-digital converter;

[0034] 108: cluster memory;

[0035] 110: sensor data interface;

[0036] 112: Algorithm;

[0037] 114: controller;

[0038] 120: sensor;

[0039] 122: image signal processor;

[0040] 150: ramp voltage generator;

[0041] Vcm: common mode voltage;

[0042] clock: frequency;

[0043] FDP, FDN: floating diffusion point;

[0044] CrP,CrN:capacitance;

[0045] SFP, SFN: source follower;

[0046] 200: dynamic vision sensor;

[0047] 2000, 2001, …, 2315: pixels;

[0048] 210,211,212,213: Cluster differential analog-to-digital converter;

[0049] 2000c, 2001c, …, 2315c: cluster memory;

[0050] 220,221,222,223: pixel subset;

[0051] 230:Substrate;

[0052] 240:Substrate;

[0053] 250,251,252,253: pixel subset;

[0054] 260:Substrate;

[0055] 270,271,272,273: pixel subset;

[0056] 300:method;

[0057] 302,304,306,308: steps;

[0058] INT0,INT1,INT2,INT3,INT4,INT5,INT6,INT7: time;

[0059] ADC0,ADC1,ADC2,ADC3,ADC4,ADC5,ADC6,ADC7: time;

[0060] DO: time;

[0061] INT0P,INT0N,INT1P,INT1N,INT2P,INT2N,INT3P,INT3N: time;

[0062] ADC0P,ADC0N,ADC1P,ADC1N,ADC2P,ADC2N,ADC3P,ADC3N: time;

[0063] DDO: Time. DETAILED DESCRIPTION

[0064] The following is a more detailed description of the embodiments of the present invention with the help of drawings and component symbols, so that those skilled in the art can implement the invention after reading this specification.

[0065] See also Figure 1AOne embodiment of the present invention provides a dynamic vision sensor (DVS) 100, including a sensor 120 and an image signal processor 122. The sensor 120 includes a pixel driver 102, a plurality of pixel groups 104a, 104b, 104c, 104d, etc., a cluster differential analog-to-digital converter (cluster differential ADC) 106, a cluster memory 108, a sensor data interface 110 and a controller 114. The image signal processor 122 has an algorithm 112. Each pixel group includes a plurality of photodiodes, and the plurality of photodiodes share a floating diffusion (FD). Each photodiode is associated with a switch (for example, but not limited to, an NMOS switch), which controls whether the charge accumulated in the photodiode is transferred to the floating diffusion. Half of the plurality of pixel groups are coupled to the positive input terminal 105p of the cluster differential analog-to-digital converter 106, and the rest are coupled to the negative input terminal 105n of the cluster differential analog-to-digital converter 106. The cluster differential analog-to-digital converter 106 outputs data to the cluster memory 108, and the cluster memory 108 transmits the stored data to the image signal processor 122 via the sensor data interface 110. The image signal processor 122 operates the data using the algorithm 112 and transmits the result to the controller 114. According to the operation result, the controller 114 controls the pixel driver 102, the cluster differential analog-to-digital converter 106, the cluster memory 108 and the sensor data interface 110 to switch between the dynamic sensing mode and the two-dimensional image sensing mode.

[0066] The number of cluster differential analog-to-digital converters 106, the number of pixel groups, and the number of photodiodes in each pixel group can be determined according to the use requirements. For example, for 16*16 pixels, 16 cluster differential analog-to-digital converters 106 can be used, each cluster differential analog-to-digital converter 106 has two pixel groups coupled to the positive input terminal 105p and two pixel groups coupled to the negative input terminal 105n, and each pixel group has four photodiodes and one floating diffusion point. Figure 1AFor example, the cluster differential analog-to-digital converter 106 has two pixel groups 104a and 104c coupled to the positive input terminal 105p, and two pixel groups 104b and 104d coupled to the negative input terminal 105n. In the pixel group 104a, there are four photodiodes PD0, PD1, PD2 and PD3, which are controlled by switches SW0, SW1, SW2 and SW3 respectively. The photodiodes PD0, PD1, PD2 and PD3 are commonly coupled to the floating diffusion point FD0, and the floating diffusion point FD0 is coupled to the ramp voltage Vr0 through a capacitor Cr0 and coupled to the supply voltage VDD through the switch SWR0. The floating diffusion point FD0 is coupled to the positive input terminal 105p of the cluster differential analog-to-digital converter 106 through the switch SWC0. There may be a source follower SF0 between the floating diffusion point FD0 and the switch SWC0. In the pixel group 104b, there are four photodiodes PD4, PD5, PD6 and PD7, which are controlled by switches SW4, SW5, SW6 and SW7 respectively. The photodiodes PD4, PD5, PD6 and PD7 are commonly coupled to the floating diffusion point FD1, and the floating diffusion point FD1 is coupled to the ramp voltage Vr1 through a capacitor Cr1, and is coupled to the supply voltage VDD through a switch SWR1. The floating diffusion point FD1 is coupled to the negative input terminal 105n of the cluster differential analog-to-digital converter 106 through a switch SWC1. There may be a source follower SF1 between the floating diffusion point FD1 and the switch SWC1. In the pixel group 104c, there are four photodiodes PD8, PD9, PD10 and PD11, which are controlled by switches SW8, SW9, SW10 and SW11 respectively. Photodiodes PD8, PD9, PD10 and PD11 are commonly coupled to a floating diffusion point FD2, which is coupled to a ramp voltage Vr0 via a capacitor Cr2 and to a supply voltage VDD via a switch SWR2. The floating diffusion point FD2 is coupled to a positive input terminal 105p of a cluster differential analog-to-digital converter 106 via a switch SWC2. A source follower SF2 may be provided between the floating diffusion point FD2 and the switch SWC2. In the pixel group 104d, there are four photodiodes PD12, PD13, PD14 and PD15, which are controlled by switches SW12, SW13, SW14 and SW15, respectively. The photodiodes PD12, PD13, PD14 and PD15 are commonly coupled to a floating diffusion point FD3, which is coupled to a ramp voltage Vr1 via a capacitor Cr3 and to a supply voltage VDD via a switch SWR3. The floating diffusion FD3 is coupled to the negative input terminal 105n of the cluster differential analog-to-digital converter 106 via the switch SWC3. A source follower SF3 may be provided between the floating diffusion FD3 and the switch SWC3.That is, the floating diffusion points FD0 and FD2 of the pixel groups 104a and 104c coupled to the positive input terminal 105p are coupled to the ramp voltage Vr0 via capacitors Cr0 and Cr2, and the floating diffusion points FD1 and FD3 of the pixel groups 104b and 104d coupled to the negative input terminal 105n are coupled to the ramp voltage Vr1 via capacitors Cr1 and Cr3.

[0067] Please refer to Figure 1B , Figure 1A The ramp voltages Vr0 and Vr1 in the figure are generated by the ramp voltage generator 150, where Vcm is the common mode voltage and clock is the frequency. The ramp voltages Vr0 and Vr1 are a pair of ramp voltages, one of which is upward and the other is downward. The directionality of the ramp voltages Vr0 and Vr1 is changed when reading the positive input terminal 105p and the negative input terminal 105n. For example (but not limited to), when reading the positive input terminal 105p, Vr0 is upward and Vr1 is downward, and when reading the negative input terminal 105n, Vr0 is downward and Vr1 is upward. The ramp voltage Vr0 is connected to a capacitor CrP (e.g. Figure 1A The capacitors Cr0, Cr2, etc. in the positive input terminal 105p are coupled to the floating diffusion point FDP (e.g. Figure 1A The ramp voltage Vr1 is connected to the floating diffusion points FD0, FD2, etc. in the circuit, and the ramp voltage Vr1 is connected to the floating diffusion points FD0, FD2, etc. in the circuit. Figure 1A The capacitors Cr1, Cr3, etc. in the negative input terminal 105n are coupled to the floating diffusion point FDN (eg Figure 1A The floating diffusion point FDP is coupled to the positive input terminal 105p of the cluster differential analog-to-digital converter 106, and the floating diffusion point FDN is coupled to the negative input terminal 105n of the cluster differential analog-to-digital converter 106. A source follower SFP may be provided between the floating diffusion point FDP and the positive input terminal 105p, and a source follower SFN may be provided between the floating diffusion point FDN and the negative input terminal 105n.

[0068] In the pixel group 104a, the control signal tx0 controls the switch SW0, the control signal tx1 controls the switch SW1, the control signal tx2 controls the switch SW2, and the control signal tx3 controls the switch SW3. The reset signal rst0 controls the switch SWR0, and the selection signal sel0 controls the switch SWC0. In the pixel group 104b, the control signal tx4 controls the switch SW4, the control signal tx5 controls the switch SW5, the control signal tx6 controls the switch SW6, and the control signal tx7 controls the switch SW7. The reset signal rst1 controls the switch SWR1, and the selection signal sel1 controls the switch SWC1. In the pixel group 104c, the control signal tx8 controls the switch SW8, the control signal tx9 controls the switch SW9, the control signal tx10 controls the switch SW10, and the control signal tx11 controls the switch SW11. The reset signal rst2 controls the switch SWR2, and the selection signal sel2 controls the switch SWC2. In the pixel group 104d, the control signal tx12 controls the switch SW12, the control signal tx13 controls the switch SW13, the control signal tx14 controls the switch SW14, and the control signal tx15 controls the switch SW15. The reset signal rst3 controls the switch SWR3, and the selection signal sel3 controls the switch SWC3. According to the instruction of the controller 114, the pixel driver 102 controls the above switches through the above control signals tx0-tx15, the reset signals rst0-rst3 and the selection signals sel0-sel3, thereby controlling the operation of the pixel groups 104a-104d.

[0069] The same number of pixels can be implemented in different configurations. For example, for 16*16 pixels, in addition to the above configuration, 32 cluster differential analog-to-digital converters 106 can also be used, each cluster differential analog-to-digital converter 106 has two pixel groups coupled to the positive input terminal 105p and two pixel groups coupled to the negative input terminal 105n, and each pixel group has two photodiodes and a floating diffusion point. The structure of this configuration can be represented by Figure 1A The embodiments shown can be derived by analogy, so they will not be described in detail.

[0070] See also Figure 2AIn one embodiment, the dynamic vision sensor 200 has a number of 8*8 pixels. The pixels on the dynamic vision sensor 200 can be divided into four pixel subsets. The first pixel subset includes pixels 2000, 2001, 2002, ..., 2015, that is, 4*4 pixels in the upper left. The second pixel subset includes pixels 2100, 2101, 2102, ..., 2115, that is, 4*4 pixels in the upper right. The third pixel subset includes pixels 2200, 2201, 2202, ..., 2215, that is, 4*4 pixels in the lower left. The fourth pixel subset includes pixels 2300, 2301, 2302, ..., 2315, that is, 4*4 pixels in the lower right.

[0071] See also Figure 2B , Figure 2A The four subsets of pixels in are similar to Figure 1A The circuit structure shown is coupled to four cluster differential analog-to-digital converters, respectively. Each pixel in the first pixel subset is coupled to the cluster differential analog-to-digital converter 210, each pixel in the second pixel subset is coupled to the cluster differential analog-to-digital converter 211, each pixel in the third pixel subset is coupled to the cluster differential analog-to-digital converter 212, and each pixel in the fourth pixel subset is coupled to the cluster differential analog-to-digital converter 213.

[0072] See also Figure 2C , Figure 2B The output data of the four cluster differential analog-to-digital converters in are stored in cluster memories 2000c to 2315c, respectively. Among them, the output data of the cluster differential analog-to-digital converter 210 is stored in cluster memories 2000c, 2001c, 2002c, ..., 2015c, that is, the 4*4 cluster memories on the upper left. The output data of the cluster differential analog-to-digital converter 211 is stored in cluster memories 2100c, 2101c, 2102c, ..., 2115c, that is, the 4*4 cluster memories on the upper right. The output data of the cluster differential analog-to-digital converter 212 is stored in cluster memories 2200c, 2201c, 2202c, ..., 2215c, that is, the 4*4 cluster memories on the lower left. The output data of the cluster differential analog-to-digital converter 213 is stored in cluster memories 2300c, 2301c, 2302c, ..., 2315c, i.e., the 4*4 cluster memories at the lower right. The cluster memory circuit can be incorporated into the pixel group circuit to reduce the chip area and improve the chip area utilization efficiency; it can also be incorporated into the cluster differential analog-to-digital converter circuit, which is beneficial to shorten the signal delay time (latence), facilitate the high-speed operation of the dynamic sensing mode, and improve the chip area utilization efficiency.

[0073] See also Figure 2D , in one embodiment, Figure 2A The pixel array shown and Figure 2B The cluster differential analog-to-digital converter array shown can be stacked above and below the substrate 230. To achieve a more ideal circuit layout, the pixel subsets and the corresponding cluster differential analog-to-digital converters can be aligned vertically. Figure 2A The first pixel subset in the cluster differential analog-to-digital converter 210 is located in the same range in the top view, and the pixel subset 221 (corresponding to Figure 2A The second pixel subset in the cluster differential analog-to-digital converter 211 is located in the same range in the top view, and the pixel subset 222 (corresponding to Figure 2A The third pixel subset in the cluster differential analog-to-digital converter 212 is in the same range in the top view, and the pixel subset 223 (corresponding to Figure 2A The fourth pixel subset in the ) and the corresponding cluster differential analog-to-digital converter 213 are located in the same range in the top view. This approach can shorten the settling time of the analog signal, greatly improve the circuit operation speed, and facilitate the high-speed operation of the dynamic sensing mode.

[0074] See also Figure 2E In another embodiment, Figure 2A The pixel array shown and Figure 2B The cluster differential analog-to-digital converter array shown can be stacked up and down on the same side of the substrate 240, wherein the pixel array can be located above or below the cluster differential analog-to-digital converter array, and the above is taken as an example here. To achieve a more ideal circuit layout, the pixel subsets and the corresponding cluster differential analog-to-digital converters can be aligned up and down. For example, the pixel subset 250 (corresponding to Figure 2A The first pixel subset in the cluster differential analog-to-digital converter 210 is located in the same range in the top view, and the pixel subset 251 (corresponding to Figure 2A The second pixel subset in the cluster differential analog-to-digital converter 211 is located in the same range in the top view, and the pixel subset 252 (corresponding to Figure 2A The third pixel subset in the cluster differential analog-to-digital converter 212 is in the same range in the top view, and the pixel subset 253 (corresponding to Figure 2A The fourth pixel subset in the ) and the corresponding cluster differential analog-to-digital converter 213 are located in the same range in the top view. This approach can shorten the settling time of the analog signal, greatly improve the circuit operation speed, and facilitate the high-speed operation of the dynamic sensing mode.

[0075] See also Figure 2F In another embodiment, Figure 2A The pixel array shown and Figure 2B The cluster differential analog-to-digital converter array shown can be arranged on the same side of substrate 260 and use the same component layer and metal interconnect layer. To achieve a more ideal circuit layout, the pixel subsets and the corresponding cluster differential analog-to-digital converters can be staggered in the same area. For example, the pixel subset 270 (corresponding to Figure 2A The first pixel subset in the cluster differential analog-to-digital converter 210 is arranged in an interlaced manner in the same area. The pixel subset 271 (corresponding to Figure 2A The second pixel subset in the cluster differential analog-to-digital converter 211 is arranged in an interlaced manner in the same area. The pixel subset 272 (corresponding to Figure 2A The third pixel subset in the cluster differential analog-to-digital converter 212 is arranged in an interlaced manner in the same area, and the pixel subset 273 (corresponding to Figure 2A The fourth pixel subset in the embodiment of the present invention and the corresponding cluster differential analog-to-digital converter 213 are arranged alternately in the same area. This method can shorten the settling time of the analog signal, greatly improve the circuit operation speed, and is conducive to the high-speed operation of the dynamic sensing mode.

[0076] Please refer to Figure 2AIn the dynamic sensing mode, when a light emitter (not shown) emits an incident light (for example but not limited to infrared light), if there is an object (not shown) in front of the dynamic vision sensor 200, each pixel of the dynamic vision sensor 200 receives the reflected light reflected from the object through the photodiode. In one embodiment, the dynamic vision sensor 200 determines whether the object is in a moving state by the intensity difference of the reflected light received between adjacent pixels. In another embodiment, the pixels in the same pixel subset form a pixel sequence in an order from top to bottom and from left to right. By calculating the intensity difference of the reflected light received between two consecutive frames of the same pixel, or calculating the intensity difference of the reflected light received between two adjacent pixel groups (represented by P and N respectively) in the pixel sequence respectively coupled to the positive input terminal 105p and the negative input terminal 105n, it can be determined whether the object is moving. For example, taking the first pixel subset as an example, the pixel sequence may be 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, wherein pixels 2000, 2001, 2004, 2005 are commonly coupled to a first floating diffusion point to form a first pixel group, pixels 2002, 2003, 2006, 2007 are commonly coupled to a second floating diffusion point to form a second pixel group, pixels 2008, 2009, 2012, 2013 are commonly coupled to a third floating diffusion point to form a third pixel group, and pixels 2010, 2011, 2014, 2015 are commonly coupled to a fourth floating diffusion point to form a fourth pixel group. The first floating diffusion point and the third floating diffusion point are coupled to the positive input terminal 105p, and the second floating diffusion point and the fourth floating diffusion point are coupled to the negative input terminal 105n. When the object moves, the intensity of the reflected light received by the same pixel between two consecutive frames will change. In addition, the intensity of the reflected light received by the pixels corresponding to each other in the adjacent pixel group will also change. Both of these will lead to differences in the accumulated charge of the photodiode. Therefore, the reflected light intensity between the same pixel in the pixel sequence between two consecutive frames can be subtracted, that is, the voltage values ​​of the two frames are converted into digital signals and then subtracted to determine whether the object is moving. Alternatively, the two adjacent pixel groups in the pixel sequence respectively coupled to the positive input terminal 105p and the negative input terminal 105n can be compared two by two, and the reflected light intensity between the pixels corresponding to each other in the two pixel groups can be subtracted, that is, the voltage values ​​of the multiple corresponding pixels are subtracted and converted into digital signals. This method can obtain a better signal-to-noise ratio (SNR). Alternatively, the voltage values ​​of the plurality of corresponding pixels are converted into digital signals and then subtracted to obtain the reflected light intensity difference to determine whether the object is moving.For example, the reflected light intensities of pixels with corresponding positions in the first pixel group (coupled to the positive input terminal 105p) and the second pixel group (coupled to the negative input terminal 105n) (for example, pixels 2000 and 2002, pixels 2001 and 2003, pixels 2004 and 2006, pixels 2005 and 2007) can be subtracted to determine whether the object is moving.

[0077] Optionally, the dynamic vision sensor 100 adopts a binning mode to superimpose adjacent pixels as a pixel output signal. In this case, the accumulated charge of each pixel in the same pixel group in the pixel sequence can be added to obtain the sum of the reflected light intensity of the pixel group, and the sum of the reflected light intensity of two consecutive frames can be compared to determine whether the object is moving. For example, for the first pixel group (including pixels 2000, 2001, 2004, and 2005), the sum of the reflected light intensity of pixels 2000, 2001, 2004, and 2005 can be calculated, and the sum of the reflected light intensity of two consecutive frames can be subtracted to determine whether the object is moving. Alternatively, the sum of the reflected light intensity of each pixel group in the pixel sequence can be calculated separately, and the sum of the reflected light intensity of two adjacent pixel groups respectively coupled to the positive input terminal 105p and the negative input terminal 105n can be compared to determine whether the object is moving. For example, for the first pixel group (coupled to the positive input terminal 105p) and the second pixel group (coupled to the negative input terminal 105n), the sum of the first reflected light intensities of pixels 2000, 2001, 2004, and 2005 and the sum of the second reflected light intensities of pixels 2002, 2003, 2006, and 2007 can be calculated, and the sum of the first reflected light intensities and the sum of the second reflected light intensities can be subtracted to determine whether the object is moving. This method can obtain a better signal-to-noise ratio (SNR).

[0078] As described above, pixels 2000, 2001, 2004, and 2005 are commonly coupled to a first floating diffusion point to form a first pixel group, pixels 2002, 2003, 2006, and 2007 are commonly coupled to a second floating diffusion point to form a second pixel group, pixels 2008, 2009, 2012, and 2013 are commonly coupled to a third floating diffusion point to form a third pixel group, and pixels 2010, 2011, 2014, and 2015 are commonly coupled to a fourth floating diffusion point to form a fourth pixel group. The first floating diffusion point and the third floating diffusion point are coupled to the positive input terminal 105p, and the second floating diffusion point and the fourth floating diffusion point are coupled to the negative input terminal 105n. For example, for the first pixel group, pixel 2000 may include Figure 1A The photodiode PD0 and the switch SW0 in the pixel 2001 may include Figure 1A The photodiode PD1 and the switch SW1 in the pixel 2004 may include Figure 1AThe pixel 2005 may include a photodiode PD2 and a switch SW2. Figure 1A In this case, the first floating diffusion point may be the floating diffusion point FD0. For the second pixel group, the pixel 2002 may include Figure 1A The photodiode PD4 and the switch SW4 in the pixel 2003 may include Figure 1A The photodiode PD5 and the switch SW5 in the pixel 2006 may include Figure 1A The photodiode PD6 and the switch SW6 in the pixel 2007 may include Figure 1A The photodiode PD7 and switch SW7 in the cluster differential analog-to-digital converter 106 are connected. At this time, the second floating diffusion point can be the floating diffusion point FD1. The same can be said. When performing the operation of pixel 2002 minus pixel 2000, the pixel driver 102 controls the switch SW0 in pixel 2000 and the switch SW4 in pixel 2002 to be turned on successively, and turns off the switches SW1, SW2, SW3, SW5, SW6, SW7, etc. in the remaining pixels. At this time, the positive input terminal 105p of the cluster differential analog-to-digital converter 106 receives the first signal from the photodiode PD0 in the pixel 2000, and the negative input terminal 105n of the cluster differential analog-to-digital converter 106 receives the second signal from the photodiode PD4 in the pixel 2002, and performs a subtraction operation on the first signal and the second signal. The same can be said.

[0079] The above is the operation method of the dynamic vision sensor 100 in the dynamic sensing mode. Figure 1A The circuit structure shown can also enable the dynamic vision sensor 100 to operate in a two-dimensional image sensing mode without increasing the hardware cost. In the two-dimensional image sensing mode, no subtraction operation is performed between the pixels, but all are read into the memory. Therefore, an operation method different from the dynamic sensing mode is required so that each pixel is read into the cluster memory 108 in the positive direction. An exemplary reading method is to alternately read the pixel coupled to the positive input terminal 105p and the pixel coupled to the negative input terminal 105n. In this example, when the pixel coupled to the positive input terminal 105p is read, the pixel driver 102 turns off all the photodiodes coupled to the negative input terminal 105n. At this time, the slope voltage Vr0 of the positive input terminal is positive, and the slope voltage Vr1 of the negative input terminal is negative. When the pixel coupled to the negative input terminal 105n is read, the pixel driver 102 turns off all the photodiodes coupled to the positive input terminal 105p. At this time, the slope voltage Vr0 of the positive input terminal is negative, and the slope voltage Vr1 of the negative input terminal is positive. The reverse is also possible, that is, when the slope voltage Vr0 or Vr1 is negative, the reading is performed, and when it is positive, it is turned off. In this way, the pixels coupled to the positive input terminal 105p and the pixels coupled to the negative input terminal 105n can all be read into the cluster memory 108 without interfering with each other.

[0080] See also Figure 3 The present invention includes a method 300 for switching a dynamic vision sensor 100 between a dynamic sensing mode and a two-dimensional image sensing mode. In step 302, the dynamic vision sensor 100 operates in the dynamic sensing mode. At this time, in order to detect fast-moving objects, the dynamic vision sensor 100 has a higher frame rate per second (FPS) and less image information, and optionally adopts a pixel merging mode (binning mode) to superimpose adjacent pixels as a pixel output signal. When the movement of the object is detected, the dynamic vision sensor 100 will switch to the two-dimensional image sensing mode to detect the complete situation of each pixel of the object. In step 304, the dynamic vision sensor 100 determines whether an object of interest is detected. Here, "object of interest" refers to a moving object. Through the pixel subtraction method described above, the image signal processor 122 can determine whether the object is moving. For example, after subtracting the pixels corresponding to each other in two consecutive frames or two adjacent pixel groups of the same pixel, if the image signal processor 122 determines that the movement of the object meets a sensing mode switching condition, it proceeds to step 306. If not, return to step 302. In this embodiment, the sensing mode switching condition can be determined based on (but not limited to) the number of pixels whose reflected light intensity changes, the moving speed of the object, the moving direction of the object, the size of the object, etc.

[0081] In step 306, the dynamic vision sensor 100 operates in a two-dimensional image sensing mode. At this time, in order to read the details of the object in detail, the dynamic vision sensor 100 has a lower frame rate per second and a higher resolution, and optionally adopts a pixel merging mode. In step 308, if the dynamic vision sensor 100 has completed the desired detailed inspection of the object (such as the size, color, type, etc. of the object), then return to step 302, so that the dynamic vision sensor 100 returns to the dynamic sensing mode to detect the next moving object. If not, then return to step 306 and continue to read the details of the object in the two-dimensional image sensing mode.

[0082] When the exposure of a pixel begins, the reset signal (e.g., rst0-rst3) and control signal (e.g., tx0-tx15) associated with the pixel are simultaneously set (asserted) to reset the floating diffusion point (e.g., FD0-FD3) associated with the pixel and to dump the charge in the photodiode (e.g., PD0-PD15) associated with the pixel, and then the reset signal and the control signal are reset (de-asserted). When the exposure of a pixel ends, the reset signal associated with the pixel is set to reset the floating diffusion point associated with the pixel, and then the reset signal is reset, the control signal is set and then reset to dump the charge in the photodiode associated with the pixel, and the readout time of the pixel begins. Please refer to Figure 4AIn the two-dimensional image sensing mode, the exposure time of each pixel is longer and the number of frames per second is lower. Taking the first pixel group (including pixels 2000, 2001, 2004, 2005, coupled to the positive input terminal 105p) and the second pixel group (including pixels 2002, 2003, 2006, 2007, coupled to the negative input terminal 105n) as examples, the reset signal associated with the first pixel group is rst0 and the floating diffusion point is FD0, while the reset signal associated with the second pixel group is rst1 and the floating diffusion point is FD1. At the beginning of time INT0, the reset signal rst0 and the control signal tx0 associated with pixel 2000 are set at the same time and then reset, so that the floating diffusion point FD0 is reset and the charge in the photodiode PD0 is dumped, so that the pixel 2000 starts to be exposed. Then, at the end of time INT0, the reset signal rst0 is set and then reset, so that the floating diffusion point FD0 is reset, and the exposure time of the pixel 2000 ends. At the beginning of time ADC0, the control signal tx0 is set and then reset, so that the charge in the photodiode PD0 is dumped, and the pixel 2000 is read out. At this time, the signal of the pixel 2000 is input to the cluster differential analog-to-digital converter 106 for analog-to-digital conversion, and the converted signal is input to the cluster memory 108. At the beginning of time INT1, the reset signal rst0 and the control signal tx1 associated with the pixel 2001 are simultaneously set and then reset, so that the floating diffusion point FD0 is reset and the charge in the photodiode PD1 is dumped, so that the pixel 2001 starts to be exposed. Then, at the end of time INT1, the reset signal rst0 is set and then reset, so that the floating diffusion point FD0 is reset, and the exposure time of the pixel 2001 ends. At the beginning of time ADC1, the control signal tx0 is set and then reset, so that the charge in the photodiode PD1 is dumped and the pixel 2001 is read out. At this time, the signal of the pixel 2001 is input to the cluster differential analog-to-digital converter 106 for analog-to-digital conversion, and the converted signal is input to the cluster memory 108. Similarly, the pixel 2004 starts to be exposed at time INT2 and is read out at time ADC2. The pixel 2005 starts to be exposed at time INT3 and is read out at time ADC3. After all the pixels of the first pixel group are exposed in sequence, the pixels of the second pixel group start to be exposed in sequence. At the beginning of time INT4, the reset signal rst1 and the control signal tx4 associated with the pixel 2002 are simultaneously set and then reset, so that the floating diffusion point FD1 is reset and the charge in the photodiode PD4 is dumped, so that the pixel 2002 starts to be exposed. Subsequently, at the end of time INT4, the reset signal rst1 is set and then reset, so that the floating diffusion point FD1 is reset and the exposure time of the pixel 2002 ends.At the beginning of time ADC4, the control signal tx1 is set and then reset, so that the charge in the photodiode PD1 is dumped and the pixel 2002 is read out. At this time, the signal of the pixel 2002 is input to the cluster differential analog-to-digital converter 106 for analog-to-digital conversion, and the converted signal is input to the cluster memory 108. Similarly, the pixel 2003 starts to be exposed at time INT5 and is read out at time ADC5. The pixel 2006 starts to be exposed at time INT6 and is read out at time ADC6. The pixel 2007 starts to be exposed at time INT7 and is read out at time ADC7. At this time, all pixels of the first pixel group and the second pixel group are read out. At time DO, all converted signals in the cluster memory 108 are input to the image signal processor 122 via the sensor data interface 110 for subsequent calculation.

[0083] See also Figure 4BIn the motion sensing mode, the exposure time of each pixel is shorter and the number of frames per second is higher. Taking the first pixel group (including pixels 2000, 2001, 2004, 2005, coupled to the positive input terminal 105p) and the second pixel group (including pixels 2002, 2003, 2006, 2007, coupled to the negative input terminal 105n) as examples, the reset signal associated with the first pixel group is rst0, the floating diffusion point is FD0, and the reset signal associated with the second pixel group is rst1, the floating diffusion point is FD1. As described above, the reflected light intensities of the pixels (for example, pixels 2000 and 2002, pixels 2001 and 2003, pixels 2004 and 2006, pixels 2005 and 2007) in the first pixel group (coupled to the positive input terminal 105p) and the second pixel group (coupled to the negative input terminal 105n) that have corresponding positions can be subtracted to determine whether the object is moving. At the beginning of time INT0P, the reset signal rst0 and the control signal tx0 associated with the pixel 2000 are simultaneously set and then reset, so that the floating diffusion point FD0 is reset and the charge in the photodiode PD0 is dumped, so that the pixel 2000 starts to be exposed. At the end of time INT0P, the reset signal rst0 is set and then reset, so that the floating diffusion point FD0 is reset, and the exposure time of the pixel 2000 ends. At the beginning of time INT0N, the reset signal rst1 and the control signal tx4 associated with the pixel 2002 are simultaneously set and then reset, so that the floating diffusion point FD1 is reset, and the charge in the photodiode PD4 is dumped, so that the pixel 2002 starts to be exposed. At the end of time INT0N, the reset signal rst1 is set and then reset, so that the floating diffusion point FD1 is reset, and the exposure time of the pixel 2002 ends. Subsequently, at time ADC0, the control signal tx0 associated with the pixel 2000 and the control signal tx4 associated with the pixel 2002 are simultaneously set and then reset, so that the pixels 2000 and 2002 are read out. The signal of the pixel 2000 and the signal of the pixel 2002 are subtracted in the cluster differential analog-to-digital converter 106, and the cluster differential analog-to-digital converter 106 converts the analog signal obtained by the subtraction into a digital signal and inputs the digital signal into the cluster memory 108. Similarly, at time INT1P, the pixel 2001 starts to be exposed, at time INT1N, the pixel 2003 starts to be exposed, at time ADC1, the pixels 2001 and 2003 are read out and subtracted. At time INT2P, the pixel 2004 starts to be exposed, at time INT2N, the pixel 2006 starts to be exposed, and at time ADC2, the pixels 2004 and 2006 are read out and subtracted. At time INT3P, pixel 2005 starts to be exposed, at time INT3N, pixel 2007 starts to be exposed, and at time ADC3, pixels 2005 and 2007 are read out and subtracted.At time DO, all converted signals in the cluster memory 108 are input to the image signal processor 122 via the sensor data interface 110 for subsequent operations.

[0084] See also Figure 5In the dynamic sensing mode, the analog signal of each pixel can also be converted into a digital signal first, and then the digital signal is differentially operated. At the beginning of time INT0P, the reset signal rst0 and the control signal tx0 associated with the pixel 2000 are set and then reset at the same time, so that the floating diffusion point FD0 is reset and the charge in the photodiode PD0 is dumped, so that the pixel 2000 starts to be exposed. At the end of time INT0P, the reset signal rst0 is set and then reset, so that the floating diffusion point FD0 is reset, and the exposure time of the pixel 2000 ends. Subsequently, at the beginning of time ADC0P, the control signal tx0 is set and then reset, so that the charge in the photodiode PD0 is dumped, and the pixel 2000 is read out. The signal of the pixel 2000 is converted into a digital signal in the cluster differential analog-to-digital converter 106, and this digital signal is input into the cluster memory 108. At the beginning of time INTON, the reset signal rst1 and the control signal tx4 associated with the pixel 2002 are simultaneously set and then reset, so that the floating diffusion point FD1 is reset and the charge in the photodiode PD4 is dumped, so that the pixel 2002 starts to be exposed. At the end of time INTON, the reset signal rst1 is set and then reset, so that the floating diffusion point FD1 is reset, and the exposure time of the pixel 2002 ends. Subsequently, at the beginning of time ADCON, the control signal tx1 is set and then reset, so that the charge in the photodiode PD1 is dumped, and the pixel 2002 is read out. The signal of the pixel 2002 is converted into a digital signal in the cluster differential analog-to-digital converter 106, and the digital signal is input into the cluster memory 108. Similarly, at time INT1P, pixel 2001 starts to be exposed, at time ADC1P, pixel 2001 is read out, at time INT1N, pixel 2003 starts to be exposed, at time ADC1N, pixel 2003 is read out. At time INT2P, pixel 2004 starts to be exposed, at time ADC2P, pixel 2004 is read out, at time INT2N, pixel 2006 starts to be exposed, at time ADC2N, pixel 2006 is read out. At time INT3P, pixel 2005 starts to be exposed, at time ADC3P, pixel 2005 is read out, at time INT3N, pixel 2007 starts to be exposed, at time ADC3N, pixel 2007 is read out.At time DDO or before DDO, the digital signals acquired at time ADC0P and ADC0N are subtracted (for example, processed by a digital circuit), the digital signals acquired at time ADC1P and ADC1N are subtracted, the digital signals acquired at time ADC2P and ADC2N are subtracted, and the digital signals acquired at time ADC3P and ADC3N are subtracted, and the values ​​obtained after the subtraction operation are input into the image signal processor 122 via the sensor data interface 110 for subsequent operation.

[0085] From the above description, it can be understood that the present invention can have a higher operating speed and a better signal-to-noise ratio, and is suitable for high-speed operation in a dynamic sensing mode, and can also enter a high-resolution two-dimensional image sensing mode by controlling the timing change. The switching between the two sensing modes can be controlled by an algorithm through a controller.

[0086] The above description is only used to explain the preferred embodiments of the present invention, and is not intended to limit the present invention in any form. Therefore, any modifications or changes made to the present invention under the same inventive spirit should still be included in the scope of protection intended by the present invention.

Claims

1. A dynamic vision sensor, characterized in that: include: A sensor; as well as an image signal processor, The sensor includes: a pixel driver; A plurality of pixel groups, each pixel group includes a plurality of pixels and a floating diffusion point, each pixel includes a photodiode and a switch, the switch controls whether the charge accumulated in the photodiode is transferred to the floating diffusion point, the switch in each pixel is controlled by the pixel driver, and each pixel in the same pixel group is commonly coupled to the floating diffusion point; a cluster differential analog-to-digital converter having a positive input terminal and a negative input terminal; A cluster of memory; a sensor data interface; and A controller, Wherein, the image signal processor includes an algorithm; Half of the plurality of pixel groups are coupled to the positive input terminal of the cluster differential analog-to-digital converter, and the rest are coupled to the negative input terminal of the cluster differential analog-to-digital converter; The cluster differential analog-to-digital converter outputs data to the cluster memory, and the cluster memory transmits the data to the image signal processor via the sensor data interface. The image signal processor operates the data using the algorithm and transmits an operation result to the controller. According to the calculation result, the controller controls the pixel driver, the cluster differential analog-to-digital converter, the cluster memory and the sensor data interface to switch between a dynamic sensing mode and a two-dimensional image sensing mode.

2. The dynamic vision sensor according to claim 1, characterized in that: Each floating diffusion point of the pixel group coupled to the positive input terminal is coupled to a first ramp voltage via a respective capacitor, and each floating diffusion point of the pixel group coupled to the negative input terminal is coupled to a second ramp voltage via a respective capacitor; The first slope voltage and the second slope voltage are a slope voltage pair in which one of the first slope voltage and the second slope voltage is upward and the other is downward, and the directions of both slope voltages are changed when the positive input terminal and the negative input terminal are read.

3. The dynamic vision sensor according to claim 1, characterized in that: The circuit of the cluster memory is integrated into the circuit of the plurality of pixel groups.

4. The dynamic vision sensor according to claim 1, characterized in that: The circuit of the cluster memory is incorporated into the circuit of the cluster differential analog-to-digital converter.

5. The dynamic vision sensor according to claim 1, characterized in that: The plurality of pixel groups and the corresponding cluster differential analog-to-digital converter stack.

6. The dynamic vision sensor according to claim 5, characterized in that: The plurality of pixel groups and the cluster differential analog-to-digital converter are respectively stacked on two opposite sides of a substrate.

7. The dynamic vision sensor according to claim 5, characterized in that: The plurality of pixel groups and the cluster differential analog-to-digital converter are disposed on the same side of a substrate and use the same device layer and metal connection layer.

8. The dynamic vision sensor according to claim 5, characterized in that: The plurality of pixel groups and the cluster differential analog-to-digital converter are stacked in layers on the same side of a substrate.

9. The dynamic vision sensor according to claim 1, characterized in that: In the motion sensing mode, the cluster differential analog-to-digital converter receives the change of the reflected light intensity from an object between two consecutive frames of the same pixel to determine whether the object is moving.

10. The dynamic vision sensor according to claim 1, characterized in that: In the dynamic sensing mode, the plurality of pixels form a pixel sequence in a top-to-bottom and left-to-right order, the pixel sequence comprising a first pixel group and a second pixel group adjacent to each other, the first pixel group being coupled to the positive input terminal of the cluster differential analog-to-digital converter, and the second pixel group being coupled to the negative input terminal of the cluster differential analog-to-digital converter; The cluster differential analog-to-digital converter receives the change of the reflected light intensity from an object through the pixels in the first pixel group and the second pixel group that have corresponding positions, and determines whether the object is moving.

11. The dynamic vision sensor according to claim 10, characterized in that: The cluster differential analog-to-digital converter converts the voltage values ​​of the plurality of corresponding pixels into digital signals after subtracting them, so as to determine whether the object is moving.

12. The dynamic vision sensor according to claim 10, characterized in that: The cluster differential analog-to-digital converter converts the voltage values ​​of the plurality of corresponding pixels into digital signals and then performs subtraction to determine whether the object is moving.

13. The dynamic vision sensor according to claim 1, characterized in that: In the dynamic sensing mode, the cluster differential analog-to-digital converter adds the accumulated charges of each pixel in the same pixel group receiving reflected light from an object to obtain the sum of the reflected light intensities of the pixel group, and subtracts the sums of the reflected light intensities of two consecutive frames to determine whether the object is moving.

14. The dynamic vision sensor according to claim 1, characterized in that: In the dynamic sensing mode, the plurality of pixels form a pixel sequence in a top-to-bottom and left-to-right order, the pixel sequence comprising a first pixel group and a second pixel group adjacent to each other, the first pixel group being coupled to the positive input terminal of the cluster differential analog-to-digital converter, and the second pixel group being coupled to the negative input terminal of the cluster differential analog-to-digital converter; The cluster differential analog-to-digital converter adds the accumulated charges of each pixel in the first pixel group and the second pixel group receiving reflected light from an object to obtain the sum of the reflected light intensities of the first pixel group and the second pixel group, and subtracts the sum of the reflected light intensities of the first pixel group and the second pixel group to determine whether the object is moving.

15. The dynamic vision sensor according to claim 1, characterized in that: If the movement of the object meets a sensing mode switching condition, the controller controls the pixel driver, the cluster differential analog-to-digital converter, the cluster memory and the sensor data interface to switch between the dynamic sensing mode and the two-dimensional image sensing mode.

16. The dynamic vision sensor according to claim 15, characterized in that: The sensing mode switching condition can be determined based on at least one of the number of pixels whose reflected light intensity changes, the moving speed of the object, the moving direction of the object, and the size of the object.

Images