Event-based visual sensor for flicker environment detection and detection method thereof

Abstract

According to the present invention, an event-based vision sensor (EVS) is provided that can capture the behavior of a moving object that is observed in a flickering environment. The EVS includes an EVS panel, a window counter unit (10), an on / off counter unit (20), and an Identity of Condition (IoC) unit (30). The window counter unit (10) divides the pixel array into a plurality of windows, the on / off counter unit counts on-events and off-events occurring for each of the plurality of windows, and the IoC unit determines whether to shield each of the corresponding windows according to the counts.

Description

Technical Field

[0001] The present invention relates to imaging devices, and more particularly, to an event vision sensor with enhanced flicker environment detection. An event vision sensor (EVS) can be used in a camera system. Background Art

[0002] An event-based vision sensor (EVS) is a sensor that detects changes in brightness of each pixel in an image sensor and only outputs the changed data. EVS simulates the mechanism of human eye perception of light and provides high-speed, low-latency data output. With the latest advances in image processing technology, EVS has begun to be used in vision systems such as automotive operations.

[0003] The working principle of EVS is as follows (see Figure 1 ).

[0004] 1. The sensor’s photodetector converts the linear current output of the photodiode portion into a logarithmic voltage in response to the incident light intensity.

[0005] 2. The amplifier detects the difference between the input voltage and the reset voltage.

[0006] 3. When the input difference reaches the preset threshold of ON event or OFF event, the comparator detects and outputs the ON event or OFF event.

[0007] 4. When an ON event or OFF event is output, the threshold is updated.

[0008] That is, when the voltage change due to incident light exceeds the positive side threshold of the reference voltage (i.e., change to the brighter direction), "1" is output as an on event (i.e., ON event). On the other hand, when the voltage change exceeds the negative side threshold of the reference voltage (i.e., change to the darker direction), "-1" is output as an off event (i.e., OFF event). When an event occurs, the threshold level is reset so that the brightness level at that point in time becomes the baseline. If the voltage change due to incident light does not exceed both thresholds, "0" is output as a null event (see Reference 1).

[0009] These operations allow EVS to provide high-speed, low-latency data output. However, when fluorescent lighting is used indoors, EVS will respond to the flickering of the fluorescent light and output events. This makes it impossible to capture the behavior of moving objects that need to be observed.

[0010] Before the advent of EVS, flicker environments were considered a problem even in conventional image sensors. In conventional sensors (such as charge-integrating sensors with rolling shutter systems), the phenomenon of "row banding" occurs due to differences in exposure time. In this case, the output signal is used to calculate the frequency of the row bands, and the problem can be solved by changing to an appropriate charge integration time synchronized with the frequency (see Reference 2). However, these solutions are not applicable to EVS because it is not a charge-integrating sensor. Therefore, camera systems that may be used in flicker environments do not usually use EVS.

[0011] The existing technical documents are as follows.

[0012] (1) Patrick Lichtsteiner et al., “A 128x128 120dB 15us Latency Asynchronous Temporal Contrast Vision Sensor,” IEEE Transactions on Solid-State Circuits, Vol. 43, No. 2, February 2008.

[0013] (2) U.S. Patent No. 8,031,237. Summary of the invention

[0014] Problems to be solved by the present invention

[0015] An object of the present invention is to provide an EVS that can be used in a flickering environment, so that a camera system using the EVS can eliminate event signals generated by light flicker and output signals caused by moving objects captured in pixels.

[0016] Solutions to the problem

[0017] In a first aspect, an embodiment of the present invention provides an event-based vision sensor (EVS). The EVS includes an EVS panel, and the EVS panel includes a pixel array, wherein each pixel in the pixel array generates an on event or an off event in response to a comparison between an output of a sensor signal and a threshold. The EVS includes the EVS panel, a window counter unit, an on / off counter unit, and an identification of condition (IoC) unit. The window counter unit divides the pixel array into a plurality of windows, the on / off counter unit counts the on / off events occurring for each of the plurality of windows, and the IoC unit determines whether to shield each window in the corresponding window according to the count.

[0018] In combination with the first aspect, in a possible implementation, the IoC unit is used to determine that the corresponding window is a flickering window and shield the window when the count is one or more consecutive on events or the count is one or more consecutive off events. If the on event and the off event are mixed in the count, the corresponding window is determined to be a non-flickering window, and the sensor signal is output from each pixel in the window.

[0019] In combination with the first aspect, in a possible implementation manner, the IoC unit is further configured to determine that the corresponding window is a flashing window when the one or more continuous on events and the one or more continuous off events are continuous in the count.

[0020] In combination with the first aspect, in a possible implementation, the EVS further includes a flicker detection (FD) unit. The FD unit is used to count the on event and the off event to be output from the on / off counter unit, increase the count in the case of the on event, and decrease the count in the case of the off event, and when the count exceeds a threshold, determine that the corresponding window is a flicker window, and output the determination result.

[0021] In combination with the first aspect, in a possible implementation, the IoC unit is further configured to output a signal when the on event and the off event occur simultaneously. The EVS is configured to calculate a logical sum of the signal and the result of the determination output by the FD, determine that the corresponding window is a non-flickering window according to the logical sum, and enable the sensor signal to be output from each pixel in the window.

[0022] In combination with the first aspect, in a possible implementation, the EVS further includes a frequency calculation block. The frequency calculation block is configured to calculate a flicker frequency according to a signal identifying the on event and the off event output from the IoC unit and a frame rate in operation.

[0023] In combination with the first aspect, in a possible implementation, the frequency calculation block is further used to compare the calculated flicker frequency with a preset frequency. The EVS is used to determine whether to shield each window according to a result of the comparison.

[0024] In a second aspect, an embodiment of the present invention provides a camera system, comprising an EVS according to any possible implementation of the first aspect.

[0025] In a third aspect, an embodiment of the present invention provides a method for detecting a flickering environment using an event-based vision sensor (EVS), wherein each pixel in a pixel array composed of an EVS panel generates an on event or an off event in response to a comparison of an output of a sensor signal with a threshold. The method comprises the following steps: dividing the pixel array into a plurality of windows, counting the on events and off events occurring for each of the plurality of windows, and determining that the corresponding window is a flickering window when the count is one or more consecutive on events among the on events or the count is one or more consecutive off events among the off events, and detecting the flickering environment.

[0026] In combination with the third aspect, in a possible implementation, the method further includes the following steps: when the one or more continuous on events and the one or more continuous off events are continuous in the count, determining that the corresponding window is a flickering window, and detecting the flickering environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly describe the embodiments of the present invention, the accompanying drawings will be briefly introduced as needed. Obviously, in the following description, the accompanying drawings only show some embodiments of the present invention, and even other drawings in these accompanying drawings can be drawn by those skilled in the art without paying creative work.

[0028] Figure 1 The operation of the EVS in the prior art is shown.

[0029] Figure 2 Schematic diagram of brightness changes in a flickering environment.

[0030] Figure 3 The operation of the EVS in a scintillation environment is shown.

[0031] Figure 4 The pixel array window provided by the present invention is shown.

[0032] Figure 5 Characteristics of events caused by flicker and by moving objects in the window are shown.

[0033] Figure 6 is a block diagram of a first embodiment of the present invention.

[0034] Figure 7 Shown in Figure 6 Table used in the Identification of Condition (IoC) unit in .

[0035] Figure 8The operations provided by the first embodiment of the present invention are summarized.

[0036] Fig. 9 is a block diagram of a second embodiment of the present invention.

[0037] Fig.10 An operation of determining whether or not a flicker detection (FD) unit is in a flicker environment is shown.

[0038] Fig.11 is a block diagram of a third embodiment of the present invention.

[0039] Fig.12 An operation of determining whether to shield a window in a flickering environment is shown.

[0040] Fig.13 is a block diagram of a fourth embodiment of the present invention.

[0041] Fig.14 is a block diagram of a fifth embodiment of the present invention.

[0042] Fig.15 is a block diagram of a sixth embodiment of the present invention. DETAILED DESCRIPTION

[0043] Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

[0044] Figure 2 Figure 1 is a schematic diagram of brightness changes in a flickering environment. For example, in a 50Hz AC power supply environment, a fluorescent lamp indicates a brightness change with a cycle of 10ms, similar to a sine wave.

[0045] Figure 3 The operation of the EVS in a flickering environment is shown in Figure 1. In the top row, the magnitude and direction of the brightness change that causes the event are given by Figure 2 The arrows in the sinusoidal brightness traces are shown. The middle row shows the relationship between brightness changes and thresholds, and the bottom row shows the resulting event outputs.

[0046] exist Figure 3In the top row of , the leftmost arrow represents the brightness change that causes the first event. This brightness change is a change in the brighter direction, that is, the brightness change that causes the on event. The length of the arrow represents the magnitude of the brightness change from the reset level shown in the middle row to reaching the threshold. As shown in the bottom row, as the flicker progresses, the brightness level will become higher and higher, and when the magnitude of the brightness change reaches the threshold, the first on event will occur and output "1". When the event occurs, the threshold level will be reset at this time. Then, as the flicker progresses and the brightness level becomes higher, when the magnitude of the brightness change reaches the threshold, the second on event will occur. Similarly, subsequent on events also occur. In other words, the output "1" is the same as the number of on events that have occurred.

[0047] During the period before and after the peak of the sine wave of brightness change, that is, during the period when the brightness changes from bright to dark, the brightness change will not occur because the amplitude of the brightness change is less than the threshold. Therefore, "0" will be output as no event.

[0048] Then, after the peak of the sine wave of brightness change, when the flicker changes in the dark, that is, when the brightness change causes an OFF event, similar to the above-mentioned ON event, the brightness level decreases as the flicker progresses, and when the amplitude of the brightness change reaches the threshold, the first OFF event occurs and "-1" is output. When the event occurs, the level will be reset at this time. Then, as the flicker progresses and the brightness level further decreases, when the amplitude of the brightness change reaches the threshold, the second OFF event occurs. Similarly, subsequent OFF events also occur. In other words, "-1" is output as many times as the number of OFF events that have occurred.

[0049] Therefore, in a flickering environment, the on event occurs continuously one or more times, or the off event occurs continuously one or more times, depending on the flickering cycle.

[0050] Figure 4 The pixel array window provided by the present invention is shown. The sensor part of the EVS is composed of a pixel array. In the present invention, the pixel array is divided into any number of windows, such as n×m window arrays. For example, in Figure 4 In the example, the 12×12 pixel array is divided into 3×3 windows. For each window, the number of on events or off events occurring on the pixels in the window is counted respectively.

[0051] Figure 5The characteristics of events caused by flicker and by moving objects in a window are shown. The left figure shows events in a window in a flickering environment, in which a window with all on events appears multiple times in succession, and then a window with all off events appears multiple times in succession as time goes by, that is, in response to the progress of the frame. On the other hand, in the right figure, the object moves from the right direction to the left direction, and the moving object is captured in the window. In this case, the on event and the off event occur at the two edges of the object, and the positions of the on event and the off event move from right to left over time (that is, as the frame advances). Therefore, the way events occur is very different between a window that captures a moving object as an object to be observed and a window in a flickering environment.

[0052] Figure 6 is a block diagram of the first embodiment of the present invention. In the first embodiment, the EVS includes a window counter unit 10, an on / off counter unit 20, and an identification of condition (IoC) unit 30. The window counter unit 10 divides the pixel array into windows of arbitrary size (e.g., an n×m matrix), and the on / off counter unit 20 counts the on / off events occurring for each window. When the counting is completed, the on / off counter unit 20 outputs the counting result to the IoC unit 30. As an output, if there is an on event in one or more consecutive times, the ON count is 1, otherwise the ON count is 0. Alternatively, as an output, if there is an off event in one or more consecutive times, the OFF count is 1, otherwise the OFF count is 0. The IoC unit 30 masks each window according to the value of each count, that is, decides whether to disable the sensor output (In_data) of the pixel in the window, or whether to enable the sensor output (cd_out) of the pixel in the window.

[0053] based on Figure 7 As shown in the table, the above decisions are as follows.

[0054] ·When the ON count is 1 and the OFF count is 0, it is determined that the output value of cd_out is 0 in the flickering environment, and the output of the corresponding window is shielded. ·When the ON count is 0 and the OFF count is 1, it is also determined that the output value of cd_out is 0 in the flickering environment, and the output of the corresponding window is shielded.

[0055] When the ON count is 1 and the OFF count is 1, it is determined that it is not in a flickering environment, and the output value of cd_out is 1, enabling the output of the corresponding window.

[0056] When the ON count is 0 and the OFF count is 0, it is also determined that it is not in a flickering environment, and the output value of cd_out is 1, enabling the output of the corresponding window.

[0057] Figure 8 The above operations are summarized. Finally, for a window where the value of cd_out is 1, the sensor output is enabled for each pixel in the window. Therefore, even in a flickering environment, a moving object as a target to be observed can be detected by using an event-based vision sensor (EVS).

[0058] Fig. 9 FIG. 1 is a block diagram of a second embodiment of the present invention. In the second embodiment, Figure 6 A flicker detection (FD) unit 40 is added to the block diagram of the EVS in the first embodiment shown. The input of the FD unit 40 is cd_out from the IoC unit 30, and the FD unit 40 outputs fd_out as a result of flicker detection determination. When it is determined to be in a flicker environment, Fd_out is 0, and if it is determined not to be in a flicker environment, Fd_out is 1.

[0059] FD 40 adds a hysteresis characteristic to determine whether it is in a flicker environment, so that flicker detection is robust to temporary changes in the external environment. In FD unit 40, for each frame, the number of times it is determined to be in a flicker environment in each window is stored in SRAM and counted. Specifically, Fig.10 As shown, the count is incremented during the period determined to be in a flickering environment (i.e., cd_out is 0), and is decremented during the period determined to be not in a flickering environment (i.e., cd_out is 1). Then, by setting a threshold for the number of counts, during the time period when the number of counts exceeds the threshold, it is determined to be in a flickering environment and an fd_out value of 0 is output. For the time period when the number of counts drops below the off threshold, it is determined not to be in a flickering environment, and an fd_out value of 1 is output. Therefore, for the time period when the count exceeds the on threshold, even if cd_out becomes 1 during this period, it can be considered as a temporary change in the external environment. Assuming that the flickering environment is generally continuous, fd_out remains at 0. Fig.10 In the example shown, when cd_out becomes 1 for the second time and the number of counts decreases below the off threshold, it is determined that there is no flicker environment, and fd_out changes from 0 to 1. In this way, robustness can be increased for flicker detection. The desired robustness can be obtained by adjusting the settings of the on threshold and the off threshold.

[0060] Fig.11 is a block diagram of a third embodiment of the present invention. The purpose of the third embodiment is to further increase the rapidity of the EVS according to the second embodiment described above by increasing robustness. Specifically, in the third embodiment, Fig. 9In the block diagram of the EVS according to the second embodiment shown, a block for logic determination is added, in which the logical sum (ie, OR) of fd_out as the output of the FD unit 40 and mix_out as the new output of the IoC unit 30 is taken.

[0061] As mentioned above Figure 5 As described above, when a moving object is captured in a window, an on event and an off event occur at both edges of the object. That is, an on event and an off event occur simultaneously in the same window, and an on count of 1 and an off count of 1 are outputted from the on / off counter unit 20 at the same time. In this case, even if the window is determined to be a flickering window (i.e., in a flickering environment), the determination needs to be immediately changed to a non-flickering window so that the sensor output caused by the moving object is enabled. Therefore, in the third embodiment, mix_out is added as an output of the IoC unit 30.

[0062] Fig.12 The table on the left shows the output of the IoC unit 30 according to the third embodiment. Figure 7 When compared with the above table shown, it can be seen that in addition to cd_out, mix_out is also prepared as the output of the IoC unit 30. Only when the ON count is 1 and the OFF count is 1 at the same time, mix_out is 1, and otherwise it is 0. In other words, a mix_out value of 1 indicates that a moving object is captured in the window.

[0063] In the third embodiment, a logical sum (OR) of mix_out as a new output from the IoC unit 30 and fd_out as an output of the FD unit 40 is taken, and jd_out is output, thereby controlling whether to mask the window. Fig.12 As can be understood from the graph on the right, even in the period determined to be in a flickering environment (fd_out is 0), for the period with mix_out having a value of 1, jd_out becomes 1, and for this period, it is assumed that it is a non-flickering window, enabling the sensor output caused by the moving object. In this way, the robustness can be increased to determine whether the object is in a flickering environment, and then even if a moving object is captured, a quick response can be obtained.

[0064] Fig.13 1 is a block diagram of a fourth embodiment of the present invention. In the fourth embodiment, a frequency calculation block 50 is added to the first embodiment described above, and the flicker frequency can be calculated.

[0065] The frequency calculation block 50 in the fourth embodiment uses the frame rate fps and the onoff_out signal as input to calculate the frequency and outputs the frequency Freq_value. Wherein onoff_out is a signal identifying an on event and an off event. Fig.13 As shown in the graph below, when the frame rate is 1kfps, the frame interval is 1ms, and when 5 consecutive off events occur after 5 consecutive on events, the flicker frequency can be calculated to be 50Hz. Therefore, this not only allows motion detection in a flickering environment, but also allows the frequency of the flicker to be identified.

[0066] Fig.14 FIG. 5 is a block diagram of a fifth embodiment of the present invention. In the fifth embodiment, a frequency calculation block 50 is added to the second embodiment or the third embodiment. Fig.14 In the third embodiment, the frequency calculation block 50 is added.

[0067] As described above, in the second embodiment or the third embodiment, robustness is added to flicker detection, so it is expected that a more robust flicker frequency can be obtained in the fifth embodiment compared to the fourth embodiment.

[0068] Fig.15 FIG. 6 is a block diagram of a sixth embodiment of the present invention. In the sixth embodiment, the flicker frequency calculated in the fourth embodiment or the fifth embodiment is compared with a preset frequency.

[0069] In the sixth embodiment, a preset frequency Hz is added as an input to the frequency calculation block 50. Fig.15 As shown in the lower half of , for example, when the preset frequency is A [Hz] and when there is a window with a calculated frequency of B [Hz], only the flicker of the specified frequency can be removed by taking the negative logical product (NAND) of the preset frequency value and the calculated frequency value for each window, and then masking the windows where the NAND result is 1.

[0070] By using the EVS according to the present invention, a camera system capable of obtaining high image quality without sacrificing the number of pixels can be provided. In addition, a small-sized image rotation correction mechanism can be implemented according to the present invention.

[0071] By using the EVS according to the present invention, a flicker detection method capable of obtaining high image quality without sacrificing the number of pixels can be provided. In addition, a small-sized image rotation correction mechanism can be realized according to the present invention.

[0072] However, the above embodiments provided in this application are not intended to limit the present invention. Any modification, equivalent replacement or improvement made without departing from the spirit and principle of the present invention shall be included in the protection scope of this application.

[0073] Explanation of symbols

[0074] 10: Window counter

[0075] 20: ON / OFF counter

[0076] 30: Conditional Identification

[0077] 40: Flicker detection

[0078] 50: Frequency calculation

Claims

1. An event-based vision sensor (EVS), characterized in that: include: An EVS panel includes a pixel array, wherein each pixel in the pixel array generates an on event or an off event in response to a comparison of an output of a sensor signal with a threshold value, Window counter unit, On / off counter unit, Identification of Condition (IoC) unit, The window counter unit divides the pixel array into a plurality of windows. The on / off counter unit counts the on events and off events occurring for each of the plurality of windows, and the IoC unit determines whether to shield each of the corresponding windows according to the count.

2. The EVS according to claim 1, characterized in that: The IoC unit is used to: In the following cases determining that the corresponding window is a flashing window, and shielding the window, The count is one or more consecutive on events of the on events, or The count is one or more consecutive OFF events among the OFF events, and In the following cases determining that the corresponding window is a non-flickering window, and causing the sensor signal to be output from each pixel in the window, The on events and the off events are mixed in the count.

3. The EVS according to claim 2, characterized in that: The IoC unit is also used to: In the following cases determining that the corresponding window is a flashing window, The one or more consecutive on events among the on events and the one or more consecutive off events among the off events are consecutive in the count.

4. The EVS according to claim 3, characterized in that: Also includes: A flicker detection (FD) unit, wherein The FD unit is used to: counting the on events and the off events output from the on / off counter unit, In the event of the on event, incrementing the count, decrementing the count in case of the OFF event, When the count exceeds a threshold, the corresponding window is determined to be a flashing window, and a result of the determination is output.

5. The EVS according to claim 4, characterized in that: The IoC unit is also used to: In the following cases Output signal The on event and the off event occur simultaneously, in, The EVS is used to: calculating a logical sum (OR) of the signal and the result of the determination output from the FD, According to the logical sum, the corresponding window is determined to be a non-flickering window, and the sensor signal is output from each pixel in the window.

6. The EVS according to any one of claims 2 to 5, characterized in that: Also includes: Frequency calculation block, Wherein, the frequency calculation block is used for: A flicker frequency is calculated based on a signal identifying the on event and the off event output from the IoC unit and a frame rate in operation.

7. The EVS according to claim 6, characterized in that: in, The frequency calculation block is also used to: Compare the calculated flashing frequency with the preset frequency, Wherein, the EVS is used for: Whether to shield each window is determined according to the comparison result.

8. A camera system, characterized in that: Comprising an EVS according to any one of claims 1 to 7.

9. A method for detecting a flickering environment using an event-based vision sensor (EVS), characterized in that: The EVS includes an EVS panel, the EVS panel including a pixel array, wherein each pixel in the pixel array generates an on event or an off event in response to a comparison of an output of a sensor signal with a threshold value, Wherein, the method comprises: dividing the pixel array into a plurality of windows, counting the number of on events and off events occurring for each of the plurality of windows, In the following cases determining that the corresponding window is a flashing window, The count is one or more consecutive on events of the on events, or The count is one or more consecutive OFF events among the OFF events, and, The flickering environment is detected.

10. The method according to claim 9, characterized in that Also includes: In the following cases determining that the corresponding window is a flashing window, the one or more consecutive on events of the on events and the one or more consecutive off events of the off events are consecutive in the count, and, The flickering environment is detected.

Images