An on-chip real-time optical flow detection system based on frame-difference sensor

Abstract

The application discloses a kind of real-time optical flow detection systems on chip based on frame difference type sensor, including frame counter, buffer array, memory array and on-chip processor, frame counter, for generating the time label of event;Buffer array, for temporarily storing the event output of sensor array;Memory array, for temporarily storing the event data of corresponding pixel position in sensor array, flag bit data and the time label of the event;On-chip processor, for completing the calculation and control of detection system;The application has the advantages that: real-time accurately carries out the estimation of optical flow direction, high precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical flow detection and analysis, more particularly to a real-time optical flow detection system on chip based on frame difference sensor. BACKGROUND

[0002] In recent years, with the increasing popularity of embedded systems such as unmanned aerial vehicle navigation, autonomous robots and assisted driving systems, the optical flow detection system applied in these systems is gradually developing towards miniaturization, integration and low power consumption. At present, most of the optical flow detection systems on chip adopt the architecture of front-end sensor + memory + processor. The front-end sensor uses an active pixel image sensor. With the increase of the resolution and frame rate of the image sensor, the amount of data output by the sensor will also increase. At the same time, the optical flow algorithm used by the processor is mostly Lucas and Kanade (L&K) or Horn and Schunck (H&S) algorithm. For example, the vehicle motion state estimation method based on optical flow disclosed in Chinese patent publication CN102999759A adopts L&K algorithm, and the motion target extraction method based on superpixel segmentation and optical flow disclosed in Chinese patent publication CN103871076A adopts H&S algorithm. Both algorithms are gradient-based algorithms with high computational complexity. The huge amount of data and high computational complexity require the entire system to have a processor with high computing power and a large amount of hardware resources, which also increases the power consumption. Compared to the traditional active pixel image sensor which quantifies the absolute amount of light intensity, the frame difference sensor disclosed in the document "Chen S, Tang W, Zhang X, et al. A 64x64 Pixels UWB Wireless Temporal-Difference Digital Image Sensor [J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2012, 20(12): 2232-2240." only quantifies the change direction of light intensity between two adjacent frames, which reduces the amount of data output and power consumption from the source of the sensor. Therefore, using frame difference sensor as the front-end of the optical flow detection system can effectively save the resources of the system and realize the miniaturization, integration and low power consumption of the optical flow detection system.

[0003] However, the current on-chip optical flow detection method based on frame difference sensor has a "aperture problem", that is, when performing local optical flow estimation, only the local normal component of the optical flow can be determined, but the actual optical flow direction is not necessarily the normal component, which leads to inaccurate estimation of the optical flow direction and reduces the accuracy. This problem limits the development and application of on-chip optical flow detection based on frame difference sensors. SUMMARY

[0004] The technical problem to be solved by the present application is that the existing on-chip optical flow detection method based on frame difference sensor has the problem of inaccurate estimation of the optical flow direction, which reduces the accuracy.

[0005] The present application solves the above technical problems by the following technical means: an on-chip real-time optical flow detection system based on frame difference sensor, comprising a frame counter, a buffer array, a memory array and an on-chip processor,

[0006] The frame counter is used to generate the time label of the event;

[0007] The buffer array is used to temporarily store the event output of the sensor array;

[0008] The memory array is used to temporarily store the event data, flag data and time label of the corresponding pixel position in the sensor array;

[0009] The on-chip processor is used to complete the calculation and control of the detection system;

[0010] The event data of the Nth row of the sensor array is transmitted to a row of the buffer array in parallel, then the row is aligned with the position of the Nth row of the corresponding pixels in the memory array, and then the on-chip processor sequentially processes the time tags of the neighboring pixels and calculates the optical flow information according to the time-domain correlation of the time tags, while another row of the buffer array stores the event data of the N-1th row of the pixel array, the row is aligned with the position of the N-1th row of the corresponding pixel array in the memory array, and then the memory array is updated position by position, the N-1th row is updated after the Nth row is calculated, and then the N+1th row is calculated and the Nth row is updated until the real-time calculation and update of the entire array are completed.

[0011] The application uses a frame counter to mark the time tags of the events triggered by the pixels, processes the time tags of the neighboring pixels in a certain order, calculates the size of the optical flow by using the time-domain correlation of the time tags of the pixels at adjacent positions, obtains a relatively accurate estimation of the direction of the optical flow, effectively suppresses the aperture problem, and realizes real-time calculation of the size of the optical flow by cooperating with the update of the memory array, thereby realizing real-time and accurate estimation of the direction of the optical flow with high precision.

[0012] Further, the flag bit is 1, indicating that the event data is not used, and the flag bit is 0, indicating that the event data is used.

[0013] Further, the event output of the sensor array includes ON events, OFF events and no events, the ON event represents that the light intensity at the pixel position changes from dark to bright, the OFF event represents that the light intensity at the pixel position changes from bright to dark, and the no event represents that the light intensity at the pixel position does not change.

[0014] Further, the process of calculating the optical flow information is:

[0015] 1) If the event at the position of the corresponding pixel (m, n) in the buffer array is no event, the optical flow output at the position of the corresponding (m, n) in the memory array is directly set to zero;

[0016] 2) If the event at the position of the corresponding pixel (m, n) in the buffer array is an ON event, unused ON events in all storage units adjacent to the position of the pixel (m, n) in the memory array are sequentially detected, whether the ON events are used is determined according to the flag bit data, and when an unused ON event is first detected at a certain position in the memory array, the size of the optical flow at the position is calculated;

[0017] 3) If the event corresponding to the pixel (m, n) position in the buffer array is OFF, then sequentially detect the unused OFF event in all storage units adjacent to the pixel (m, n) position in the memory array, calculate the optical flow amplitude size of the position, and after each calculation, set the flag bit data in the storage unit used for calculation to 0.

[0018] Further, the method for calculating the optical flow amplitude size is:

[0019] The amplitude size of the optical flow is obtained by the formula

[0020] Where i and j represent the storage unit at the i-th row and j-th column position, (i, j) corresponds to the position where the unused ON event is first detected in the memory array, stp(i, j) represents the time tag of the unit where (i, j) is located, stp_counter represents the time tag in the current frame counter, and MAX is the upper limit value of the counting range of the frame counter.

[0021] Further, the process of data updating of the memory array is:

[0022] After the update at the pixel (a, b) position in the memory array starts, first, it is judged whether the data has been used according to the flag bit in the storage unit corresponding to the position, if it has been used, the overwrite operation is performed to end the update, if the data at the (a, b) position has not been used, it is judged whether the storage time is overdue, if it is overdue, the overwrite operation is performed to complete the update, if it is not overdue, the data at the (a, b) position is retained and the update is ended.

[0023] Further, the method for performing the overwrite operation is:

[0024] The event data at the pixel (a, b) position in the buffer array is overwritten to the original event data, the time tag stp_counter data in the current frame counter is stored, the flag bit data is set to 1, and the update is ended.

[0025] Further, the process of judging whether the storage time is overdue is:

[0026] The time tag stp_counter of the current frame counter is subtracted from the time tag stp_(a, b) at the (a, b) position, if the difference is 0, it is overdue, the overwrite operation is performed to complete the update, if the difference is not 0, it means that it is not overdue, the data at the (a, b) position is retained and the update is ended.

[0027] ​Further, the input of the detection system is the output of the sensor array, the output of the sensor array is sequentially read out in a row-column scanning manner, and the output of the detection system is the optical flow data at each pixel position.

[0028] Further, the sensor in the sensor array is a frame difference sensor, the counting frequency of the frame counter is consistent with the frame frequency of the sensor, the buffer array size is 2xM, M is the number of columns of the sensor, and the size of the memory array is consistent with the size of the sensor array.

[0029] The present application has the advantages that:

[0030] (1) The present application uses a frame counter to mark the time label of the event triggered by the pixel, processes the time labels of the neighboring pixels in a certain order, calculates the size of the optical flow by using the time domain correlation of the time labels of the pixels at adjacent positions, obtains a relatively accurate estimation of the direction of the optical flow, effectively suppresses the aperture problem, and can calculate the size of the optical flow in real time by cooperating with the update of the memory array, so that the estimation of the direction of the optical flow is performed in real time and accurately, and the precision is high.

[0031] (2) The present application uses the data of the frame difference sensor as the input, effectively saves the resources of the system, and is beneficial to realize the miniaturization, integration and low power consumption of the optical flow detection system. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The present application provides a kind of based on frame difference sensor's on-chip real-time optical flow detection system overall architecture schematic diagram for embodiment of the application;

[0033] Figure 2 The processing order of the neighborhood of (m, n) position in the buffer array of the on-chip real-time optical flow detection system based on frame difference sensor provided by the embodiment of the application;

[0034] Figure 3 The update method flow chart of each storage unit of the buffer array in the on-chip real-time optical flow detection system based on frame difference sensor provided by the embodiment of the application;

[0035] Figure 4 The simulation result schematic diagram of the on-chip real-time optical flow detection system based on frame difference sensor provided by the embodiment of the application, wherein, Figure 4 (a) is the running direction schematic diagram of moving object A, Figure 4 (b) is the running direction schematic diagram of moving object B, Figure 4 (c) is the optical flow extraction result schematic diagram of moving object A, Figure 4 (d) is the optical flow extraction result schematic diagram of moving object B. DETAILED DESCRIPTION

[0036] The technical solutions and advantages of the embodiments of the present application will be described clearly and completely below with reference to the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0037] As shown in Figure 1 The present application provides a real-time optical flow detection system based on frame-difference sensor on chip, which has the overall architecture as shown in Figure 1 The input of the detection system is the output of the frame-difference sensor, and the output mode is sequential readout in row-column scanning mode. The output data format of the frame-difference sensor is given in Table 1. Each pixel only outputs 2-bit event data, in which the ON event represents that the light intensity at the pixel position changes from dark to bright, the OFF event represents that the light intensity at the pixel position changes from bright to dark, and no event represents that the light intensity at the pixel position is approximately constant. The output of the detection system is the optical flow data at each pixel position. The frame counter is used to generate the time label of the event, and the counting frequency is consistent with the frame frequency of the frame-difference sensor; the buffer is used to temporarily store the event output of the sensor, and the buffer array size is 2xM, M being the number of columns of the sensor; the memory array is used to temporarily store the 2-bit event data, 1-bit flag data (1 for event data not used; 0 for event data used) and the time label data of the event corresponding to the pixel position in the sensor array, and the size of the memory array is consistent with the size of the sensor array; the on-chip processor is used to complete the calculation and control of the detection system.

[0038] Table 1 2-bit output data of each pixel of the frame-difference sensor

[0039]

[0040] The working process of the present application is as follows:

[0041] The event data of the Nth row of the sensor is first transmitted to a row of the buffer array in parallel, and then the row is aligned with the position of the Nth row of the corresponding pixels in the memory array, and the optical flow information is calculated one by one, while another row of the buffer array stores the event data of the (N-1)th row of the pixel array, and the row is aligned with the position of the (N-1)th row of the corresponding pixel array in the memory array, and the position is updated one by one. While the Nth row is calculated, the (N-1)th row is updated, and then the (N+1)th row is calculated, and the Nth row is updated, until the real-time calculation and update of the entire array are completed. One row of the buffer array is used for calculation, and the other row is used for update, and the two are alternated. The specific process of calculation and update is described below:

[0042] The calculation process of the optical flow information is as follows:

[0043] After the event data of the buffer array is aligned with the memory array, the calculation is started one by one. The following takes the pixel in the mth row and the nth column of the pixel array as an example to illustrate:

[0044] 1) If there is no event in the event data of the corresponding pixel (m, n) position of the buffer, the optical flow output in the corresponding (m, n) position of the memory array is directly set to zero;

[0045] 2) If the event in the corresponding pixel (m, n) position of the buffer is ON, then the unused ON events in the surrounding eight storage units are detected in the order of Figure 2 According to the flag bit data, it is judged whether it is used or not. When the unused ON event is first detected in the (i, j) position of the memory array, the time tag stp(i, j) of the unit is subtracted from the time tag data stp_counter in the current frame counter, and then the reciprocal is taken to obtain the amplitude size of the optical flow and record the direction Figure 2 of the 8 positions corresponding to 8 directions.

[0046] For the calculation of stp(i, j)-stp_counter is positive, it can be directly calculated, and if stp(i, j)-stp_counter is negative, it needs to be added with MAX and then calculated by taking the reciprocal.

[0047]

[0048] Where i and j represent the storage unit at the i-th row and j-th column position, (i, j) corresponds to the position of the first detected unused ON event in the 1-8 position of Figure 2 , and MAX is the upper limit value of the frame counter counting range.

[0049] 3) If the event at the pixel (m, n) position in the buffer is OFF, then Figure 2 The unused OFF events in the surrounding eight storage units are sequentially detected, and the optical flow information at the position is calculated. The calculation method is the same as that for the ON event. After each calculation, the flag bit data in the storage unit used for calculation is set to 0.

[0050] The update process of the memory array for data update:

[0051] After the event data of the buffer array is aligned with the memory array, the update is started one by one. The update process is shown in Figure 3 After the update at the pixel (a, b) position in the memory array is started, it is first determined whether the data has been used according to the flag bit in the storage unit at the position. If the data has been used, the overwrite operation is performed. The process of the overwrite operation is as follows: the event data at the pixel (a, b) position in the buffer is overwritten on the original event data, the time tag data stp_counter in the current frame counter is stored, the flag bit data is set to 1, and the update is ended.

[0052] If the data at the (a, b) position has not been used, it is further determined whether the storage time is overdue. The determination method is that the time tag stp_counter in the current frame counter is subtracted from the time tag stp_(a,b) at the (a, b) position. If the difference is 0, it is overdue, and the overwrite operation is performed to complete the update. If the difference is not 0, it means that it is not overdue, the data at the (a, b) position in the memory array is retained, and the update is ended. The addition of the determination of whether it is overdue can avoid that the event is stored for too long time and interferes with the calculation of new data.

[0053] In this embodiment, the array size of the frame difference type sensor as the data input is 128x128, the frame frequency of the sensor is 310 fps, the row processing period of the sensor is 25 μs, and therefore the optical flow processing time of each row in the memory array is also 25 μs. The frequency of the frame counter is 310 Hz, and the counting range is set to 0-255, so that 8-bit data representing the optical flow size is obtained. Finally, the optical flow data format at each pixel position is 11 bits: 3 bits of direction information + 8 bits of size information. The memory array has the same size as the sensor array, and the storage unit can use a static random-access memory (SRAM). In actual scene testing, a single moving target with obvious brightness difference from the background is selected to have better optical flow extraction effect. As shown in Figure 4 The behavior level simulation is performed on the proposed detection system, and the optical flow information of two moving objects A and B with different shapes is extracted, respectively. It can be seen that the optical flow extraction result does not show the aperture problem.

[0054] Through the technical scheme, the on-chip optical flow detection system designed by the application uses a frame counter to mark a time tag for an event triggered by a pixel, uses time domain correlation of time tags of pixels at adjacent positions to calculate a size of the optical flow, and processes the time tags of the adjacent pixels in a certain order to obtain a relatively accurate estimation of a direction of the optical flow, thereby effectively suppressing an aperture problem. In combination with an updating scheme of the storage unit, the optical flow can be calculated in real time. In addition, the application uses data of a frame difference type sensor as input, thereby effectively saving resources of the system and being conducive to miniaturization, integration and low power consumption of the optical flow detection system.

[0055] The above examples are only used to illustrate the technical solutions of the application, rather than limit the same; although the application is described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced equivalently; and the modification or replacement does not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. An on-chip real-time optical flow detection system based on a frame difference sensor, characterized in that, Includes a frame counter, buffer array, memory array, and on-chip processor. The frame counter is used to generate time stamps for events; The buffer array is used to temporarily store the event outputs of the sensor array; the event outputs of the sensor array include ON events, OFF events, and no events. An ON event represents that the light intensity at the pixel location changes from dark to bright, an OFF event represents that the light intensity at the pixel location changes from bright to dark, and no event represents that the light intensity at the pixel location remains unchanged. The memory array is used to temporarily store event data, flag data, and the time tag of the event at the corresponding pixel position in the sensor array; The on-chip processor is used to perform calculations and control of the detection system; The event data of the Nth row of pixels in the sensor array is transmitted in parallel to one row of the buffer array. After this row is aligned with the corresponding Nth row of pixels in the memory array, the on-chip processor processes the time tags of neighboring pixels sequentially and calculates optical flow information one by one based on the temporal correlation of the time tags. Simultaneously, another row of the buffer stores the event data of the (N-1)th row of the pixel array. After this row is aligned with the corresponding (N-1)th row of the pixel array in the memory array, the memory array is updated position by position. The update of the (N-1)th row is completed as the calculation of the Nth row is completed, and then the calculation of the (N+1)th row is performed, followed by the update of the Nth row, until the real-time calculation and update of the entire array is completed. The process of calculating the optical flow information is as follows: 1) If there is no event at the corresponding pixel position (m, n) in the buffer array, then directly set the optical flow output at the corresponding position (m, n) in the memory array to zero; 2) If the event at the corresponding pixel (m, n) position in the buffer array is an ON event, then sequentially detect all unused ON events in all storage cells adjacent to the pixel (m, n) position in the memory array, and determine whether they have been used based on the flag bit data. When an unused ON event is detected for the first time at a certain position in the memory array, calculate the optical flow amplitude at that position. 3) If the event at the corresponding pixel (m, n) position in the buffer array is OFF, then sequentially detect the unused OFF events in all storage cells adjacent to the pixel (m, n) position in the memory array, calculate the optical flow amplitude at that position, and set the flag bit data in the storage cell used for the calculation to 0 after each calculation.

2. The on-chip real-time optical flow detection system based on a frame difference sensor according to claim 1, characterized in that, A flag of 1 indicates that the event data has not been used, while a flag of 0 indicates that the event data has been used.

3. The on-chip real-time optical flow detection system based on a frame difference sensor according to claim 1, characterized in that, The method for calculating the magnitude of the optical flow is as follows: Through formula Obtain the magnitude of the optical flow; in, and They represent the first Line 1 Storage unit at column position, This corresponds to the location where an unused ON event is first detected in the memory array. express The time stamp of the unit. This represents the time stamp in the current frame counter. MAX This is the upper limit of the frame counter's counting range.

4. The on-chip real-time optical flow detection system based on a frame difference sensor according to claim 1, characterized in that, The process of updating data in the memory array is as follows: After the update at the corresponding pixel (a, b) position in the memory array begins, it first determines whether the data has been used based on the flag bit in the storage cell corresponding to that position. If it has been used, an overwrite operation is performed to end the update. If the data at position (a, b) has not been used, it checks whether the storage time has expired. If it has expired, an overwrite operation is performed to complete the update; if it has not expired, the data at position (a, b) is retained and the update ends.

5. The on-chip real-time optical flow detection system based on a frame difference sensor according to claim 4, characterized in that, The method for performing the overwrite operation is as follows: Overwrite the existing event data with the event data at the corresponding pixel (a, b) position in the buffer array, and add the time tag from the current frame counter. The data is stored, and the flag is set to 1 to end the update.

6. The on-chip real-time optical flow detection system based on a frame difference sensor according to claim 4, characterized in that, The process for determining whether the storage time has expired is as follows: Time tag of the current frame counter Time label at position (a, b) Subtract the values; if the difference is 0, it indicates a timeout, and an overwrite operation is performed to complete the update; if the difference is not 0, it means there was no timeout, the data at position (a, b) is retained, and the update ends.

7. The on-chip real-time optical flow detection system based on a frame difference sensor according to claim 1, characterized in that, The input to the detection system is the output of the sensor array, which is read out sequentially in a row and column scanning manner. The output of the detection system is the optical flow data at each pixel position.

8. The on-chip real-time optical flow detection system based on a frame difference sensor according to claim 1, characterized in that, The sensors in the sensor array are frame difference type sensors. The counting frequency of the frame counter is consistent with the frame frequency of the sensor. The size of the buffer array is 2×M, where M is the number of columns of the sensor. The size of the memory array is consistent with the size of the sensor array.

Images