An Event Sensor Optical Flow Estimation Method and Device Based on Plane Fitting

Abstract

The present invention discloses an event sensor optical flow estimation method and device based on plane fitting. The method includes: performing pre-filtering processing on the event stream output by the event sensor; using the Prim greedy algorithm to select the optimal local neighboring event set and extract valid events; according to the optimal local neighboring event set, selecting the optimal M events to build an initial plane fitting model; substituting other events in the optimal local neighboring event set into the initial model, and iteratively selecting the optimal plane fitting model through the RANSAC algorithm to achieve optical flow estimation. The advantages of the present invention are: reducing the error influence of abnormal data points in the event stream on the extraction of valid events and the establishment of the plane fitting model, and improving the accuracy of optical flow estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and more particularly to an event sensor optical flow estimation method and device based on plane fitting. Background Art

[0002] Optical flow estimation is one of the important research directions in computer vision. Accurate and fast optical flow estimation is an inevitable requirement for visual tasks. Event sensors, due to their characteristics of low latency, low power consumption, high dynamic range, and high time resolution, effectively solve the motion blur problem existing in traditional image sensors in high-speed motion and high-intensity contrast scenarios. The optical flow estimation algorithm for event sensors can give full play to the advantages of event sensors and achieve visual perception in high-speed and high-dynamic scenarios.

[0003] Currently, the optical flow estimation methods for event sensors at home and abroad are mainly divided into three types: plane fitting algorithm, contrast maximization framework algorithm, and neural network architecture-based method. Chinese Patent Publication No. CN112529944A discloses an end-to-end unsupervised optical flow estimation method based on an event camera, which uses the neural network architecture-based method for event sensor optical flow estimation. It needs to train and test a large amount of data sets, with a high calculation cost and a long time consumption. The method based on the contrast framework also relies on the previously accumulated events to indirectly achieve optical flow estimation and lacks advantages in the real-time optical flow estimation of event sensors. The plane fitting optical flow estimation algorithm can directly process the data generated by event sensors, with lower calculation costs and faster response times. However, the event stream often contains abnormal data points, and these invalid events are not conducive to the establishment of subsequent plane fitting models, affecting the accuracy of event sensor optical flow estimation. Summary of the Invention

[0004] The technical problem to be solved by the present invention is how to reduce the error influence of abnormal data points in the event stream on the extraction of valid events and the establishment of the plane fitting model, and improve the accuracy of optical flow estimation.

[0005] The present invention solves the above technical problems through the following technical means: An event sensor optical flow estimation method based on plane fitting, the method comprising:

[0006] Step 1: Perform pre-filtering processing on the event stream output by the event sensor;

[0007] Step 2: Use the Prim greedy algorithm to select the optimal local neighboring event set and extract valid events;

[0008] Step 3: According to the optimal local neighboring event set, select the optimal M events to build an initial plane fitting model;

[0009] Step 4: Substitute other events in the optimal local neighboring event set into the initial model, and iteratively select the optimal plane fitting model through the RANSAC algorithm to achieve optical flow estimation.

[0010] The present invention performs pre-filtering processing on the event stream output by the event sensor, extracts valid events using the Prim greedy algorithm, reduces the error influence of abnormal data points in the event stream on the extraction of valid events and the establishment of the plane fitting model, selects the optimal local neighboring event set, improves the data stream quality, and at the same time, uses the event quality ranking of the optimal local neighboring event set to select the optimal M events to build the initial plane fitting model, performs plane fitting using a better initial inlier set, and combines with the RANSAC algorithm to obtain the optimal local plane fitting optical flow estimation algorithm model, realizing high-precision optical flow estimation of the event sensor.

[0011] Further, the Step 1 includes:

[0012] If there are two incoming events e i and e j , and their pixel address expressions are (x i , y i , t i ), (x j , y j , t j ) respectively, when their pixel expression relationships are x i =x j , y i =y j , t j >t i , t j <t i +T rf , then the event e j is a synchronous event caused by the wide-edge effect, and this event is filtered, where T rf is a threshold, representing the minimum time difference between two events occurring at the same position.

[0013] Furthermore, the rule for using the Prim greedy algorithm to select the optimal local neighboring event set N n in the Step 2 is:

[0014] (1) The event N n in N * has the same polarity as the current trigger event E n ;

[0015] (2) The event N n in N * is located within the L×L neighborhood of the current trigger event E n ;

[0016] (3)N n The middle event does not include the current triggering event E n ;

[0017] (4)N n The middle event is the current triggering event E n or the event with the smallest spatial position difference among other events in the event set within the L×L spatial region. If the spatial distance differences are the same, select the event with the smallest n timestamp difference from the current triggering event E;

[0018] (5) Select the total number of events n≥L.

[0019] Furthermore, the third step includes:

[0020] After selecting the optimal local neighboring event set, use the Prim greedy algorithm to sort the superior and inferior events in the event set, and select the optimal M events in the event set as the initial inlier set;

[0021] Assume the expression of the initial plane fitting model is ax + by + ct = d. Under the constraint condition of a 2 +b 2 +c 2 =1, minimize the spatial distance d of each event in the initial inlier set to the fitting plane, and use the Lagrange multiplier method for estimation. Its expression is as follows: i d

[0022] d i =|ax + by + ct - d| (1)

[0023] where a, b, c, and d are all fitting parameters to be solved. Since the value of d has nothing to do with the optical flow estimation value, assume the value of d is 0. At this time, the expression of the Lagrangian function corresponding to formula (1) is as follows:

[0024]

[0025] where λ is a constant to be solved;

[0026] Take the partial derivative of formula (2) and set it to 0. Its expression is as follows:

[0027]

[0028] At this time, the eigenvector corresponding to the smallest eigenvalue is the fitting parameter a, b, c to be obtained. According to the fitting parameters, the initial plane fitting model can be obtained.

[0029] Furthermore, the fourth step includes:

[0030] (1) Substitute the remaining events in the optimal local neighboring event set into the initial model, and calculate the fitting error at this time. If the fitting error value is less than the set threshold, add it to the inlier set S0;

[0031] (2) If the number of events contained in the current inlier set S0 is greater than the current optimal inlier set S pre , then update S pre = S0;

[0032] (3) Repeat the above steps iteratively. The model obtained from the S pre with the most inliers is the optimal plane fitting model;

[0033] According to the formula

[0034]

[0035] Use the optimal plane fitting model to achieve event sensor optical flow estimation, where g is the spatial gradient of the fitting plane, a1, b1, and c1 represent the fitting parameters in the optimal plane fitting model, and v x represents the horizontal direction optical flow estimation value, and v y represents the vertical direction optical flow estimation value.

[0036] The present invention also provides an event sensor optical flow estimation device based on plane fitting. The device includes:

[0037] A preprocessing module for pre-filtering the event stream output by the event sensor;

[0038] An event set selection module for using the Prim greedy algorithm to select the optimal local neighboring event set and extract valid events;

[0039] A model construction module for selecting the optimal M events according to the optimal local neighboring event set to build an initial plane fitting model;

[0040] An optical flow estimation module for substituting other events in the optimal local neighboring event set into the initial model, and iteratively selecting the optimal plane fitting model through the RANSAC algorithm to achieve optical flow estimation.

[0041] Further, the preprocessing module is further configured to:

[0042] If there are two incoming events e i and e j , and their pixel address expressions are (x i , y i , t i ), (x j , y j , t j ) respectively, when their pixel expression relationship is x i = xj , y i = y j , t j > t i , t j < t i + T rf , then event e j is a synchronization event caused by the wide-edge effect. Filter this event, where T rf is the threshold, representing the minimum time difference between two events occurring at the same position.

[0043] Furthermore, the rule for selecting the optimal local neighboring event set N n in the event set selection module by using the Prim greedy algorithm is as follows:

[0044] (1) The event N n in N * has the same polarity as the current trigger event E n ;

[0045] (2) The event N n in N * is located within the L×L neighborhood of the current trigger event E n ;

[0046] (3) The events in N n do not include the current trigger event E n ;

[0047] (4) The events in N n are the events with the smallest spatial position difference of the current trigger event E n or other events in the event set within the L×L spatial region. If the spatial distance difference is the same, select the event with the smallest timestamp difference from the current trigger event E n ;

[0048] (5) The total number of selected events n ≥ L.

[0049] Furthermore, the model construction module is also used for:

[0050] After selecting the optimal local neighboring event set, use the Prim greedy algorithm to sort the superior and inferior events in the event set, and select the optimal M events in the event set as the initial inlier set;

[0051] Assume the initial model expression of plane fitting is ax + by + ct = d. Under the constraint condition of a 2 + b 2 + c 2 = 1, make the spatial distance d iMinimize and estimate using the Lagrange multiplier method, and its expression is as follows:

[0052] d i = |ax + by + ct - d| (1)

[0053] Among them, a, b, c, and d are all fitting parameters to be determined. Since the value of d has nothing to do with the optical flow estimation value, it is assumed that the value of d is 0. At this time, the expression of the Lagrangian function corresponding to formula (1) is as follows:

[0054]

[0055] Among them λ is a constant to be determined;

[0056] Take the partial derivative of formula (2) and make it equal to 0, and its expression is as follows:

[0057]

[0058] At this time, the eigenvector corresponding to the minimum eigenvalue is the fitting parameter a, b, c to be obtained. According to the fitting parameter, the initial plane fitting model can be obtained.

[0059] Furthermore, the optical flow estimation module is also used for:

[0060] (1) Substitute the remaining events in the optimal local neighboring event set into the initial model, and calculate the fitting error at this time. If the fitting error value is less than the set threshold, add it to the inlier set S0;

[0061] (2) If the number of events contained in the current inlier set S0 is greater than the current optimal inlier set S pre , then update S pre = S0;

[0062] (3) Repeat the above steps iteratively. The model obtained by S pre with the largest number of inliers is the optimal plane fitting model;

[0063] According to the formula

[0064]

[0065] Use the optimal plane fitting model to realize the optical flow estimation of the event sensor. Among them, g is the spatial gradient of the fitting plane, a1, b1, and c1 represent the fitting parameters in the optimal plane fitting model, and v x represents the horizontal direction optical flow estimation value, and v y represents the vertical direction optical flow estimation value.

[0066] The advantages of the present invention are as follows: The present invention performs pre-filtering processing on the event stream output by the event sensor, extracts valid events using the Prim greedy algorithm, reduces the error impact of abnormal data points in the event stream on the extraction of valid events and the establishment of the plane fitting model, selects the optimal local neighboring event set, improves the data stream quality, and at the same time uses the event quality ranking of the optimal local neighboring event set to select the optimal M events to build the initial plane fitting model, performs plane fitting using a better initial inlier set, and combines with the RANSAC algorithm to obtain the optimal local plane fitting optical flow estimation algorithm model, realizing high-precision optical flow estimation of the event sensor. Description of the Drawings

[0067] Figure 1 It is a flowchart of a method for optical flow estimation of an event sensor based on plane fitting provided in Embodiment 1 of the present invention;

[0068] Figure 2 It is a comparison of the optical flow estimation images of the event sensor rotation motion event set and the results of the prior art method in the method for optical flow estimation of an event sensor based on plane fitting provided in Embodiment 1 of the present invention; Figure 2 (a) is the optical flow estimation image using the traditional plane fitting algorithm; Figure 2 (b) is the optical flow estimation image using the edge denoising plane fitting algorithm; Figure 2 (c) is the optical flow estimation image using the small area plane fitting algorithm; Figure 2 (d) is the optical flow estimation image using the algorithm of the present invention;

[0069] Figure 3 It is a comparison of the optical flow estimation images of the event sensor horizontal motion event set and the results of the prior art method in the method for optical flow estimation of an event sensor based on plane fitting provided in Embodiment 1 of the present invention; Figure 3 (a) is the optical flow estimation image using the traditional plane fitting algorithm; Figure 3 (b) is the optical flow estimation image using the edge denoising plane fitting algorithm; Figure 3 (c) is the optical flow estimation image using the small area plane fitting algorithm; Figure 3 (d) is the optical flow estimation image using the algorithm of the present invention. Detailed Embodiments

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0071] Example 1

[0072] As Figure 1 shown, an event sensor optical flow estimation method based on plane fitting, the method comprising:

[0073] S1: Perform pre-filtering processing on the event stream output by the event sensor.

[0074] Define a threshold T rf = 500 μs, this threshold represents the minimum time difference between two events occurring at the same position. If there are two incoming events e i and e j , whose pixel address expressions are respectively (x i , y i , t i ), (x j , y j , t j ), when their pixel expression relationships are x i = x j , y i = y j , t j > t i , t j < t i + T rf , then the event e j is considered a synchronous event caused by the wide-edge effect, and at this time, this event is filtered.

[0075] S2: Use the Prim greedy algorithm to select the optimal local neighboring event set and extract valid events. The specific rule for selecting the optimal local neighboring event set N n is as follows:

[0076] (1) The event N n in N * has the same polarity as the current trigger event E n .

[0077] (2) The event N n in N * is located within the 5×5 neighborhood of the current trigger event E n .

[0078] (3) To exclude the accidental error characteristics of the current trigger event E n , the event in N n does not include the current trigger event E n .

[0079] (4) Assume that within a small area, its edge is nearly constant-speed motion. At this time, the edge and natural structure that trigger the event will eventually be close to the current trigger event E nOccurs at adjacent positions and time instants. Through the above judgment, the current triggering event E is selected n As the initial event, using the idea of the Prim greedy algorithm, the optimal local adjacent event set N n The events in should be the current triggering event E n Or the event with the smallest spatial position difference among other events in the event set within the 5×5 spatial region. If the spatial distance differences are the same, select the event with the smallest timestamp difference from the current triggering event E n Time stamp difference minimum event.

[0080] (5) To avoid the events in N n Showing a collinear state and unable to achieve subsequent plane fitting, select the total number of events n≥5.

[0081] S3: Establishment of the initial plane fitting model

[0082] The traditional plane fitting optical flow estimation algorithm uses the least squares method to establish the initial model. However, the least squares method only considers the error in a single direction. After selecting the optimal local adjacent event set in the present invention, select the optimal 5 events in the event set as the initial inlier set, and use the eigenvalue algorithm to establish the initial plane fitting model.

[0083] Assume that the expression of the initial plane fitting model is ax + by + ct = d. Under the constraint condition of a 2 +b 2 +c 2 = 1, make the spatial distance d of each event in the initial inlier set to the fitting plane i Minimize, and use the Lagrange multiplier method for estimation. Its expression is as follows:

[0084] d i = |ax + by + ct - d| (1)

[0085] Among them, a, b, c, and d are all fitting parameters to be determined. Since the magnitude of d has nothing to do with the optical flow estimation value, assume that the value of d is 0. At this time, the expression of the Lagrange function corresponding to formula (1) is as follows:

[0086]

[0087] Among them λ is a constant to be determined;

[0088] Take the partial derivative of formula (2) and make it 0. Its expression is as follows:

[0089]

[0090] Let According to the eigenvalue calculation formula, |λE - A| = 0. At this time, substituting matrix A into it can obtain the minimum eigenvalue of formula (3), where E is the identity matrix. At this time, the eigenvector corresponding to the minimum eigenvalue is the fitting parameter a, b, c obtained, and the initial plane fitting model can be obtained according to the fitting parameter.

[0091] Formula (3) is to select several events in the event set to obtain the fitting plane. At this time, the obtained fitting plane cannot replace all the events in the event set due to the uncertainty of the initially selected events. At this time, there will be a certain error in directly using this result to calculate the optical flow using formula (4). Therefore, it is necessary to design the iterative work in step S4 to select the optimal model for plane fitting. The traditional algorithm randomly selects several events for initial plane fitting, which will have greater uncertainty and affect the subsequent iterative process. Therefore, the present invention adopts the Prim greedy algorithm to select the optimal M events for initial plane fitting, maximizing the accuracy of the initial plane fitting and improving the accuracy of the subsequent iteration to achieve high-precision optical flow estimation.

[0092] S4: Substitute other events in the optimal local neighboring event set into the initial model, and iteratively select the optimal plane fitting model through the RANSAC algorithm to achieve optical flow estimation. The specific process is as follows:

[0093] Substitute the remaining events in the optimal local neighboring event set into the initial model, and calculate the fitting error at this time. If the fitting error value is less than the set threshold, add it to the inlier set S0; if the number of events contained in the current inlier set S0 is greater than the current optimal inlier set S pre , then update S pre = S0; repeat the above steps iteratively. The model with the largest number of inliers S pre obtained is the optimal plane fitting model.

[0094] Regarding the properties of the plane fitting optical flow estimation algorithm, the spatial gradient g of the fitting plane encodes the optical flow information related to the active events, and its expression is as follows:

[0095]

[0096] According to formula (4), use the optimal plane fitting model to achieve event sensor optical flow estimation, where a1, b1, and c1 represent the fitting parameters in the optimal plane fitting model, and v x represents the horizontal direction optical flow estimation value, and v y represents the vertical direction optical flow estimation value.

[0097] The following uses specific simulation examples to make a detailed description of the effect of the optical flow estimation method provided by the present invention.

[0098] Figure 2 is the optical flow estimation result of the rotating motion image, whereFigure 2 (a) is the optical flow estimation image using the traditional plane fitting algorithm; Figure 2 (b) is the optical flow estimation image using the edge-denoising plane fitting algorithm; Figure 2 (c) is the optical flow estimation image using the small-region plane fitting algorithm; Figure 2 (d) is the optical flow estimation image using the algorithm of the present invention. The small-region local plane fitting algorithm and the algorithm of the present invention both have more accurate performances in optical flow estimation. At the same time, compared with the small-region local plane fitting algorithm, the algorithm of the present invention optimizes the extraction of effective events and the establishment of the plane fitting model through the Prim greedy algorithm, making the plane fitting result clearer and more accurate.

[0099] Figure 3 is the optical flow estimation result of the horizontal motion image, where Figure 3 (a) is the optical flow estimation difference image using the traditional plane fitting algorithm; Figure 3 (b) is the optical flow estimation difference image using the edge-denoising plane fitting algorithm; Figure 3 (c) is the optical flow estimation difference image using the small-region plane fitting algorithm; Figure 3 (d) is the optical flow estimation difference image using the algorithm of the present invention. It can be seen from Figure 3 that the edge-denoising plane fitting algorithm has a certain improvement in the accuracy of optical flow estimation compared with the traditional optical flow estimation algorithm by filtering the event stream, but the accuracy of its estimated value is still not ideal; the small-region plane fitting algorithm improves the quality of the data stream by extracting effective events in small regions, but its restricted region also limits the goodness of fitting; the algorithm of the present invention optimizes the extraction of effective events through the Prim greedy algorithm idea and optimizes the establishment of the plane fitting model through the eigenvalue algorithm. Compared with other algorithms, the algorithm of the present invention has higher accuracy and better optical flow estimation effect.

[0100] Through the above technical solutions, the present invention pre-filters the event stream output by the event sensor, extracts effective events using the Prim greedy algorithm, reduces the error influence of abnormal data points in the event stream on the extraction of effective events and the establishment of the plane fitting model, selects the optimal local neighboring event set, improves the quality of the data stream, and at the same time uses the event quality ranking of the optimal local neighboring event set to select the optimal M events to build the initial plane fitting model, performs plane fitting using a better initial inlier set, and combines with the RANSAC algorithm to obtain the optimal local plane fitting optical flow estimation algorithm model, realizing high-precision optical flow estimation of the event sensor.

[0101] Embodiment 2

[0102] Based on Embodiment 1, Embodiment 2 of the present invention further provides an optical flow estimation device for an event sensor based on plane fitting, and the device includes:

[0103] A preprocessing module for pre-filtering the event stream output by the event sensor;

[0104] An event set selection module for selecting an optimal local neighboring event set using the Prim greedy algorithm and extracting valid events;

[0105] A model construction module for selecting the optimal M events according to the optimal local neighboring event set to build an initial plane fitting model;

[0106] An optical flow estimation module for substituting other events in the optimal local neighboring event set into the initial model and iteratively selecting the optimal plane fitting model through the RANSAC algorithm to achieve optical flow estimation.

[0107] Specifically, the preprocessing module is further configured to:

[0108] If there are two incoming events e i and e j , and their pixel address expressions are (x i , y i , t i ), (x j , y j , t j ) respectively. When their pixel expression relationship is x i = x j , y i = y j , t j > t i , t j < t i + T rf , then the event e j is a synchronous event caused by the wide-edge effect, and this event is filtered. Among them, T rf is a threshold, indicating the minimum time difference between two events occurring at the same position.

[0109] More specifically, the rule for the event set selection module to select the optimal local neighboring event set N n using the Prim greedy algorithm is as follows:

[0110] (1) The event N n in N * has the same polarity as the current trigger event E n ;

[0111] (2) The event N n in N * is located within the L×L neighborhood of the current trigger event E n ;

[0112] (3) N nThe middle event does not contain the currently triggered event E n ;

[0113] (4)N n The middle event is the currently triggered event E n Or the event with the smallest spatial position difference among other events in the event set within the L×L spatial region. If the spatial distance differences are the same, select the event with the smallest timestamp difference from the currently triggered event E n ;

[0114] (5) Select the total number of events n≥L

[0115] More specifically, the model construction module is further configured to:

[0116] After selecting the optimal local neighboring event set, use the Prim greedy algorithm to sort the superior and inferior events in the event set, and select the optimal M events in the event set as the initial inlier set;

[0117] Assume that the initial model expression of plane fitting is ax + by + ct = d. Under the constraint condition of a 2 +b 2 +c 2 = 1, minimize the spatial distance d from each event in the initial inlier set to the fitting plane, and use the Lagrange multiplier method for estimation. Its expression is as follows: i d

[0118] d i = |ax + by + ct - d| (1)

[0119] where a, b, c, and d are all fitting parameters to be solved. Since the magnitude of d has nothing to do with the optical flow estimation value, assume that the value of d is 0. At this time, the expression of the Lagrangian function corresponding to formula (1) is as follows:

[0120]

[0121] where λ is a constant to be solved;

[0122] Take the partial derivative of formula (2) and set it to 0. Its expression is as follows:

[0123]

[0124] At this time, the eigenvector corresponding to the smallest eigenvalue is the fitting parameter a, b, c to be obtained. According to the fitting parameters, the initial model of plane fitting can be obtained.

[0125] More specifically, the optical flow estimation module is further configured to:

[0126] (1) Substitute the remaining events in the optimal local neighboring event set into the initial model, and calculate the fitting error at this time. If the fitting error value is less than the set threshold, add it to the inlier set S0;

[0127] (2) If the number of events contained in the current inlier set S0 is greater than the current optimal inlier set S pre , then update S pre = S0;

[0128] (3) Repeat the above steps iteratively. The model obtained from the S pre with the largest number of inliers is the optimal plane fitting model;

[0129] According to the formula

[0130]

[0131] Use the optimal plane fitting model to implement event sensor optical flow estimation. Among them, g is the spatial gradient of the fitting plane, a1, b1, and c1 represent the fitting parameters in the optimal plane fitting model, and v x represents the horizontal direction optical flow estimation value, and v y represents the vertical direction optical flow estimation value.

[0132] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An event sensor optical flow estimation method based on plane fitting, characterized in that, The method includes the following steps: Step 1: Perform pre-filtering processing on the event stream output by the event sensor; Step 2: Use the Prim greedy algorithm to select the optimal local neighboring event set and extract valid events; Step 3: According to the optimal local neighboring event set, select the optimal M events to build an initial plane fitting model; Step 4: Substitute the other events in the optimal local neighboring event set into the initial model, and iteratively select the optimal plane fitting model through the RANSAC algorithm to achieve optical flow estimation.

2. The method for estimating the optical flow of an event sensor based on plane fitting according to claim 1, wherein, The said Step 1 includes: If there are two incoming events e i and e j , whose pixel address expressions are (x i , y i , t i ), (x j , y j , t j ) respectively. When their pixel expression relationships are x i = x j , y i = y j , t j > t i , t j < t i + T rf , then the event e j is a synchronous event caused by the wide-edge effect, and this event is filtered. Among them, T rf is the threshold, indicating the minimum time difference between two events occurring at the same position.

3. The method for estimating the optical flow of an event sensor based on plane fitting according to claim 2, wherein, In the second step, the Prim greedy algorithm is used to select the optimal local neighboring event set N n The rule is as follows: (1)N n Medium event N * has the same polarity as the currently triggered event E n ; (2)N n Medium event N * Is located within the L×L neighborhood of the currently triggered event E n ; (3)N n The middle event does not include the currently triggered event E n ; (4)N n The medium event is the currently triggered event E n Or the event with the smallest spatial position difference among other events in the event set within the L×L spatial region. If the spatial distance differences are the same, select the event with the smallest timestamp difference from the currently triggered event E n ; (5) Select the total number of events n≥L.

4. The method for event sensor optical flow estimation based on plane fitting according to claim 3, wherein The said Step 3 includes: After selecting the optimal local neighboring event set, use the sorting of good and bad events in the event set under the Prim greedy algorithm to select the optimal M events in the event set as the initial inlier set; Assume that the initial model expression for plane fitting is ax + by + ct = d. Under the constraint condition of a 2 + b 2 + c 2 = 1, minimize the spatial distance d from each event in the initial inlier set to the fitting plane, and use the Lagrange multiplier method for estimation. The expression is as follows: i ​ d i = |ax + by + ct - d| (1) Wherein, a, b, c, and d are all fitting parameters to be solved. Since the value of d has nothing to do with the optical flow estimation value, it is assumed that the value of d is 0. At this time, the expression of the Lagrangian function corresponding to formula (1) is as follows: wherein λ is a constant to be determined; Take the partial derivative of formula (2) and make it equal to 0. Its expression is as follows: At this time, the eigenvector corresponding to the minimum eigenvalue is the fitting parameter a, b, c obtained, and the initial plane fitting model can be obtained according to the fitting parameters.

5. A method for estimating the optical flow of an event sensor based on plane fitting according to claim 4, characterized in that, The said Step 4 includes: (1) Substitute the remaining events in the optimal local neighboring event set into the initial model, and calculate the fitting error at this time. If the fitting error value is less than the set threshold, add it to the inlier set S0; (2) If the number of events contained in the current inlier set S0 is greater than the current optimal inlier set S pre , then update S pre = S0; (3) Repeat the above steps iteratively, and the S with the largest number of interior points pre The obtained model is the optimal plane fitting model; According to the formula Optical flow estimation of event sensors is achieved using an optimal plane fitting model, where g is the spatial gradient of the fitting plane, a1, b1, and c1 represent the fitting parameters in the optimal plane fitting model, and v x represents the estimated value of the optical flow in the horizontal direction, and v y represents the estimated value of the optical flow in the vertical direction.

6. An event sensor optical flow estimation device based on plane fitting, characterized in that The device includes: A preprocessing module for performing pre-filtering processing on the event stream output by the event sensor; An event set selection module for using the Prim greedy algorithm to select the optimal local neighboring event set and extract valid events; A model construction module for selecting the optimal M events according to the optimal local neighboring event set to build an initial plane fitting model; An optical flow estimation module for substituting the other events in the optimal local neighboring event set into the initial model, and iteratively selecting the optimal plane fitting model through the RANSAC algorithm to achieve optical flow estimation.

7. An event sensor optical flow estimation device based on plane fitting according to claim 6, characterized in that, The said preprocessing module is also used for: If there are two incoming events e i and e j , whose pixel address expressions are (x i , y i , t i ), (x j , y j , t j ) respectively. When their pixel expression relationships are x i = x j , y i = y j , t j > t i , t j < t i + T rf , then the event e j is a synchronous event caused by the wide-edge effect, and this event is filtered. Among them, T rf is the threshold, indicating the minimum time difference between two events occurring at the same position.

8. An event sensor optical flow estimation device based on plane fitting according to claim 7, characterized in that, In the event set selection module, the Prim greedy algorithm is used to select the optimal local neighboring event set N n The rule is as follows: (1)N n Medium event N * has the same polarity as the currently triggered event E n ; (2)N n Medium event N * Is located within the L×L neighborhood of the currently triggered event E n ; (3)N n The middle event does not include the currently triggered event E n ; (4)N n The medium event is the currently triggered event E n Or the event with the smallest spatial position difference among other events in the event set within the L×L spatial region. If the spatial distance differences are the same, select the event with the smallest timestamp difference from the currently triggered event E n ; (5) Select the total number of events n≥L.

9. The event sensor optical flow estimation device based on plane fitting according to claim 8, characterized in that The said model construction module is also used for: After selecting the optimal local neighboring event set, use the sorting of good and bad events in the event set under the Prim greedy algorithm to select the optimal M events in the event set as the initial inlier set; Assume that the initial model expression for plane fitting is ax + by + ct = d. Under the constraint condition of a 2 + b 2 + c 2 = 1, minimize the spatial distance d from each event in the initial inlier set to the fitting plane, and use the Lagrange multiplier method for estimation. The expression is as follows: i ​ d i = |ax + by + ct - d| (1) Wherein, a, b, c, and d are all fitting parameters to be solved. Since the value of d has nothing to do with the optical flow estimation value, it is assumed that the value of d is 0. At this time, the expression of the Lagrangian function corresponding to formula (1) is as follows: wherein λ is a constant to be determined; Take the partial derivative of formula (2) and make it equal to 0. Its expression is as follows: At this time, the eigenvector corresponding to the minimum eigenvalue is the fitting parameter a, b, c obtained, and the initial plane fitting model can be obtained according to the fitting parameters.

10. The event sensor optical flow estimation device based on plane fitting according to claim 9, wherein, The said optical flow estimation module is also used for: (1) Substitute the remaining events in the optimal local neighboring event set into the initial model, and calculate the fitting error at this time. If the fitting error value is less than the set threshold, add it to the inlier set S0; (2) If the number of events contained in the current inlier set S0 is greater than the current optimal inlier set S pre , then update S pre = S0; (3) Repeat and iterate the above steps, and the S with the largest number of internal points pre The obtained model is the optimal plane fitting model; According to the formula Event sensor optical flow estimation is achieved using an optimal plane fitting model, where g is the spatial gradient of the fitting plane, and a1, b1, and c1 represent the fitting parameters in the optimal plane fitting model, and v x represents the estimated value of the optical flow in the horizontal direction, and v y represents the estimated value of the optical flow in the vertical direction.

Images