A straight line detection method based on dynamic vision sensor

Abstract

The present invention relates to a straight line detection method based on a dynamic visual sensor. An event camera is used on an aircraft for real-time shooting, and a filtering operation is performed on the initial event stream shot by the event camera; the filtered events are added to the detected existing lines, and the distance evaluation is performed on the incoming events to determine whether they belong to the straight line, and the events that meet the distance evaluation requirements are added to the line; cluster processing is performed on the unadded events, and the minimum eigenvalue is obtained using the covariance matrix, and then clustering is promoted to a straight line. The problem that the traditional camera of the aircraft cannot complete the straight line detection and tracking task due to motion blur, poor lighting conditions and other problems when operating at high altitudes is solved. The straight line detection and tracking algorithm based on the event camera is used to achieve the task of completing the straight line detection and tracking by the aircraft using the event camera.

Description

Technical Field

[0001] The invention belongs to the field of target detection and recognition, and relates to a straight line detection method based on a dynamic vision sensor. Background Art

[0002] Autonomous inspection of power cables is challenging, and manual inspection is time-consuming and laborious. In some high-altitude operations, visual perception equipment is often installed on aircraft to complete straight line detection. Among them, the aircraft needs continuous perception to carefully detect thin strips. Using traditional visual perception equipment, such as ordinary cameras, motion blur often occurs when the aircraft moves, and poor lighting conditions cause the camera to be unable to recognize. However, event cameras are robust to motion blur and have advantages such as low latency and high dynamic range. These characteristics are conducive to the autonomous inspection of power cables by event cameras, and event cameras can be installed on aircraft for straight line detection and tracking of power cables.

[0003] In the prior art, traditional cameras located in aircraft cannot complete line detection and tracking tasks when operating at high altitudes due to motion blur, poor lighting conditions, and other issues. This paper proposes a line detection and tracking method based on an event camera, which identifies lines in the event stream by detecting planes in the spatiotemporal signal and completes tracking. Summary of the invention

[0004] Technical issues to be solved

[0005] In order to avoid the shortcomings of the prior art, the present invention proposes a line detection method based on a dynamic vision sensor, which is aimed at the problem that the traditional camera located in the aircraft cannot complete the line detection and tracking tasks due to motion blur, poor lighting conditions and other problems when operating at high altitudes. Based on the characteristics of the event camera, a line detection and tracking algorithm based on the event camera is proposed to enable the aircraft to complete the line detection and tracking tasks using the event camera.

[0006] Technical Solution

[0007] A straight line detection method based on a dynamic visual sensor is characterized in that an event camera is used on an aircraft, and the steps of implementing straight line detection are as follows:

[0008] Step 1: Filter the initial event stream captured by the event camera;

[0009] Step 2: Add the events that pass the filter to the detected existing lines, perform distance evaluation on the incoming events to determine whether they belong to a straight line, and add the events that meet the distance evaluation requirements to the line; if multiple lines meet the requirements, ignore the event;

[0010] The distance assessment requirements are: the vertical distance a from the event to the existing line is less than 1.8 pixels, and the length from the event to the midpoint of the known line is less than the length of the known line;

[0011] Step 3: If the event in step 2 is not added to the line, try to add it to the set cluster;

[0012] When the distance of an event to the midpoint of the extrapolated line and the distance to the extrapolated line are less than 1 pixel, the event is added to the cluster;

[0013] If the angles between different clusters are less than 15°, the clusters are merged. If the event is successfully added to the cluster, the events in the cluster are detected. If the cluster has collected more than 35 events, step 4 is executed to promote the cluster to a line.

[0014] The set cluster is a set of events that are not added to the line;

[0015] Step 4: Calculate the covariance matrix of the cluster event coordinates in step 3, perform eigendecomposition on the covariance matrix, and obtain the minimum eigenvalue; when the minimum eigenvalue is less than 1.2 pixels, the cluster is promoted to a straight line;

[0016] Then calculate the characteristic information of the straight line to obtain the straight line length l; replace the known line length in step 2 with the straight line length l;

[0017] Step 5: Events that are neither added to existing rows nor added to clusters remain unassigned and stored in the SAE. Using the event position as a starting point, the event is detected in the SAE.

[0018] Find a chain of events connected to each other. The elements in the chain are called chain elements. There are two independent SAEs, one for each polarity. The first chain element is the incoming event, and the second chain element is the latest event among the adjacent elements, which is no more than 10ms old. Search for the next element in a predefined search space based on the relative position of the last element and the current element. If no element is found, expand this space.

[0019] Step 6: After completing the above steps, check the current clusters and lines regularly according to the above steps; if there are events in the cluster or line that exceed 50 milliseconds, delete it; clusters whose latest events exceed 40ms will also be deleted; for clusters that are not deleted, update their inference lines by merging the newly added events.

[0020] The filtering operation is as follows: Step 1: All events are triggered at the same position. After the event is triggered, the events with the same polarity are suppressed for 8ms and no response occurs. In this way, the filter suppresses the triggering of false events with the same polarity. Step 2: Check the range of 5*5 pixels around the incoming event stream. If at least three events are found, the filter passes the screening filter.

[0021] In step 4, the covariance matrix is ​​calculated: the coordinates are centered so that the mean is zero, and the covariance matrix Where X is the matrix containing the coordinates of the event centers, N is the number of events, and QΔQ T It is a positive definite matrix expression, and the minimum eigenvalue is obtained according to the covariance matrix.

[0022] In step 4, the characteristic information of the straight line is calculated: the line direction d, the midpoint p, and the direction line length l are calculated; d is the intersection of the xy plane at t and the mounting plane; the normal vector et of the xy plane is [0 0 1], and the normal vector n of the fitting plane is [nx ny nt]; the direction of the line d is n×e t =[n2 -n1 0] T ;point The events are evenly distributed based on the straight line, so the standard deviation formula of the uniform distribution should be used to calculate the line length. The data of the event distribution is centered, and E[Γ] = 0 is obtained, thus obtaining the straight line length

[0023] Beneficial Effects

[0024] The present invention proposes a method for detecting a straight line based on a dynamic visual sensor. An event camera is used on an aircraft for real-time shooting. The initial event stream shot by the event camera is filtered. Events that pass the filtering are added to the detected existing lines. The incoming events are evaluated for distance to determine whether they belong to a straight line. Events that meet the distance evaluation requirements are added to the line. Clustering is performed on the unadded events, and the minimum eigenvalue is obtained using the covariance matrix, and then clustered and promoted to a straight line. The problem that the traditional camera of an aircraft cannot complete the straight line detection and tracking task due to motion blur, poor lighting conditions and other problems when operating at high altitudes is solved. A straight line detection and tracking algorithm based on an event camera is used to achieve the task of using an event camera to complete the straight line detection and tracking task on an aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 :Event flow processing flow chart

[0026] Figure 2 :Event flow processing flow chart DETAILED DESCRIPTION

[0027] The present invention will now be further described with reference to the embodiments and the accompanying drawings:

[0028] The present invention aims at the problem that the traditional camera located in the aircraft cannot complete the line detection and tracking tasks due to motion blur, poor lighting conditions and other problems when operating at high altitude. Based on the characteristics of event camera, a line detection and tracking algorithm based on event camera is proposed to enable the aircraft to complete the line detection and tracking tasks using event camera.

[0029] Step 1: The initial event stream is filtered by two filtering methods: non-response event stream information and neighborhood filtering. The filtering operation mainly consists of two steps; the first step: set the refractory period to 8ms if the trigger event polarity is the same, and the refractory period to 1ms if the trigger event polarity is different; no response occurs. In this way, the filter suppresses the triggering of false events of the same polarity. The second step: check the range of 5*5 pixels around the incoming event stream, and detect the event stream of 5*5 pixels 70ms volume. If at least three events are detected, it passes the screening filter.

[0030] Step 2: Events that pass the initial filtering process can be added to the existing detected lines, and the distance of the incoming events can be evaluated to determine whether they belong to a straight line.

[0031] By judging two distances to decide whether it can be added to the existing line, the distance of the incoming event to the line a and the distance to the center of the line b are calculated. Figure 2 As shown, the gray square is an event. If the distance a from the event to the existing line is less than 1.8 pixels and the length from the event to the midpoint of the known line is less than the line length, the event is added to the line; if multiple lines meet the requirements, the event is ignored.

[0032] Step 3: If the event in step 2 is not added to the line, try to add it to the set cluster. The cluster is a collection of events that are not added to the line. There will be different clusters around the line. This step depends on two distances, which are the distance from the event to the midpoint of the inferred line and the distance to the inferred line. Among them, the inferred line is where the events in the cluster are clustered together, and it will have a state similar to a line. The event can only be added to the cluster if the distance from the event to the midpoint and the distance to the inferred line are less than 1 pixel. If the angles between different clusters are less than 15°, the clusters are merged. If the event is successfully added to the cluster, detect the events in the cluster. If the cluster has collected more than 35 events, execute step 4 to promote the cluster to a line;

[0033] Step 4: If the events belonging to the clusters in step 3 lie on a plane, calculate the covariance matrix of the event coordinates and if its smallest eigenvalue is below 1.2 pixels, lift the cluster to a straight line. To calculate the covariance matrix, the coordinates are centered so that the mean is zero. Perform eigendecomposition, and if the smallest eigenvalue is less than 1.2 pixels, the cluster is lifted to a straight line, and the fitting plane is located at the center of gravity of the event.

[0034] Where X is the matrix containing the coordinates of the event centers, N is the number of events, and QΔQ T is the expression of a positive definite matrix.

[0035] Calculate the line direction d, midpoint p, and direction line length l; replace the known line length in step 2 with the straight line length l;

[0036] d is the intersection of the xy plane at t and the mounting plane; the normal vector of the xy plane et = [0 0 1], 1.2 the normal vector of the fitting plane n = [nx ny nt]; the direction of the line d = n×e t =[n2 -n1 0] T .point The events are evenly distributed based on the straight line, so the standard deviation formula of the uniform distribution should be used to calculate the line length. The data of the event distribution is centered, and we can obtain E[Γ]=0, so we can get the length of the line

[0037] Step 5: Events that are neither added to existing rows nor added to clusters will remain unassigned and stored in the SAE. Using the event position as a starting point, the event is detected in the SAE to find a chain of events connected to each other. The elements in the event chain are called chain elements. There are two independent SAEs, one for each polarity. The first chain element is the incoming event. The second chain element is the latest event among the adjacent elements, which is no more than 10ms. The next element is searched in the predefined search space based on the relative position of the last element and the current element. If the element is not found, this space is expanded.

[0038] Step 6: The tracker performs a periodicity check on the current clusters and lines. Events belonging to clusters or lines older than 50ms are removed. Clusters whose latest events are older than 40ms are also removed. For the remaining clusters, their inferred lines are updated by merging the newly added events. The periodicity check of the line depends on the state of the line. For initialized lines, we check if their initialization period has ended, in which case we calculate their connection length. If the connection length exceeds 70 pixels, the line will change from the initialized state to the activated state.

Claims

1. A straight line detection method based on a dynamic vision sensor, characterized in that: The event camera is used on the aircraft, and the steps to achieve straight line detection are as follows: Step 1: Filter the initial event stream captured by the event camera; Step 2: Add the filtered events to the detected existing lines, perform distance evaluation on the incoming events to determine whether they belong to the straight line, and add the events that meet the distance evaluation requirements to the line; If multiple lines match, the event is ignored; The distance assessment requirements are: the vertical distance a from the event to the existing line is less than 1.8 pixels, and the length from the event to the midpoint of the known line is less than the length of the known line; Step 3: If the event in step 2 is not added to the line, try to add it to the set cluster; When the distance of an event to the midpoint of the extrapolated line and the distance to the extrapolated line are less than 1 pixel, the event is added to the cluster; If the angles between different clusters are less than 15°, the clusters are merged; If the event is successfully added to the cluster, detect the events in the cluster. If the cluster has collected more than 35 events, execute step 4 to promote the cluster to a line. The set cluster is a set of events that are not added to the line; Step 4: Calculate the covariance matrix of the cluster event coordinates in step 3, perform eigendecomposition on the covariance matrix, and obtain the minimum eigenvalue; When the minimum eigenvalue is less than 1.2 pixels, the cluster is promoted to a straight line; Then calculate the characteristic information of the straight line to obtain the straight line length l; replace the known line length in step 2 with the straight line length l; Step 5: Events that are neither added to existing rows nor added to clusters remain unassigned and stored in the SAE. Using the event position as a starting point, the event is detected in the SAE. Find a chain of events connected to each other. The elements in the chain are called chain elements. There are two independent SAEs, one for each polarity. The first chain element is the incoming event, and the second chain element is the latest event among the adjacent elements, which is no more than 10ms old. Search for the next element in a predefined search space based on the relative position of the last element and the current element. If no element is found, expand this space. Step 6: After completing the above steps, check the current clusters and lines regularly according to the above steps; if there are events in the cluster or line that exceed 50 milliseconds, delete it; clusters whose latest events exceed 40ms will also be deleted; for clusters that are not deleted, update their inference lines by merging the newly added events.

2. The straight line detection method based on dynamic vision sensor according to claim 1 is characterized in that: The filtering operation is as follows: Step 1: All events are triggered at the same position. After the event is triggered, the same polarity event is suppressed for 8ms without response. In this way, the filter suppresses the triggering of false events of the same polarity; Step 2: Check the 5*5 pixel range around the incoming event stream. If there are at least three events in it, they pass the screening filter.

3. The straight line detection method based on dynamic vision sensor according to claim 1 is characterized in that: In step 4, the covariance matrix is ​​calculated: the coordinates are centered so that the mean is zero, and the covariance matrix Where X is the matrix containing the coordinates of the event centers, N is the number of events, and QΔQ T It is a positive definite matrix expression, and the minimum eigenvalue is obtained according to the covariance matrix.

4. The straight line detection method based on dynamic vision sensor according to claim 1 is characterized in that: In step 4, the characteristic information of the straight line is calculated: the line direction d, the midpoint p, and the direction line length l are calculated; d is the intersection of the xy plane at t and the mounting plane; the normal vector et of the xy plane is [0 0 1], and the normal vector n of the fitting plane is [nx ny nt]; the direction of the line d is n×e t =[n2 -n1 0] T ;point The events are evenly distributed based on the straight line, so the standard deviation formula of the uniform distribution should be used to calculate the line length. The data of the event distribution is centered, and E[Γ] = 0 is obtained, thus obtaining the straight line length

Images