A method for visual templates of event data based on multi-angle projection

Abstract

This invention proposes a method for visual templates of event data based on multi-angle projection, comprising the following steps: Step 1, normalizing the time axis of the event data; Step 2, projecting all event data onto the projection plane; Step 3, dividing the projection plane into multiple concentric circular sectors; Step 4, selecting multiple sets of different azimuth and elevation angles to obtain multiple projection planes; Step 5, when matching event data obtained from two scenes, calculating the cosine distance based on each row of the global descriptor matrix, and finally accumulating the sum as the similarity; Step 6, when calculating the similarity between two global descriptor matrices, the matrices are shifted column by column by a certain step size to improve the robustness of matching; Step 7, applying the above scene description and scene matching methods to the visual template module in the RatSLAM model; ultimately, scenes can be effectively identified, and the number of incorrectly identified scenes can be reduced.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a method for visual templates of event data based on multi-angle projection, which can solve the problem of scene recognition using event cameras. Background Technology

[0002] Mobile robots perform localization and mapping tasks in unfamiliar environments. When encountering the same scene, the robot can identify that scene and influence its localization results, causing loop closures in its map. In the field of traditional cameras, many mature scene recognition methods have been applied to loop closure detection, such as SeqSLAM and NetVLAD. SeqSLAM relies on manually set image frame features, performing scene matching on short-term local navigation sequences and identifying coherent sequences of local best matches. NetVLAD (VLAD for Vector of Locally Aggregated Descriptors) is a weakly supervised neural network learning method based on triple joint loss, which can combine multiple local descriptors of an image frame into a global descriptor through a neural network. In contrast, there is less research on loop closure detection methods based on event data. Existing methods often reconstruct event frames from event data over a time window using motion compensation and then apply traditional camera-based methods for scene recognition. For example, after reconstructing event frames from event data over a period of time, downsampling them and inputting them into a low-resolution SeqSLAM to achieve scene recognition. Such methods are often computationally complex and do not take advantage of the characteristics of the event data itself.

[0003] The key challenge in loop closure detection based on event data lies in the representation of the event data, i.e., how to use the information provided by the event data to describe and match the scene. Inspired by a point cloud scene recognition method, this paper implements a loop closure detection method based on event data and integrates it with the RatSLAM model. This model is then used for loop closure detection to obtain better experience map results.

[0004] Loop closure detection technology primarily assists mobile robots in localization by identifying previously traversed scenes. Therefore, the quality of loop closure detection results directly impacts the success of the robot's navigation task. To evaluate the results, metrics such as the number of correctly detected loop closures and the number of correctly undetected loop closures are typically used as evaluation criteria. To implement a loop closure detection method based on event data, the following two aspects need to be considered:

[0005] (1) Encode the event data so that its expression can reflect the visual information contained in the scene and can be used as a visual template in the RatSLAM model.

[0006] (2) The global descriptor obtained from scene recognition based on event data, and the matching rules established, can effectively measure the similarity between different scenes, and ultimately ensure the matching and generation of visual templates. Summary of the Invention

[0007] The purpose of this invention is to provide a method for visual templates of event data based on multi-angle projection, in order to solve the problem of loop closure detection using event data.

[0008] The technical solution of this invention is: a method for visual templates of event data based on multi-angle projection, comprising the following steps:

[0009] Step 1: After normalizing the event data time axis, perform principal component analysis to obtain eigenvectors. Use the eigenvectors as new coordinate axes to establish a new local coordinate system and align the event data, thereby achieving rotation invariance of the global descriptor.

[0010] Step 2: Determine a projection plane based on a set of azimuth and elevation angles, and project all event data onto this plane;

[0011] Step 3: Divide the projection plane into multiple concentric circular sectors, count the number of projection event points in each sector in a certain order, and finally normalize the quantity histogram.

[0012] Step 4: Select multiple sets of different azimuth and elevation angles to obtain multiple projection planes. Project the event data onto each projection plane to obtain multiple quantity histograms. Finally, combine the quantity histograms to obtain the global descriptor matrix.

[0013] Step 5: When matching the event data obtained from the two scenarios, calculate the cosine distance based on each row of the global descriptor matrix, and finally accumulate the sum to obtain the similarity.

[0014] Step 6: When calculating the similarity between two global descriptor matrices, the matrices are shifted column by column by a certain step size to improve the robustness of the matching.

[0015] Step 7: Apply this scene description and scene matching method to the visual template module in the RatSLAM model.

[0016] In the above technical solution, the event data in step 1 consists of n events over a period of time, which are obtained by an event camera. The event camera is a bionic vision sensor with asynchronous output, and its output information is called an event, including image plane coordinates, timestamps, and polarity.

[0017] In the above technical solution, step 1 normalizes the event data timeline. The most recent and furthest time values ​​of events in the event data are found, and all time values ​​within that interval are recalculated.

[0018] In the above technical solution, the principal component analysis (PCA) in step 1 is a method for reducing the dimensionality of data, and the resulting feature vectors represent the main trends of all data.

[0019] In the above technical solution, the projection plane in step 2 is represented by a normal vector, which can be represented by a set of azimuth and elevation angles.

[0020] In the above technical solution, the maximum concentric circle in step 3 is determined by finding the projection event point farthest from the origin on the projection plane, and using this as the maximum radius of the concentric circle.

[0021] In the above technical solution, in step 4, each row of the global descriptor matrix represents a specific set of azimuth angles θ and elevation angles. The distribution of projection event points on the lower projection plane.

[0022] In the above technical solution, step 5 shows that the cosine distance is the cosine value of the angle between two vectors, which can reflect the consistency of the two vectors in multiple dimensions.

[0023] In the above technical solution, the matrix translation in step 6 means shifting the matrix to the left or right column by column, which is reflected on the projection plane, i.e., sector rotation.

[0024] In the above technical solution, RatSLAM in step 7 is a rodent-like SLAM method. The principle of the visual template is to convert the current scene into a descriptor as a visual template, and match it with the stored historical templates one by one. When the similarity is small, a new visual template is stored.

[0025] Beneficial effects: This invention provides a method for visual templates of event data based on multi-angle projection. The global descriptor extraction method in point cloud is modified and applied to event data to obtain a global descriptor matrix based on multi-angle projection. A scene matching principle based on geometric signature is established, and the similarity of global descriptors is calculated by cosine distance and matrix translation. This method can directly use event data for scene recognition. Attached Figure Description

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0027] Figure 1 This is a flowchart of the method;

[0028] Figure 2 A flowchart describing the scene;

[0029] Figure 3 Event data prior to the transformation of the local coordinate system;

[0030] Figure 4 This refers to event data after the local coordinate system transformation.

[0031] Figure 5 Explanation of pitch and azimuth angles;

[0032] Figure 6 Explanation of sector division for projected event points. Detailed Implementation

[0033] In order to achieve the purpose of this invention, such as Figures 1-6 As shown, in one embodiment of the present invention, a method for creating an event data visual template based on multi-angle projection includes the following steps:

[0034] Step 1: After normalizing the event data time axis, perform principal component analysis to obtain eigenvectors. Use the eigenvectors as new coordinate axes to establish a new local coordinate system and align the event data, thereby achieving rotation invariance of the global descriptor.

[0035] Step 2: Determine a projection plane based on a set of azimuth and elevation angles, and project all event data onto this plane;

[0036] Step 3: Divide the projection plane into multiple concentric circular sectors, count the number of projection event points in each sector in a certain order, and finally normalize the quantity histogram.

[0037] Step 4: Select multiple sets of different azimuth and elevation angles to obtain multiple projection planes. Project the event data onto each projection plane to obtain multiple quantity histograms. Finally, combine the quantity histograms to obtain the global descriptor matrix.

[0038] Step 5: When matching the event data obtained from the two scenarios, calculate the cosine distance based on each row of the global descriptor matrix, and finally accumulate the sum to obtain the similarity.

[0039] Step 6: When calculating the similarity between two global descriptor matrices, the matrices are shifted column by column by a certain step size to improve the robustness of the matching.

[0040] Step 7: Apply this scene description and scene matching method to the visual template module in the RatSLAM model;

[0041] In the above technical solution, the event data in step 1 consists of n events EventBin = {e1, e2, e3, ..., e...} within a certain period of time. n}, where each event e i ={u i , t i} contains pixel position information u i and timestamp t i information.

[0042] In the above technical solution, step 1 normalizes the event data timeline to find the most recent time value t of the event in the event data. max and the farthest time value t min All event data are normalized as shown in Formula 1 to facilitate subsequent calculations:

[0043]

[0044] In the above technical solution, the specific steps to achieve rotation invariance in step 1 involve extracting the first three dimensions of all event data and writing them in matrix form:

[0045]

[0046] Principal Component Analysis (PCA) is performed on matrix E to obtain three eigenvectors P1, P2, and P3, representing the three main trends of all event data. These are used as the three coordinate axes of a local coordinate system, and a coordinate system transformation is performed on all event data.

[0047]

[0048] The event data matrix E obtained after the coordinate system transformation is:

[0049] E = [X] T Y T Z T (4)

[0050] In the above technical solution, the projection plane in step 2 is defined as Pl, which can be represented by the corresponding normal vector m. Specifically, the normal vector m is composed of a specific set of azimuth angles θ and pitch angles. The elevation angle is uniquely determined by selecting a set of azimuth angles θ and elevation angles. Then, its corresponding normal vector is

[0051] In the above technical solution, the projection step of the event data in step 2 is to project each event point EP in each row of the event data matrix E.i =[x i y i , z i The event projection point is ep, which is projected onto the plane Pl corresponding to m. i The specific projection formula is as follows:

[0052]

[0053] In the above technical solution, step 3 is specifically described as follows: Find the projection event point farthest from the origin on the projection plane P1, and use this as the maximum radius of the concentric circles. Then, divide the projection plane into L concentric circles, and each concentric circle into T sectors. For each sector, after labeling the sector, count the number N of projection event points within each sector in sequence. i Finally, a vector of length L*T is obtained to describe the distribution of event data points on the projection plane:

[0054] des = [N 1 N 2 N 3 , ..., N L*T (6)

[0055] Find the maximum value N in des max and minimum value N min Normalize the vector to facilitate subsequent similarity matching:

[0056]

[0057] In the above technical solution, step 4 combines the DES obtained from different projection planes together as a global descriptor matrix. Each row of the global descriptor matrix represents a specific set of azimuth angles θ and elevation angles. The distribution of projection event points on the lower projection plane.

[0058] In the above technical solution, step 5 is specifically described as follows: Each row of the global descriptor matrix represents the distribution of projected event points on a specific 2D plane. Therefore, the matrix can be written in the following form:

[0059]

[0060] There are two global descriptor matrices obtained based on event data. A and descriptor B The rules for judging their similarity are as follows: the two row vectors of each row and Calculate its corresponding cosine value, and finally sum all the cosine values, that is:

[0061]

[0062] The above describes the process of evaluating the similarity of the global descriptor matrices generated from the event data obtained from two scenarios, ultimately resulting in a similarity score. The higher the score, the higher the similarity between the two scenarios.

[0063] In the above technical solution, the specific steps of step 6 are as follows:

[0064] When calculating the similarity between two global descriptor matrices, the matrices are shifted left or right column-wise, multiple similarity scores are calculated, and the maximum value is taken as the score result. The practical significance of shifting the global descriptor matrices to the left and right is reflected in the projected 2D plane as a sector rotation.

[0065] Expanding the global descriptor matrix results in:

[0066]

[0067] When performing similarity scoring, the global descriptor matrix can be shifted to the left or right by a certain step size, as shown in Formulas 11 and 12, with a corresponding step size of 1.

[0068]

[0069]

[0070] In the above technical solution, the specific steps of step 7 are as follows:

[0071] The principle of visual templates is to convert the current scene into a descriptor as a visual template, and match it one by one with the stored historical templates. When the similarity is small, the new visual template is stored. When a new scene is encountered, the event data is converted into a global descriptor matrix, and then the similarity is calculated with the stored historical visual templates. If the maximum similarity score still does not reach the threshold, the global descriptor matrix is ​​stored as a new visual template.

[0072] Of course, the above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be used to limit the scope of protection of the present invention. All modifications made according to the spirit and essence of the main technical solution of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for creating visual templates for event data based on multi-angle projection, characterized in that, Includes the following steps: Step 1: After normalizing the event data time axis, perform principal component analysis to obtain eigenvectors. Use the eigenvectors as new coordinate axes to establish a new local coordinate system and align the event data, thereby achieving rotation invariance of the global descriptor. Step 2: Determine a projection plane based on a set of azimuth and elevation angles, and project all event data onto this projection plane; Step 3: Divide the projection plane into multiple concentric circular sectors, count the number of projection event points in each sector in a certain order, and finally normalize the quantity histogram. Step 4: Select multiple sets of different azimuth and elevation angles to obtain multiple projection planes. Project the event data onto each projection plane to obtain multiple quantity histograms. Finally, combine the quantity histograms to obtain the global descriptor matrix. Step 5: When matching the event data obtained from the two scenarios, calculate the cosine distance based on each row of the global descriptor matrix, and finally accumulate the sum to obtain the similarity. Step 6: When calculating the similarity between two global descriptor matrices, the matrices are shifted column by column by a certain step size to improve the robustness of the matching. Step 7: Apply the above scene description and scene matching method to the visual template module in the RatSLAM model.

2. The method for creating a visual template for event data based on multi-angle projection according to claim 1, characterized in that, The event data in step 1 consists of n events over a period of time. Each event Has pixel position information and timestamp information; Normalize the event data timeline to find the most recent time value of each event in the data. and farthest time value All event data is normalized to facilitate subsequent calculations: （1） The specific steps to achieve rotation invariance in step 1 involve extracting the first three dimensions of all event data and rewriting them in matrix form: （2） For matrix Principal component analysis was performed to obtain three eigenvectors. , and These represent the three main trends of all event data. Using them as the three axes of a local coordinate system, we can transform the coordinate system of all event data as follows: （3） Obtain the event data matrix after coordinate transformation for: （4） 。 3. The method for creating a visual template for event data based on multi-angle projection according to claim 1, characterized in that, In step 2, the projection plane is defined as It can be derived from the corresponding normal vector. Specifically, the normal vector It is also composed of a specific set of azimuth angles and pitch angle The only certainty is that, after selecting a set of azimuth angles and pitch angle Then, its corresponding normal vector is ; The projection step of the event data in step 2 is to project the event data matrix. Each row of event points Projected onto Corresponding plane The event projection point obtained is as follows: The specific projection formula is as follows: （5） 。 4. The method for creating a visual template for event data based on multi-angle projection according to claim 3, characterized in that, The specific description of step 3 is as follows: on the projection plane Find the projection event point farthest from the origin, and use it as the maximum radius of the concentric circles. Then divide the projection plane into... There are 10 concentric circles, and each concentric circle is further divided into 10 concentric circles. Each sector; for each sector, after marking the sequence number, the number of projected event points within that sector is counted sequentially. The final length is Vectors are used to describe the distribution of event data points on the projection plane: （6） turn up The maximum value in and minimum value Normalize the vector to facilitate subsequent similarity matching: （7） 。 5. The method for visual templates of event data based on multi-angle projection according to claim 4, characterized in that: In step 4, different projection planes are obtained Combined together as a global descriptor submatrix Each row of the global descriptor submatrix represents a specific set of azimuth angles. and pitch angle The distribution of projection event points on the lower projection plane.

6. The method for creating a visual template for event data based on multi-angle projection according to claim 5, characterized in that, Step 5 Global Description Submatrix Each row represents the distribution of projected event points on a specific 2D plane. The matrix can be written in the following form: （8） There are two global descriptor matrices obtained based on event data. and The rules for judging their similarity are as follows: the two row vectors of each row and Calculate the corresponding cosine value, and finally sum all the cosine values, that is: （9） The above describes the process of evaluating the similarity of the global descriptor matrices generated from the event data obtained from two scenarios, ultimately resulting in a similarity score. The higher the score, the higher the similarity between the two scenarios.

7. The method for creating a visual template for event data based on multi-angle projection according to claim 6, characterized in that, In step 6, when calculating the similarity between two global descriptor matrices, the matrices are shifted to the left or right by columns, multiple similarity scores are calculated, and the maximum value is taken as the score result. The actual meaning of shifting the global descriptor matrices to the left and right is reflected on the 2D projection plane as the rotation of the sector. global description submatrix Expanded, it means: （10） When performing similarity scoring, the global descriptor matrix can be shifted left or right by a certain step size, with a corresponding step size of 1: （11） （12） 。 8. The method for creating a visual template for event data based on multi-angle projection according to claim 1, characterized in that, In step 7, RatSLAM is a rodent-like SLAM method. The principle of the visual template is to convert the current scene into a descriptor as a visual template, and match it with the stored historical templates one by one. When the similarity is small, a new visual template is stored.

Images