Event-based feature extraction and corner detection method
By using a combination of high-dimensional feature extraction core and multiplication and addition models in event cameras, the problems of corner point detection efficiency and accuracy on low-performance hardware are solved, and efficient corner point detection in low-power devices are achieved.
Patent Information
- Application Number
- CN202510191451.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-20
- Publication Date
- 2025-06-10
AI Technical Summary
The prior art is difficult to achieve efficient and accurate corner detection on low-performance hardware, and most methods require large amounts of data to ensure the extracted feature robustness.
The feature extraction and corner point detection method based on event cameras are used to generate high-dimensional feature extraction kernels and use multiplication and addition models to perform high-dimensional coding, and combine binding and unbinding operations to determine corner points.
Effectively save computing resources and energy in low-power equipment, achieving efficient corner detection, taking into account detection efficiency and accuracy.
Smart Images
Figure CN120125838A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of event cameras, and in particular to an event-based feature extraction and corner point detection method. Background Art
[0002] An event camera is a new type of bionic sensor. Unlike traditional frame cameras, it asynchronously measures the brightness changes of each pixel and outputs a series of events, including time, position, and brightness changes. Compared with traditional cameras, event cameras can capture moving images in the scene in real time. They have the advantages of low latency, low power consumption, and high dynamic range. They are suitable for emerging fields such as self-driving cars, drones, and autonomous robots. Figure 1 Schematic diagram of the output of event camera and traditional camera.
[0003] Currently, most of the corner detection algorithms for events are derived from the frame-based accelerated segment test feature detection algorithm (Features from Accelerated Segment Test, FAST) and the Harris Corner Detection Algorithm. These methods are relatively simple to calculate, easy to implement and understand, can quickly detect corners to a certain extent, and can adapt to different scenarios and needs by adjusting parameters such as thresholds and radius.
[0004] However, frame-based corner detection algorithms are all based on artificial neural networks for feature extraction, which usually requires high hardware computing power. For example, corner detection algorithms based on Harris features often have high detection accuracy, but low detection efficiency, while algorithms based on accelerated segmented test features are the opposite. In addition, some technologies use pulse neural networks for feature extraction, but this requires proprietary hardware support and is relatively limited in application. At the same time, both of these feature extraction methods often require a large amount of data. If the amount of data is insufficient, the extracted features may be inaccurate or not robust.
[0005] How to achieve effective corner detection on low-performance hardware while balancing efficiency and accuracy is a challenge that needs to be addressed urgently. Summary of the invention
[0006] In order to alleviate or partially alleviate the above technical problems, the solution of the present invention is as follows:
[0007] On the one hand, the present invention discloses an event-based feature extraction and corner point detection method, comprising the following steps:
[0008] Step S1, generating an event frame based on the event stream output by the event camera;
[0009] Step S2: Through the first basis super vector P∈Bd and the second basis hypervector Θ ∈ B d The binding operation generates a high-dimensional feature extraction kernel, and the high-dimensional feature extraction kernel is used to perform high-dimensional encoding on the event frame to generate a high-dimensional feature vector of the event frame;
[0010] Among them, the basis hypervector is a set of hypervectors, which is the smallest information unit encoded in a high-dimensional space or hyperspace. d refers to the dimension, B refers to the encoding domain of the multiply-accumulate model, and the value of each bit of the vector in the multiply-accumulate model is 1 or -1; the binding operation is used to associate information, the first basis hypervector is the basis hypervector representing the radius, and the second basis hypervector is the basis hypervector representing the angle; the second basis hypervector is a cyclic basis hypervector;
[0011] Step S3: Based on the unbinding operation and similarity calculation between the high-dimensional feature vector of the event frame and the first basis hypervector or the second basis hypervector, a similarity matrix is obtained for corner point determination; among them, the unbinding operation is the inverse operation of binding and is used to separate information.
[0012] Furthermore, the unbinding operation includes:
[0013] Performing an unbinding operation on the high-dimensional feature vector of the event frame and the first basis hypervector to obtain a second high-dimensional feature vector at each pixel, and the second high-dimensional feature vector is a high-dimensional feature vector representing the angle;
[0014] Or / and, performing an unbinding operation on the high-dimensional feature vector of the event frame and the second basis hypervector to obtain a first high-dimensional feature vector at each pixel, and the first high-dimensional feature vector is a high-dimensional feature vector representing the radius.
[0015] Furthermore, the similarity calculation is to calculate the cosine similarity of the vectors.
[0016] Furthermore, the similarity calculation includes:
[0017] Performing a similarity calculation on the second high-dimensional feature vector and the second basis hypervector to obtain a second similarity matrix;
[0018] Or / and, performing a similarity calculation on the first high-dimensional feature vector and the first basis hypervector to obtain a first similarity matrix;
[0019] Among them, the first similarity matrix and the second similarity matrix are transpose relations with each other.
[0020] Furthermore, projecting the first similarity matrix or / and the second similarity matrix in two dimensions of angle and radius respectively, and calculating using a sparsity evaluation function to obtain the sparsity of the angle and the sparsity of the radius;
[0021] When the sparsity of the radius is less than or equal to the first threshold and the angular sparsity is greater than or equal to the second threshold, the pixel is determined to be a corner point.
[0022] Further, the generation of the high-dimensional feature extraction kernel by the binding operation of the first basis hypervector and the second basis hypervector includes:
[0023] S211. Designate the center point of the high-dimensional feature extraction kernel as the coordinate origin, so as to obtain the abscissa and ordinate (x, y) of each pixel in the high-dimensional feature extraction kernel;
[0024] S212. Use the coordinate transformation formula to convert the abscissa and ordinate of each pixel into polar coordinates (ρ, θ) in the polar coordinate system, where ρ represents the radius and θ represents the angle
[0025] S213. Use the first basis hypervector and the second basis hypervector to encode the radius and the angle respectively;
[0026] S214. Use the binding operation to associate the first basis hypervector and the second basis hypervector to obtain the high-dimensional feature extraction kernel K ∈ B N×N×d , where N is a natural number, and N×N represents the size of the high-dimensional feature extraction kernel.
[0027] Further, the high-dimensional encoding of the event frame using the high-dimensional feature extraction kernel includes:
[0028] Perform a convolution operation on the event frame and the high-dimensional feature extraction kernel to obtain the high-dimensional feature vector of the event frame;
[0029] Assign 1 to all bits greater than 0, -1 to bits less than 0, and randomly assign 1 or -1 to bits equal to 0 in the high-dimensional feature vector of the event frame.
[0030] Further, frame compression processing is performed on the event stream within a preset time range to obtain an event frame.
[0031] The technical solution of the present invention has one or more of the following beneficial technical effects:
[0032] (1) Using the vector symbol framework algorithm, the event frame is encoded with high-dimensional vectors. Each bit value in the high-dimensional vector constituting the VSA is 1 or -1, which is very friendly to hardware. Especially in low-power devices, it can effectively save computing resources and energy.
[0033] (2) During the encoding of the event frame, the binding operation and the superposition operation are used, so that the calculation only involves bitwise multiplication and addition, reducing the calculation complexity and resource consumption, and having the characteristics of being highly hardware-friendly.
[0034] (3) The present invention utilizes the highly parallel and robust features of the multiply-add model, as well as its efficient encoding ability, to achieve a balance between hardware friendliness and algorithm precision performance.
[0035] In addition, other beneficial effects of the present invention will be mentioned in the specific embodiments. Description of the Drawings
[0036] Figure 1 It is a schematic diagram of the outputs of an event camera and a traditional camera;
[0037] Figure 2 It is a flowchart of the event-based feature extraction and corner detection method of the present invention;
[0038] Figure 3 It is a schematic diagram of corner detection in a certain preferred embodiment of the present invention;
[0039] Figure 4 It is a schematic diagram of the similarity matrix of corners and non-corners. Detailed Embodiments
[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.
[0041] The multiply-add-permute (MAP) model only requires simple bits to represent information, and each bit only requires 2-bit hardware resources. During calculation, it only involves bitwise multiplication and addition, making it highly hardware-friendly.
[0042] In this model, the value of each bit in the high-dimensional vector is either 1 or -1. The binding and unbinding operations are both achieved through component multiplication, while the superposition operation is achieved through component addition. The result of superposition can remain as it is. If 0 appears after superposition, it is randomly assigned as 1 or -1. Specifically, binding: represented by the symbol °, used to associate information; unbinding: represented by the symbol represented, which is the inverse operation of binding and is used to separate information; superposition: represented by the symbol +, which is the combination of information and is used to represent a group of information.
[0043] Figure 2 It is a flowchart of the event-based feature extraction and corner detection method of the present invention, including the following steps:
[0044] Step S1, generating an event frame.
[0045] Obtain the event stream output by the event camera, perform count frame compression processing on the event stream within a preset time range, and obtain an event frame.
[0046] Step S2: Extract features from the event frame.
[0047] The present invention uses a high-dimensional feature extraction kernel, that is, a high-dimensional vector, to encode the event frame to form a high-dimensional feature vector. The high-dimensional feature vector can contain more information, facilitating similarity analysis in subsequent corner detection and significantly improving the accuracy of corner detection. It mainly includes the following two steps:
[0048] Step S21: Generate a high-dimensional feature extraction kernel.
[0049] The present invention can generate a high-dimensional feature extraction kernel through the binding operation of base hypervectors.
[0050] The base hypervector is a group of hypervectors and is the smallest information unit encoded in a high-dimensional space or hyperspace. Usually, the base hypervector needs to be generated before feature extraction. At the same time, due to the topological similarity in the two-dimensional space, the randomness in the event can be reduced to affect the representation accuracy. The present invention uses the base hypervector to represent the radius and corner information of each pixel point of the high-dimensional feature extraction kernel in the polar coordinate system. Preferably, the present invention uses a cyclic base hypervector in the polar coordinate system to generate a high-dimensional feature extraction kernel. The polar coordinate system contains angle information and is more accurate in some rotating scenarios.
[0051] In the polar coordinate system, due to the cyclic nature of the angle information, for example, 0 and 2π are essentially the same angle on the plane, and the similarity of the base hypervectors representing 0 and 2π using the existing method is the worst.
[0052] To ensure cyclic correlation, the present invention uses the binding operation of the horizontal base hypervector corresponding to the radius and the cyclic base hypervector corresponding to the angle to represent the information of the N×N-dimensional high-dimensional feature extraction kernel. The specific steps are as follows:
[0053] S211: Designate the center point of the high-dimensional feature extraction kernel as the coordinate origin, so as to obtain the abscissa and ordinate (x, y) of each pixel within the high-dimensional feature extraction kernel;
[0054] S212: Use the coordinate conversion formula to convert the abscissa and ordinate of each pixel into polar coordinates (ρ, θ) in the polar coordinate system, where ρ represents the radius and θ represents the angle. The specific conversion process is as follows:
[0055]
[0056]
[0057] S213: Utilize two groups of base hypervectors P∈B dand Θ ∈ B d Encode ρ and θ respectively, where P represents the base hypervector of the radius ρ, also known as the first base hypervector; Φ represents the base hypervector of the angle θ, also known as the second base hypervector, and Φ is a set of cyclically related base hypervectors; B refers to the encoding domain of the MAP model, and the value of each bit is either 1 or -1; d refers to the dimension.
[0058] For the encoding of the radius ρ in the value range [0, r max , use a set of first base hypervectors P = {P 1 , P 2 , ……, P m}, P i ∈ B d Characterize the values within this radius range after equally dividing the interval m - 1 times. The higher the similarity in the closer places to ensure the topological similarity in the two-dimensional space.
[0059] Among them, m represents the degree of discretization of the maximum radius, P i = Φ(ρ i ), P i represents the hypervector corresponding to each radius ρ i , P m represents the hypervector corresponding to the maximum radius r max ), and P 1 represents the hypervector corresponding to the minimum radius. The minimum radius is the origin of the coordinate, that is, the radius value is 0.
[0060] For the encoding of the angle θ, due to the cyclic nature of the angle information itself, such as -π and π being essentially the same angle on the plane, the present invention uses a set of cyclically related base hypervectors Θ = {Θ 1 ,..., Θ n}, Θ j ∈ B d Characterize the values within this range after equally dividing the interval n times. n represents the degree of angle discretization in the interval -π ~ π; Θ j = Φ(θ j ), Θ j represents the hypervector corresponding to each angle θ j .
[0061] In the high-dimensional feature extraction kernel of the present invention, the radius and angle information at each pixel are a series of discrete values. By selecting sufficiently large m and n, the accuracy of the encoding is ensured. At the same time, using the mappings Φ(ρ i ) and Φ(θ j ), map the radius ρ and angle θ in each coordinate to the nearest neighbor P i and Θ j in the base hypervectors P and Θ, to achieve the correspondence between continuous values and discrete values.
[0062] S214. Use the binding operation to associate the first basis hypervector P and the second basis hypervector Θ to obtain a high-dimensional feature extraction kernel:
[0063]
[0064] where K represents the high-dimensional feature extraction kernel, and the symbol represents the binding operation, P (x,y) represents the basis hypervector related to the radius of the pixel point (x, y), and Φ (x,y) represents the cyclic basis hypervector of the pixel point (x, y).
[0065] In a preferred embodiment of the present invention, the high-dimensional feature extraction kernel is realized by performing a binding operation on the first basis hypervector and the second basis hypervector in the polar coordinate system. Among them, the binding operation is realized by component multiplication. The polar coordinate system contains angle information and is more accurate in some scenarios with rotation.
[0066] Step S22. Encode the event frame using the high-dimensional feature extraction kernel.
[0067] Let I represent the event frame with a size of H×W. The high-dimensional feature vector F of the event frame I I needs to contain the features from the N×N neighborhood centered on the pixel point (x, y). The present invention uses the convolution operation of the event frame I and the high-dimensional feature extraction kernel K to characterize this process, and assigns all bits greater than 0 in F I to 1, assigns bits less than 0 to -1, and randomly assigns 1 or -1 to bits equal to 0 to maintain the characteristics of the MAP model. The convolution operation formula is as follows:
[0068] F I = I * K, F ∈ B H×W×d
[0069] where F I represents the high-dimensional feature vector of the event frame, that is, the high-dimensional feature vector, I represents the event frame, and K represents the high-dimensional feature extraction kernel.
[0070] The high-dimensional feature vector F of the event frame generated by the present invention I contains the spatial structure and texture information of the pixel point itself and its neighborhood at each pixel point, and has rich information content.
[0071] Step S3. Detect corner points.
[0072] Figure 3 The following is a schematic diagram for corner point detection in a preferred embodiment of the present invention, including the following steps:
[0073] Step S31: Unbind the high-dimensional feature vector of the event frame from the first basis hypervector or the second basis hypervector that generates the high-dimensional feature extraction kernel, to obtain the high-dimensional feature vector representing the angle or the high-dimensional feature vector representing the radius at each pixel point.
[0074]
[0075] Among them, F I is the high-dimensional feature vector of the event frame, denotes the unbinding operation, P is the first basis hypervector, Θ is the second basis hypervector, and P and Θ correspond to the radius and the angle respectively; F θ ∈B m×d is the high-dimensional feature vector representing the angle, also known as the second high-dimensional feature vector, F ρ ∈B n×d is the high-dimensional feature vector representing the radius, also known as the first high-dimensional feature vector, and both are angle information and radius information described in the form of high-dimensional features.
[0076] Step S32: Calculate the similarity between the two high-dimensional feature vectors representing the angle and the radius and the two basis hypervectors that generate the high-dimensional feature extraction kernel, to obtain a similarity matrix. Specifically, calculate the similarity between the second high-dimensional feature vector F θ and the second basis hypervector Θ, or / and calculate the similarity between the first high-dimensional feature vector F ρ and the first basis hypervector P:
[0077]
[0078] Among them, sim refers to the similarity calculation, S θ,ρ is the second similarity matrix, S ρ,θ is the first similarity matrix, and the two are transpose relations, and R refers to the real number field.
[0079] Preferably, the similarity calculation is to calculate the cosine similarity between two vectors to be measured (such as the second high-dimensional feature vector F θ and the second basis hypervector Θ).
[0080] Step S33: Determine corner points based on the similarity matrix.
[0081] The angle information at the corner pixel is discrete and sparse, and the radius information is continuous and non-sparse. Figure 4 is a schematic diagram of the similarity matrix for corner points and non-corner points. For corner points and non-corner points, their similarity matrices are significantly different.
[0082] Optionally, the first similarity matrix S ρ,θ or / and the second similarity matrix S θ,ρProject in two dimensions of radius and angle respectively, and calculate the sparsity of the angle and radius by using the sparsity evaluation function.
[0083] Preferably, when the radius sparsity is less than or equal to the first threshold and the angle sparsity is greater than or equal to the second threshold, then the pixel point is determined to be a corner point.
[0084] The present invention utilizes the multiply-accumulate model of VSA, where each bit of its high-dimensional vector consists of only -1 and 1, and the calculation only involves bitwise multiplication and addition, which is very friendly to hardware. Especially in low-power devices, it can effectively save computing resources and energy.
[0085] In addition, the present invention utilizes a cyclic basis hypervector to generate a high-dimensional feature extraction kernel, which is more accurate in some scenarios with rotation.
[0086] In order to better illustrate the present invention, numerous specific details are given in the above specific embodiments. Those skilled in the art should understand that the present invention can also be implemented without certain specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail to highlight the gist of the present invention.
[0087] The above is only the specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.
Claims
1. An event-based feature extraction and corner point detection method, characterized in that: The steps include: Step S1, generating an event frame based on the event stream output by the event camera; Step S2: Through the first basis super vector P∈B d and the second basis supervector Θ∈B d The binding operation generates a high-dimensional feature extraction kernel, and uses the high-dimensional feature extraction kernel to perform high-dimensional encoding on the event frame to generate a high-dimensional feature vector of the event frame; Wherein, a base supervector is a group of supervectors, which is the smallest information unit encoded in a high-dimensional space or a hyperspace, d refers to the dimension, B refers to the encoding domain of the multiplication-addition model, and the value of each bit of the vector in the multiplication-addition model is 1 or -1; the binding operation is used to associate information, the first base supervector is a base supervector representing a radius, and the second base supervector is a base supervector representing an angle; Step S3, obtaining a similarity matrix based on an unbinding operation and similarity calculation of the high-dimensional feature vector of the event frame and the first basis supervector or the second basis supervector to perform corner point determination; The unbinding operation is the inverse operation of binding and is used to separate information.
2. The event-based feature extraction and corner detection method according to claim 1, characterized in that: The unbinding operation includes: Unbinding the high-dimensional feature vector of the event frame from the first basis supervector to obtain a second high-dimensional feature vector at each pixel, wherein the second high-dimensional feature vector is a high-dimensional feature vector representing an angle; Or / and, unbinding the high-dimensional feature vector of the event frame from the second basis supervector to obtain a first high-dimensional feature vector at each pixel, wherein the first high-dimensional feature vector is a high-dimensional feature vector of a radius.
3. The event-based feature extraction and corner detection method according to claim 2, characterized in that: The similarity calculation is to calculate the cosine similarity of the vectors.
4. The event-based feature extraction and corner detection method according to claim 3, characterized in that: The similarity calculation includes: Performing similarity calculation on the second high-dimensional feature vector and the second basis super vector to obtain a second similarity matrix; or / and, performing similarity calculation on the first high-dimensional feature vector and the first basis super vector to obtain a first similarity matrix; The first similarity matrix and the second similarity matrix are in a transposed relationship with each other.
5. The event-based feature extraction and corner detection method according to claim 4, characterized in that: Projecting the first similarity matrix or / and the second similarity matrix in two dimensions of angle and radius respectively, and calculating using a sparsity evaluation function to obtain the sparsity of the angle and the sparsity of the radius; When the radius sparsity is less than or equal to a first threshold, and the angle sparsity is greater than or equal to a second threshold, the pixel point is determined to be a corner point.
6. The event-based feature extraction and corner detection method according to any one of claims 1 to 5, characterized in that: The generating of a high-dimensional feature extraction kernel by binding the first basis supervector and the second basis supervector comprises: S211, designating the center point of the high-dimensional feature extraction kernel as the coordinate origin, thereby obtaining the horizontal coordinate and the vertical coordinate (x, y) of each pixel in the high-dimensional feature extraction kernel; S212, using a coordinate conversion formula to convert the horizontal coordinate and the vertical coordinate of each pixel into polar coordinates (ρ, θ) in a polar coordinate system, where ρ represents the radius and θ represents the angle S213, using the first basis supervector and the second basis supervector to encode the radius and the angle respectively; S214, using a binding operation to associate the first basis supervector with the second basis supervector to obtain a high-dimensional feature extraction kernel K∈B N×N×d , where N is a natural number and N×N represents the size of the high-dimensional feature extraction kernel.
7. The event-based feature extraction and corner detection method according to claim 6, characterized in that: The high-dimensional encoding of the event frame using the high-dimensional feature extraction core comprises: Performing a convolution operation on the event frame and the high-dimensional feature extraction kernel to obtain a high-dimensional feature vector of the event frame; All bits greater than 0 in the high-dimensional feature vector of the event frame are assigned a value of 1, bits less than 0 are assigned a value of -1, and bits equal to 0 are randomly assigned a value of 1 or -1.
8. The event-based feature extraction and corner detection method according to claim 7, characterized in that: The event stream within a preset time range is compressed to obtain event frames.