A method for event camera target detection and hardware acceleration based on polarity matching

Abstract

The application relates to a polar matching-based event camera target detection and hardware acceleration method, and belongs to the technical field of brain-like perception and calculation. The method solves the problem that an event camera in the prior art is difficult to provide accurate and efficient image processing due to background events and background noise caused by movement, and comprises the following steps: S1, receiving an original image collected by an event camera, separating a positive polarity event stream and a negative polarity event stream in real time, and establishing a positive and negative event time surface graph; S2, constructing a binary image, obtaining a binary equivalent frame of the binary image based on core set processing; S3, clustering the binary equivalent frame of the binary image, marking and classifying non-zero pixels of the binary equivalent frame, and obtaining a class label frame; S4, clustering the positive polarity event stream and the negative polarity event stream, obtaining a positive event clustering cluster and a negative event clustering cluster; and S5, performing cross-polarity clustering on the positive event clustering cluster and the negative event clustering cluster, and outputting a clustering result.

Description

Technical Field

[0001] This invention relates to the field of neuromorphic perception and computing technology, specifically to an event camera target detection and hardware acceleration method based on polarity matching. Background Technology

[0002] Event cameras are neuromorphic vision sensors inspired by biological vision, offering advantages such as high temporal resolution, high dynamic range (up to 140 dB), and low power consumption. Unlike traditional cameras that capture images at a fixed frame rate, event cameras are sensitive to relative changes in brightness and output asynchronous event streams. They independently encode positive relative increases and negative relative decreases in light intensity, outputting events representing these changes. Therefore, event cameras perform exceptionally well in a variety of challenging applications, such as object detection, pose estimation, motion analysis, and SLAM (Simultaneous Localization and Mapping). Event cameras simulate the imaging mechanism of the human eye by implementing a three-level abstraction of the photoreceptor layer, bipolar cell layer, and ganglion cell layer in the retinal structure.

[0003] Event cameras possess a unique visual perception mechanism that enables them to respond naturally to dynamic changes, making them ideal for high-speed target detection. However, when deployed on drones or intelligent robots, the camera's own motion generates a large number of background events, requiring powerful data processing capabilities and motion compensation to filter them out. When the accumulation time is short, they are susceptible to interference from background noise. High-performance central processing units (CPUs) alone are insufficient to meet these demanding real-time and data throughput requirements.

[0004] Most existing event-based clustering algorithms struggle with post-motion-compensated frame data because traditional pixel clustering fails to capture object motion. Frequent class merging leads to data collisions, complicating efficient parallel acceleration on FPGAs (Field-Programmable Gate Arrays).

[0005] Existing techniques employ density-based noisy application spatial clustering (DBSCAN) to process event frames. However, traditional frame image clustering methods do not fully utilize the unique visual perception capabilities of event cameras. Due to the significant grayscale differences between moving objects and the background, as well as the similarity of their local grayscale values, events of opposite polarities are often generated along the two edges along the direction of the moving object's motion.

[0006] Chinese invention patent application, publication number CN119579644A, entitled "A Non-Line-of-Sight Imaging Detection Method and System for Fusion Event Cameras," discloses an enhanced image sequence obtained and processed by an event camera and an optical camera. However, this method fixes the event camera or reduces vibration interference using a gimbal; it does not provide a solution for the large number of background events caused by the event camera being deployed on a drone or intelligent robot during image processing.

[0007] Chinese invention patent application, publication number CN114663492A, entitled "A Method for Determining Aircraft Attitude Using an Event Camera as a Star Sensor," discloses a method for obtaining event information through an event camera mounted on an aircraft and solving for the camera's absolute and relative pose. However, this method also fails to address the issue of handling the large number of background events caused by the event camera being deployed on a drone or intelligent robot during image processing.

[0008] Therefore, there is a need in this technical field for methods to effectively address the large amount of background noise caused by the motion of event cameras and to improve computational speed. Summary of the Invention

[0009] In view of the above problems, the present invention provides a target detection and hardware acceleration method for event cameras based on polarity matching, which solves the problem that in the prior art, event cameras deployed on drones or intelligent robots cause a large number of background events and a large amount of background noise due to motion, making it difficult to provide accurate and efficient image processing.

[0010] According to an embodiment of the present invention, an event camera target detection and hardware acceleration method based on polarity matching is provided, comprising:

[0011] Step S1: Receive the original images captured by the event camera, separate the positive and negative event streams in real time through the spatiotemporal filtering module, establish a time surface map of positive and negative events, filter high-density core points, and obtain a set of filtered core points, including a set of positive event core points and a set of negative event core points.

[0012] Step S2: Based on the set of core points after filtering, perform time window slicing to obtain the core set, construct a binary image, and process the core set to obtain the binary equivalent frame of the binary image, including the binary equivalent frame of positive events and the binary equivalent frame of negative events.

[0013] Step S3: Cluster the binary equivalent frames of the binary image to obtain the clustering results of non-zero pixels in the binary equivalent frames. Then, use a pipelined dual-scan circuit to scan the binary equivalent frames line by line from the top left to the bottom right to mark and classify the non-zero pixels of the binary equivalent frames and obtain the class label frames.

[0014] Step S4: Dynamically bind positive and negative event streams to class label frames using spatial association to achieve clustering of positive and negative event streams, resulting in positive event clusters and negative event clusters respectively.

[0015] Step S5: Analyze the spatial distribution characteristics of positive event clusters and negative event clusters, perform cross-polarity clustering, and output the clustering results as the target detection results.

[0016] Optionally, step S1 specifically includes:

[0017] Step S1.1: Receive the raw images captured by the event camera, separate the positive and negative event streams through the spatiotemporal filtering module, and construct the time surface maps of the positive and negative events respectively;

[0018] Step S1.2: Use the positive and negative event time surface map to select high-density core points. Specifically, select positive and negative event flows in the positive and negative event time surface map separately to obtain a set of filtered core points, including a set of positive event core points and a set of negative event core points.

[0019] Optionally, selecting high-density core points in step S1.2 specifically includes:

[0020] For each event in the positive polarity event stream in the positive and negative event time surface diagram, if each event satisfies the condition that the difference between the timestamp of at least four adjacent pixels and the timestamp of the event is less than a set time threshold, then the event is classified as a high-density core point, thus obtaining a set of positive polarity event core points.

[0021] For each event in the negative polarity event stream in the positive and negative event time surface diagram, if each event satisfies the condition that the difference between the timestamp of at least four adjacent pixels and the timestamp of the event is less than a set time threshold, then the event is classified as a high-density core point, thus obtaining a set of negative polarity event core points.

[0022] The set of core points of positive polarity events and the set of core points of negative polarity events constitute the set of core points after screening, including all high-density core points;

[0023] Events not classified as high-density core points in the positive and negative event time surface diagram are non-core point events.

[0024] Optionally, step S2 includes performing the following operations on the set of positive polarity event core points and the set of negative polarity event core points in the set of positive and negative core points respectively:

[0025] Step S2.1: Set the time window to 4 milliseconds. From the set of core points of positive polarity events, take the set of events of all core points within a 4-millisecond interval as the core set of positive polarity events, and obtain the pixel coordinates of all core points in the core set of positive polarity events. Set the time window to 4 milliseconds. From the set of core points of negative polarity events, take the set of events of all core points within a 4-millisecond interval as the core set of negative polarity events, and obtain the pixel coordinates of all core points in the core set of negative polarity events. The obtained core set of positive polarity events and core set of negative polarity events constitute the core set within a 4-millisecond interval.

[0026] Step S2.2: First, based on the resolution of the event camera, a binary image with the resolution size of the event camera is formed; then, the binary image is processed based on the core set within a 4-millisecond interval to obtain the binary equivalent frame of the binary image, including the binary equivalent frame of positive events and the binary equivalent frame of negative events.

[0027] Optionally, the processing of the binary image based on the core set within a 4-millisecond interval in S2.2 specifically includes:

[0028] The pixel position of each pixel in the binary image is compared with the pixel coordinate position of the core point in the positive polarity event core set of the core set within a 4-millisecond interval. If the pixel position in the binary image matches the pixel coordinate position of the core point, the pixel position in the binary image is marked as 1, otherwise it is marked as 0. The above operation is performed on all pixels in the binary image to mark all pixel positions, and a positive event binary equivalent frame with the resolution size of the event camera is obtained.

[0029] The pixel position of each pixel in the binary image is compared with the pixel coordinate position of the core point in the negative polarity event core set of the core set within a 4-millisecond interval. If the pixel position in the binary image matches the pixel coordinate position of the core point, the pixel position in the binary image is marked as 1; otherwise, it is marked as 0. The above operation is performed on all pixels in the binary image to mark all pixel positions, thereby obtaining a negative event binary equivalent frame with the resolution size of the event camera.

[0030] The resulting binary equivalent frames for positive and negative events constitute a binary equivalent frame.

[0031] Optionally, step S3 specifically includes performing the following operations for the binary equivalent frames of positive and negative events respectively:

[0032] Step S3.1: Perform the first row scan on the binary equivalent frame and assign an initial class label to all non-zero pixels. Each pixel has three neighborhoods, including the first neighborhood on the top, the second neighborhood on the left, and the third neighborhood on the right and bottom. If a non-zero pixel has other non-zero pixels in its first and second neighborhoods, the initial class label of the non-zero pixel is set to the smallest label among these non-zero pixels, and the joint lookup table is updated. Otherwise, a new incremental label is assigned to the non-zero pixel.

[0033] Step S3.2: Perform a second scan on the region containing the core point in the binary equivalent frame. Generate a class label frame according to the joint lookup table. The value of the class label frame is the root label of the class label corresponding to the pixel in the binary equivalent frame.

[0034] Optionally, step S3.1 specifically includes:

[0035] Neighborhood pixel judgment: Read each group of pixels from the top left to the bottom right of the binary equivalent frame, and check whether there are non-zero pixels in the first and second neighborhoods of each non-zero pixel. Perform processing (1) on pixels that do not have non-zero pixels in the first and second neighborhoods, and perform processing (2) on all other pixels.

[0036] (1) Based on the current maximum class label, assign the initial class label of the non-zero pixel to the new incremental label, and update the maximum class label value;

[0037] (2) Assign the initial class label of the non-zero pixel to the smallest class label of the non-zero pixel in the neighborhood, and update the joint lookup table.

[0038] Optionally, in step S3, eight consecutive pixels in the same row are grouped together and represented by an 8-bit number so that the pixel group containing eight pixels can be read and processed at the same time during line-by-line scanning.

[0039] Optionally, step S4 specifically includes:

[0040] All events within a preset neighborhood of each core point in the set of positive polarity event core points in the positive and negative event time surface diagram are assigned the same class label as the core point, resulting in a positive event cluster that includes positive events.

[0041] All events within a preset neighborhood of each core point in the set of core points of negative polarity events in the positive and negative event time surface diagram are assigned the same class label as the core point, resulting in a negative event cluster that includes classes of negative events.

[0042] Event clusters are composed of positive event clusters and negative event clusters.

[0043] Optionally, step S5 specifically includes:

[0044] For positive event clusters and negative event clusters, classes containing fewer than 10 events are filtered out to obtain filtered positive event clusters and filtered negative event clusters.

[0045] The classes of positive events in the filtered positive event clusters are sequentially matched with the classes of all negative events in the filtered negative event clusters. If the class distance between the positive event class and the negative event class is less than the set class distance threshold, the matching requirement is met, and the positive event class and the negative event class are merged; otherwise, the matching requirement is not met, and the positive event class and the negative event class whose class distance does not meet the class distance threshold requirement are discarded.

[0046] Iterate through the filtered positive event clusters, perform the above operation on all positive event classes, and take all the merged positive event classes and negative event classes as the clustering result, and output them as the target detection result.

[0047] Compared with the prior art, the event camera target detection and hardware acceleration method based on polarity matching provided by the present invention has at least the following beneficial effects.

[0048] (1) Use binary encoding with a bit width and other categories to encode all child nodes associated with the same root node, thereby achieving efficient merging operation.

[0049] (2) The original image is downsampled by 1 / 16 to improve computation speed and reduce resource consumption. Processing positive and negative event frames sequentially requires only 5,496 clock cycles.

[0050] (3) This invention provides a two-stage polarity-based spatiotemporal clustering algorithm and a pipeline scanning hardware data stream for event stream processing, so as to accelerate the clustering process and alleviate the problem of low efficiency in traditional clustering. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the accompanying drawings. The accompanying drawings are schematic and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the hardware accelerator principle used for clustering in an event camera target detection and hardware acceleration method based on polarity matching according to an embodiment of the present invention.

[0053] Figures 2a to 2d A schematic diagram of the clustering process is shown in an embodiment of an event camera target detection and hardware acceleration method based on polarity matching provided according to an embodiment of the present invention.

[0054] Figure 3 In an embodiment of the event camera target detection and hardware acceleration method based on polarity matching provided according to an embodiment of the present invention, the detection results on the recorded dataset are shown.

[0055] Figure 4 This is a schematic diagram comparing the detection results of an event camera target detection and hardware acceleration method based on polarity matching provided according to an embodiment of the present invention with those of existing technologies on the shape_6DOF dataset. Detailed Implementation

[0056] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0057] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein. Therefore, the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0058] The following detailed description, with reference to the accompanying drawings, illustrates an event camera target detection and hardware acceleration method based on polarity matching according to an embodiment of the present invention.

[0059] like Figure 1 As shown, an event camera target detection and hardware acceleration method based on polarity matching according to an embodiment of the present invention includes the following steps.

[0060] Step S1: Receive the raw images captured by the event camera, perform bipolar event stream separation and core point extraction. A dedicated hardware module (spatiotemporal filtering module) separates the positive and negative event streams in real time, establishing a time surface map of positive and negative events. High-density core points are quickly filtered to obtain a set of filtered core points. The event camera can be deployed on a drone or a mobile intelligent robot. In this step, the raw images captured by the event camera can be downsampled by 1 / 16 to improve computational speed and reduce resource consumption. The core points are high-density core points, also known as core event points, which are points with high spatiotemporal density.

[0061] Step S1.1: Receive the raw images captured by the event camera, separate the positive and negative event streams using a spatiotemporal filtering module, and construct positive and negative event time surface maps (TS) respectively, storing the timestamps of the most recently generated positive and negative events for each pixel. For example... Figure 2a This schematically illustrates the time surface diagram of positive and negative events, representing the original event frames.

[0062] In this step, the original image captured by the event camera can be downsampled by 1 / 16 and then the positive and negative event streams can be separated to improve computing speed and reduce resource consumption.

[0063] Step S1.2: For each event in the positive and negative event streams of the positive and negative event time surface diagrams, make a separate judgment. If each event satisfies the condition that the difference between the timestamp of at least four adjacent pixels and the timestamp of the event is less than a set time threshold, then the event is classified as a high-density core point. All the high-density core points obtained constitute the set of core points after filtering, including the set of core points of positive events and the set of core points of negative events.

[0064] High-density core points are selected using a positive and negative event time surface plot (TS), as shown in the following expression:

[0065]

[0066] in, It is the timestamp of the current event. This indicates the current event in TS. The timestamp values ​​of adjacent pixels Indicates the first event of the current event Adjacent pixels, Indicates the index of the neighboring pixel of the current event. This indicates the set time threshold. This is an indicator function that equals 1 when the condition is met, and 0 otherwise.

[0067] In this embodiment, each current event in the positive and negative event timeline is configured to have 8 neighboring pixels, and therefore, the 8 neighboring pixels are compared and compared for judgment. It should be understood that in other implementations, there may be more or fewer adjacent pixels for the current event, depending on the requirements.

[0068] For positive and negative event time surface plots (TS) that satisfy The point represents a point that, for the current event, has at least four adjacent pixels that satisfy the condition. The point of the current event is then set. These are categorized as high-density core points (i.e., core event points). The core point is numbered. Events that do not meet the above conditions and are not classified as high-density core points in the positive and negative event time surface diagram are non-core point events.

[0069] In this step, each event in the positive polarity event stream of the positive and negative event time surface map separated in step S1.1 is judged. If each event satisfies the condition that the difference between the timestamp of at least four adjacent pixels and the timestamp of the event is less than a set time threshold, then the event is classified as a high-density core point, thus obtaining a set of positive polarity event core points. For each event in the negative polarity event stream of the positive and negative event time surface map, if an event satisfies the condition that the difference between the timestamps of at least four adjacent pixels and the event's timestamp is less than a set time threshold, then the event is classified as a high-density core point, thus obtaining a set of negative polarity event core points. The set of core points of positive polarity events and the set of core points of negative polarity events constitute the set of core points after screening, including all high-density core points.

[0070] Step S2: Based on the filtered set of core points, perform time window slicing to obtain a core set, construct a binary image, and process the core set to obtain binary equivalent frames of the binary image, including binary equivalent frames for positive events and binary equivalent frames for negative events. Specifically, based on the filtered set of core points, perform time window accumulation, including using a time window slicing mechanism to accumulate the core points filtered in Step S1, forming two binary equivalent frames, one positive and one negative. Step S2 performs the same operation on the set of positive polarity event core points and the set of negative polarity event core points in the filtered set of core points, specifically including the following steps.

[0071] Step S2.1: Set the time window to 4 milliseconds. From the filtered set of core points, that is, from the set of core points of positive polarity events or the set of core points of negative polarity events, take the set of all core point events within a 4-millisecond interval, represented as: , as the core set. Among them, To retrieve the first core from the core set within a 4-millisecond interval The location of the core point Represents the index of the core point in the core set within a 4-millisecond interval and , This represents the total number of core points in the core set within a 4-millisecond interval. Indicates the first The pixel coordinates of each core point are obtained. This gives the pixel coordinates of all core points in the core set.

[0072] Specifically, in this step, the time window is set to 4 milliseconds. From the set of core points of positive polarity events, the set of events of all core points within a 4-millisecond interval is taken as the core set of positive polarity events, and the pixel coordinate positions of all core points in the core set of positive polarity events are obtained. The time window is set to 4 milliseconds. From the set of core points of negative polarity events, the set of events of all core points within a 4-millisecond interval is taken as the core set of negative polarity events, and the pixel coordinate positions of all core points in the core set of negative polarity events are obtained. The core set of positive polarity events and the core set of negative polarity events constitute the core set within a 4-millisecond interval.

[0073] Step S2.2: First, construct a sheet of size... The binary image is then processed based on a core set within a 4-millisecond interval to obtain equivalent frames of the binary image, including positive event binary equivalent frames and negative event binary equivalent frames. This refers to the resolution of the event camera.

[0074] Specifically for sizes of Given a binary image, the pixel position of each pixel is compared with the core set within a 4-millisecond interval. The pixel coordinates of the core point are compared. If a pixel in the binary image matches the pixel coordinates of the core point, that pixel is marked as 1; otherwise, it is marked as 0. In this step, all pixels in the binary image are traversed and all pixel positions are marked, thus obtaining a value of [size missing]. The binary equivalent frames are used as the equivalent frames of the binary image. For the binary image, the positive polarity event core set and the negative polarity event core set obtained in step S2.1 are processed respectively to obtain the positive event binary equivalent frames and the negative event binary equivalent frames.

[0075] The pixel position of each pixel in the binary image and the core set within a 4-millisecond interval are determined in the following way. The pixel coordinates of the core points are compared and marked:

[0076]

[0077] in, This represents the coordinate position of a pixel in a binary image. A binary equivalent frame representing a pixel.

[0078] The above formula indicates that if the core point set obtained in step S2.1 contains a pixel coordinate that corresponds to the pixel coordinate of the binary image... The core of the matching is the pixels of the binary image. The coordinates of the mismatched pixels are marked as 1, while the coordinates of all mismatched pixels are marked as 0.

[0079] After traversing the pixel positions of all pixels in the binary image and marking all pixel positions, a group of sizes is formed from all the marks. The binary equivalent frame.

[0080] In this step, all pixels of the binary image are first matched with the positive polarity event core set of the core set to obtain a positive event binary equivalent frame; then, all pixels of the binary image are matched with the negative polarity event core set of the core set to obtain a negative event binary equivalent frame. The obtained positive event binary equivalent frames and negative event binary equivalent frames constitute a binary equivalent frame.

[0081] Step S3: Design a dedicated hardware circuit to cluster the binary equivalent frames (including positive event binary equivalent frames and negative event binary equivalent frames) obtained in step S2, obtain the clustering results of non-zero pixels in the binary equivalent frames, and perform line-by-line scanning of the binary equivalent frames from the top left to the bottom right through a pipelined dual-scan circuit. Based on the non-zero pixels in the eight neighborhoods of each pixel, realize the fast labeling and classification of non-zero pixels in the binary equivalent frames.

[0082] In this step, to improve the bandwidth of frame scan data reading, a row-by-row concatenation data storage method is used, that is, eight consecutive pixels in the same row are grouped into a pixel group and represented by an 8-bit number. This allows a group of eight pixels to be read and processed at the same time during line-by-line scanning.

[0083] Optionally, such as Figure 1 As shown, based on the positional relationship between neighboring pixels and the current pixel, the neighboring pixels are divided into three neighborhoods: the first neighborhood (the upper neighbor on the top side), the second neighborhood (the left neighbor on the left side), and the third neighborhood (the right and lower right neighbors). This implementation divides the entire calculation process into 5 processing units (PE1-PE5) and implements pipelined processing between different processing units. Step S3 specifically includes the following steps. Since the processing of positive event binary equivalent frames and negative event binary equivalent frames is the same, the following description only uses positive event binary equivalent frames as an example.

[0084] Step S3.1: Perform an initial row scan on the binary equivalent frame and assign an initial class label to all non-zero pixels. If a non-zero pixel has other non-zero pixels in its first and second neighborhoods, set the initial class label of the non-zero pixel to the smallest label among these non-zero pixels; otherwise, assign a new incremental label to the non-zero pixel. This step specifically includes the following processing procedures.

[0085] First, neighboring pixels are judged. Each pixel is read from the top left to the bottom right of the binary equivalent frame (for spliced ​​group pixels, each pixel group is read). PE1 and PE2 are used to detect whether there are non-zero pixels in the first and second neighboring regions of each non-zero pixel. For pixels that do not have non-zero pixels in the first and second neighboring regions, process (1) is performed; otherwise, process (2) is performed.

[0086] (1) Assign the initial class label of the non-zero pixel to the new incremental label. Specifically, first obtain the current maximum class label. Assign a new class label value to the non-zero pixel. And update the maximum class label value. . This is the preset label increment value for each pixel.

[0087] For a pixel group consisting of 8 pixels, count the number of pixels in the group that does not contain any non-zero pixels that need to be assigned a new class label. The label increment value of these pixels is From front to back, they are: Assign new class label values ​​to each core point At the same time, update the maximum class label value to This method supports parallel processing of 8 pixels in the same group that are stitched together.

[0088] (2) Assign the initial class label of the non-zero pixel to the smallest class label of the non-zero pixel in the neighborhood, and update the joint lookup table.

[0089] For a pixel group consisting of 8 pixels, the same operation is performed in parallel for each group of pixels that need to obtain the minimum class label in this step. First, obtain the class label set S of all non-zero pixels in the first and second neighborhoods of the current pixel group, and set the initial class label of the current pixel group to the minimum value in the class label set S.

[0090] Meanwhile, since these class labels in the class label set S should eventually be the same, a joint lookup table is constructed. That is, an array A[a] is used to store the root labels of all class labels, where a represents the largest class label. For example, A[5]=1 means that the root label of the largest class label 5 is 1. At the same time, an array B[b] is used to store all class labels under the root label b. For example, B[1]=8'b01010001 means that the root labels of labels 1, 5, and 7 are all 1.

[0091] The specific update steps of the joint lookup table are as follows: For all class labels in S, first find the set of all root labels of these class labels through A[a]. Then find the set of root labels through B[b]. The collection of all class tags under each root tag in . ,renew The root tag of all class tags in is The smallest label in the set is the preset label value. Simultaneously update for This results in an updated join lookup table.

[0092] Step S3.2: Perform a second scan on the region containing the core point in the binary equivalent frame, and generate a class label frame based on the joint lookup table obtained in step S3.1.3. And make the class label frame The value is the root label of the class label corresponding to the pixel in the binary equivalent frame.

[0093] Step S4: Dynamically bind the original event stream (including positive and negative event streams) received in step S1 to the class label frame generated in step S3.2 using spatial association. This process achieves complete clustering of the original event stream, resulting in event clusters, including positive event clusters and negative event clusters. In this step, spatial association is used to dynamically bind non-core event points to nearby core points, thus achieving complete event clustering.

[0094] Perform the following operations on the sets of core points for positive and negative events in the time surface diagram of positive and negative events, respectively. Complete event clustering includes selecting core points from the core set within the 4-millisecond interval obtained in step S2. neighborhood Non-core events within Assigning core points Same class tag:

[0095] .

[0096] in, Indicate the core point Non-core events in the neighborhood With core points Class spacing, Indicate the core point An index of non-core events within the neighborhood. Indicate the core point The preset neighborhood outer diameter. Indicates from class label frame The core points found in Class tags, The set of core points of positive polarity events Or a set of core points of negative polarity events This is obtained from step S1.2.

[0097] Through the above processing, the core points in the core set within a 4-millisecond interval are... Clustering non-core events within their neighborhood yields event classes, which can be either positive or negative events. Then, all classes of positive events are grouped into positive event clusters, and all classes of negative events are grouped into negative event clusters. For example... Figure 2b and Figure 2c The positive event cluster and the negative event cluster are schematically shown respectively.

[0098] Specifically, in this step, for the set of core points of positive polarity events in the positive and negative event time surface map, all non-core events in the neighborhood of each core point in the core set of positive polarity events within a 4-millisecond interval are assigned the same class label as the core point, resulting in a positive event cluster including classes of positive events; for the set of core points of negative polarity events in the positive and negative event time surface map, all non-core events in the neighborhood of each core point in the core set of negative polarity events within a 4-millisecond interval are assigned the same class label as the core point, resulting in a negative event cluster including classes of negative events; the resulting positive event cluster and negative event cluster constitute an event cluster.

[0099] All common events and key points Performing the above operations yields event clusters, including positive event clusters and negative event clusters. Ordinary events are non-core events in the positive and negative event streams obtained from the original image received in step S1.

[0100] Step S5: Analyze the spatial distribution characteristics of the positive and negative event clusters, perform cross-polarity clustering, and output the merged clustering results as the target detection results. See also... Figure 2d This is a schematic diagram of the target detection results after the output is merged, which is the final clustering result.

[0101] First, for both positive and negative event clusters, classes containing fewer than 10 events are filtered out, resulting in filtered positive and negative event clusters. This process removes noise interference.

[0102] Next, the classes of each positive event in the filtered positive event cluster are matched sequentially with the classes of all negative events in the filtered negative event cluster. If the class distance between the positive event class and the negative event class is less than the set class distance threshold, the matching requirement is met, and the positive event class and the negative event class are merged; otherwise, the matching requirement is not met, and the positive event class and the negative event class whose class distance does not meet the class distance threshold requirement are discarded.

[0103] Finally, iterate through the filtered positive event clusters, perform the above operations on all positive event classes, and take all the merged positive event classes and negative event classes as the clustering results, and output them as the target detection results.

[0104] The final target detection results can be used in fields such as autonomous driving, robot control, and industrial automation to provide subsequent processing such as analysis and prediction, obstacle warning, fault feedback, and precise control.

[0105] Example 1

[0106] To facilitate understanding of the polarity matching-based event camera target detection and hardware acceleration method provided by the embodiments of the present invention, the following describes Example 1 of applying this embodiment.

[0107] In terms of hardware, in this embodiment, the object detection algorithm of the present invention is implemented on the AVNET Ultra96 development board, which is supported by the Xilinx Zynq UltraScale+ MPSoC ZU3EG. This chip integrates a 64-bit quad-core ARM Cortex-A53 processor and programmable logic units. Its programmable logic units include 154,000 system logic units, comprising 70,560 lookup tables (LUTs), 141,120 flip-flops (FFs), and 432 × 18k block random access memory (BRAMs) with 7.6 Mb of memory and 360 digital signal processor (DSP) units. In this embodiment, high-level synthesis (HLS) is used to complete the final design.

[0108] In this embodiment 1, to simulate a real-world drone operating scenario, a DAVIS346 event camera was mounted on a quadcopter, and a basketball was thrown in front of the drone. Nine datasets of this type of scenario were recorded to evaluate the performance of the algorithm of this invention. Furthermore, the performance of the proposed clustering algorithm in object detection tasks was quantitatively evaluated using the Shapes_6DOF dataset (a 6-DOF tracking synthetic image dataset). The Shapes_6DOF dataset uses a DAVIS346 event camera to capture synchronized frames and event data of multiple regularly shaped moving objects. To establish quantitative evaluation metrics, the true location of moving objects was extracted from high-fidelity frame sequences using a morphological edge detection algorithm, enabling a systematic comparison of the precision and recall of different methods.

[0109] Figure 3The detection sequences in the dataset validate the effectiveness of Example 1, which applies the algorithm of this invention, in dynamic aviation scenes, as shown in the dataset. The results demonstrate that the model can accurately locate moving targets while suppressing false alarms caused by complex background events resulting from the self-motion of the event camera.

[0110] To provide a more comprehensive quantitative evaluation of the clustering algorithm, it was compared with existing object detection methods during camera ego motion in the Shapes_DOF dataset (a dataset of synthesized images with degrees of freedom). Object detection was performed using the DBSCAN (density-based clustering method with noise) and eCluster (event-by-event clustering) algorithms. The results are shown in Table 1. Figure 4 As shown. Compared with traditional methods, the polarity-driven clustering framework proposed in Example 1 exhibits superior performance, improving detection accuracy by 14% while maintaining comparable recall.

[0111] Table 1 Comparison of detection accuracy of the three methods

[0112]

[0113] The processing speed of Example 1, which applies the method of the present invention, was compared and analyzed in terms of computational throughput between an FPGA-accelerated implementation and a CPU-only execution of the complete algorithm. The processing speeds of each component are shown in Table 2. Compared to the pure software implementation running on an Arm Cortex-A53 processor in Ultra96, the heterogeneous computing paradigm of Example 1 shows an overall speedup of 19.93 times. The limited speedup of denoising in Example 1 is related to sequential dependency constraints, while the motion compensation accelerator extensively benefits from the massive parallelism achieved through spatial architecture optimization.

[0114] Table 2 Processing speed of each component

[0115]

[0116] Table 3 shows the resource consumption per module reported by Xilinx Vivado in Example 1. DSPs are mainly used for the multiplication and accumulation operations required in Gaussian filtering. Xilinx Vivado reports that the logic module consumes approximately 0.1158W, demonstrating the low power consumption advantage of the clustering processing in the method of Example 1.

[0117] Table 3 Comparison of resource consumption and power consumption in Example 1

[0118]

[0119] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0120] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0121] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for target detection and hardware acceleration using an event camera based on polarity matching, characterized in that, include: Step S1: Receive the original images captured by the event camera, separate the positive and negative event streams in real time through the spatiotemporal filtering module, establish a time surface map of positive and negative events, filter high-density core points, and obtain a set of filtered core points, including a set of positive event core points and a set of negative event core points. Step S2: Based on the set of core points after filtering, perform time window slicing to obtain the core set, construct a binary image, and process the core set to obtain the binary equivalent frame of the binary image, including the binary equivalent frame of positive events and the binary equivalent frame of negative events. Step S3: Cluster the binary equivalent frames of the binary image to obtain the clustering results of non-zero pixels in the binary equivalent frames. Then, use a pipelined dual-scan circuit to scan the binary equivalent frames line by line from the top left to the bottom right to mark and classify the non-zero pixels of the binary equivalent frames and obtain the class label frames. Step S4: Dynamically bind positive and negative event streams to class label frames using spatial association to achieve clustering of positive and negative event streams, resulting in positive event clusters and negative event clusters respectively. Step S5: Analyze the spatial distribution characteristics of positive event clusters and negative event clusters, perform cross-polarity clustering, and output the clustering results as the target detection results.

2. The event camera target detection and hardware acceleration method based on polarity matching according to claim 1, characterized in that, Step S1 specifically includes: Step S1.1: Receive the raw images captured by the event camera, separate the positive and negative event streams through the spatiotemporal filtering module, and construct the time surface maps of the positive and negative events respectively; Step S1.2: Use the positive and negative event time surface map to select high-density core points. Specifically, select positive and negative event flows in the positive and negative event time surface map separately to obtain a set of filtered core points, including a set of positive event core points and a set of negative event core points.

3. The event camera target detection and hardware acceleration method based on polarity matching according to claim 2, characterized in that, Step S1.2, which selects high-density core points, specifically includes: For each event in the positive polarity event stream in the positive and negative event time surface diagram, if each event satisfies the condition that the difference between the timestamp of at least four adjacent pixels and the timestamp of the event is less than a set time threshold, then the event is classified as a high-density core point, thus obtaining a set of positive polarity event core points. For each event in the negative polarity event stream in the positive and negative event time surface diagram, if each event satisfies the condition that the difference between the timestamp of at least four adjacent pixels and the timestamp of the event is less than a set time threshold, then the event is classified as a high-density core point, thus obtaining a set of negative polarity event core points. The set of core points of positive polarity events and the set of core points of negative polarity events constitute the set of core points after screening, including all high-density core points; Events not classified as high-density core points in the positive and negative event time surface diagram are non-core point events.

4. The event camera target detection and hardware acceleration method based on polarity matching according to claim 1, characterized in that, Step S2 includes performing the following operations on the set of positive polarity event core points and the set of negative polarity event core points in the positive and negative core point sets respectively: Step S2.1: Set the time window to 4 milliseconds. From the set of core points of positive polarity events, take the set of events of all core points within a 4-millisecond interval as the core set of positive polarity events, and obtain the pixel coordinates of all core points in the core set of positive polarity events. Set the time window to 4 milliseconds. From the set of core points of negative polarity events, take the set of events of all core points within a 4-millisecond interval as the core set of negative polarity events, and obtain the pixel coordinates of all core points in the core set of negative polarity events. The obtained core set of positive polarity events and core set of negative polarity events constitute the core set within a 4-millisecond interval. Step S2.2: First, based on the resolution of the event camera, a binary image with the resolution size of the event camera is formed; then, the binary image is processed based on the core set within a 4-millisecond interval to obtain the binary equivalent frame of the binary image, including the binary equivalent frame of positive events and the binary equivalent frame of negative events.

5. The event camera target detection and hardware acceleration method based on polarity matching according to claim 1, characterized in that, The processing of the binary image based on the core set within a 4-millisecond interval in S2.2 specifically includes: The pixel position of each pixel in the binary image is compared with the pixel coordinate position of the core point in the positive polarity event core set of the core set within a 4-millisecond interval. If the pixel position in the binary image matches the pixel coordinate position of the core point, the pixel position in the binary image is marked as 1, otherwise it is marked as 0. The above operation is performed on all pixels in the binary image to mark all pixel positions, and a positive event binary equivalent frame with the resolution size of the event camera is obtained. The pixel position of each pixel in the binary image is compared with the pixel coordinate position of the core point in the negative polarity event core set of the core set within a 4-millisecond interval. If the pixel position in the binary image matches the pixel coordinate position of the core point, the pixel position in the binary image is marked as 1; otherwise, it is marked as 0. The above operation is performed on all pixels in the binary image to mark all pixel positions, thereby obtaining a negative event binary equivalent frame with the resolution size of the event camera. The resulting binary equivalent frames for positive and negative events constitute a binary equivalent frame.

6. The event camera target detection and hardware acceleration method based on polarity matching according to claim 1, characterized in that, Step S3 specifically includes performing the following operations for the binary equivalent frames of positive and negative events respectively: Step S3.1: Perform the first row scan on the binary equivalent frame and assign an initial class label to all non-zero pixels, where each pixel has three neighborhoods, including the first neighborhood on the top, the second neighborhood on the left, and the third neighborhood on the right and bottom. If a non-zero pixel has other non-zero pixels in its first and second neighborhoods, then the initial class label of the non-zero pixel is set to the smallest label among these non-zero pixels, and the joint lookup table is updated. Otherwise, assign a new incremental label to the non-zero pixel. Step S3.2: Perform a second scan on the region containing the core point in the binary equivalent frame. Generate a class label frame according to the joint lookup table. The value of the class label frame is the root label of the class label corresponding to the pixel in the binary equivalent frame.

7. The event camera target detection and hardware acceleration method based on polarity matching according to claim 6, characterized in that, Step S3.1 specifically includes: Neighborhood pixel judgment: Read each group of pixels from the top left to the bottom right of the binary equivalent frame, and check whether there are non-zero pixels in the first and second neighborhoods of each non-zero pixel. Perform processing (1) on pixels that do not have non-zero pixels in the first and second neighborhoods, and perform processing (2) on all other pixels. (1) Based on the current maximum class label, assign the initial class label of the non-zero pixel to the new incremental label, and update the maximum class label value; (2) Assign the initial class label of the non-zero pixel to the smallest class label of the non-zero pixel in the neighborhood, and update the joint lookup table.

8. The event camera target detection and hardware acceleration method based on polarity matching according to claim 1, characterized in that, In step S3, eight consecutive pixels in the same row are grouped together and represented by an 8-bit number so that the pixel group containing eight pixels can be read and processed at the same time during line-by-line scanning.

9. The event camera target detection and hardware acceleration method based on polarity matching according to claim 1, characterized in that, Step S4 specifically includes: All events within a preset neighborhood of each core point in the set of positive polarity event core points in the positive and negative event time surface diagram are assigned the same class label as the core point, resulting in a positive event cluster that includes positive events. All events within a preset neighborhood of each core point in the set of core points of negative polarity events in the positive and negative event time surface diagram are assigned the same class label as the core point, resulting in a negative event cluster that includes classes of negative events. Event clusters are composed of positive event clusters and negative event clusters.

10. The event camera target detection and hardware acceleration method based on polarity matching according to claim 1, characterized in that, Step S5 specifically includes: For positive event clusters and negative event clusters, classes containing fewer than 10 events are filtered out to obtain filtered positive event clusters and filtered negative event clusters. The classes of positive events in the filtered positive event clusters are sequentially matched with the classes of all negative events in the filtered negative event clusters. If the class distance between the positive event class and the negative event class is less than the set class distance threshold, the matching requirement is met, and the positive event class and the negative event class are merged; otherwise, the matching requirement is not met, and the positive event class and the negative event class whose class distance does not meet the class distance threshold requirement are discarded. Iterate through the filtered positive event clusters, perform the above operation on all positive event classes, and take all the merged positive event classes and negative event classes as the clustering result, and output them as the target detection result.

Images