Event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision

Abstract

The invention belongs to the technical field of artificial intelligence chips and reconfigurable computing, and particularly relates to an event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision. Aiming at the problems of low storage utilization rate, obvious data access bottleneck, limited degree of parallelism, too high calculation redundancy and the like of a traditional GNN accelerator in an event vision scene, the invention provides a novel architecture combining efficient graph feature storage, hierarchical graph construction and redundancy elimination convolution calculation. By introducing parallel read-write optimization, low-dependence graph structure generation and a reusable computing cache mechanism, the computing delay is remarkably reduced while resource consumption is kept controllable, the overall throughput rate is increased, and therefore the sub-microsecond real-time reasoning capacity is achieved. The method is suitable for unmanned driving, intelligent monitoring, robot navigation and other application scenes needing low-delay visual calculation on edge equipment.

Description

Technical Field

[0001] This invention relates to an event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision, belonging to the field of artificial intelligence chips and reconfigurable computing technology. Background Technology

[0002] Event-based vision, with its microsecond-level temporal resolution and sparse data output, has shown significant advantages in real-time vision scenarios such as autonomous driving, robotics, and high-speed monitoring. Event-driven asynchronous graph neural networks (GNNs) can capture the fine spatiotemporal dependencies between events through dynamic graph structure construction and message-passing-based convolution, making them an important method for achieving real-time, robust edge vision. They are more suitable for asynchronous and sparse event stream processing requirements than traditional frame-based deep learning models ([1][2]). At the hardware level, FPGAs, as reconfigurable computing platforms, are widely used in GNN accelerator research due to their advantages such as strong parallelism, customizable data paths, and low power consumption. Various FPGA GNN acceleration schemes, such as SpGCN ([3]), FP-GNN ([4]), and Dynasparse ([5]), have improved inference performance by optimizing sparse computation modes, pipeline structures, and parallel memory access. However, they are generally based on static graphs or batch graphs and cannot adapt to the continuously updated dynamic graph structure in event-driven scenarios. For event-driven visual scenarios, some hardware research has also emerged for event-driven GNNs. For example, EFGCN ([6]) significantly reduces the graph size through 3D pooling and can deploy deep networks on FPGAs; EvGNN ([7]) realizes event-level local subgraph construction and asynchronous convolution, and significantly reduces end-to-end latency. These studies have laid the foundation for accelerating event-driven GNNs, but there are still key bottlenecks such as low memory access efficiency, insufficient parallelism, large computational redundancy, and difficulty in reducing latency to sub-microsecond levels.

[0003] References [1] G. Gallego, T. Delbr¨uck, G. Orchard, C. Bartolozzi, B. Taba, A.Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis et al.,“Event-based vision: A survey,” IEEE transactions on pattern analysis andmachine intelligence, vol. 44, no. 1, pp. 154–180, 2020. [2] P. Lichtsteiner, C. Posch, and T. Delbruck, “A 128× 128 120 db15 μs latency asynchronous temporal contrast vision sensor,” IEEE Journal ofSolid-State Circuits, vol. 43, no. 2, pp. 566–576, 2008. [3] X. Xu, Q. Liu, W. Huang, W. Peng, and Y. Huang, “Spgcn: An fpga-based graph convolutional network accelerator for sparse graphs,” in 2024IEEE 32nd Annual International Symposium on Field-Programmable CustomComputing Machines (FCCM). IEEE, 2024, pp. 216–216. [4] T. Tian, L. Zhao, X. Wang, Q. Wu, W. Yuan, and X. Jin, “Fp-gnn:adaptive fpga accelerator for graph neural networks,” Future GenerationComputer Systems, vol. 136, pp. 294–310, 2022. [5] B. Zhang and V. Prasanna, “Dynasparse: Accelerating gnn inference through dynamic sparsity exploitation,” in 2023 IEEE International Paralleland Distributed Processing Symposium (IPDPS). IEEE, 2023, pp. 233–244. [6] K. Jeziorek, P. Wzorek, K. Blachut, A. Pinna, and T. Kryjak, “Embedded graph convolutional networks for real-time event data processing onsoc fpgas,” arXiv preprint arXiv:2406.07318, 2024. [7] Y. Yang, A. Kneip, and C. Frenkel, “Evgnn: An event-driven graphneural network accelerator for edge vision,” IEEE Transactions on Circuitsand Systems for Artificial Intelligence, vol. 2, no. 1, pp. 37–50, 2025. Summary of the Invention The purpose of this invention is to provide an event-driven asynchronous GNN FPGA acceleration solution that can be optimized in terms of storage structure, parallel graph construction and convolution redundancy elimination, and supports sub-microsecond event processing.

[0004] To achieve the above objectives, the technical solution of this invention discloses an event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision, characterized in that it includes a graph construction module, a graph feature storage module, and a graph convolution module: When the event camera outputs event ev=(x,y,t,p), the graph construction module retrieves historical events in parallel within a preset spatial window and generates a neighbor buffer in combination with time constraints to form a local subgraph, which is then stored in the graph feature storage module. Here, (x,y) are the spatial coordinates of the event node, t is the timestamp, and p is the polarity. The graph convolution module reads neighbor node features or cached feature items in parallel from the graph feature storage module based on the neighbor buffer, completes message generation, aggregation and activation update, and obtains the target node update feature. While the target node update feature outputs the inference result through the readout layer and the fully connected layer, the graph convolution module writes the target node update feature back to the graph feature storage module for reuse in subsequent events.

[0005] Preferably, the graph features generated by the graph construction module and the graph convolution module are stored in the graph feature storage module using a graph feature storage mechanism, which includes: The graph features are stored using a fine-grained spatial tiling mapping mechanism: taking the spatial coordinates (x, y) of the event node as input, the graph features are divided into repeated spatial blocks of a fixed size in two-dimensional space, and the positions in different spatial blocks are mapped to the storage blocks of different graph feature storage modules according to periodic rules, so as to ensure that the candidate features in any neighborhood search window are distributed and stored in multiple storage blocks. A depth-addressing and compressed storage mechanism is adopted: In the time dimension, only the latest historical events and corresponding features are retained for each spatial location; in the spatial dimension, a sparse-aware compressed storage method is adopted, and each storage block is organized into a first-in-first-out queue of features written in the order of event arrival, storing only valid event features; an index buffer is set to record the depth index of the compressed queue corresponding to each spatial location. When it is necessary to read the historical features of a certain spatial location, the graph feature storage module first reads the index buffer in parallel to obtain the depth address of the current spatial location in the storage block, and then accesses the corresponding storage block and reads the feature data according to the obtained depth address.

[0006] Preferably, based on the fine-grained spatial tiling mapping mechanism, when the graph construction module or the graph convolution module reads the historical event features within a certain local window, it generates the corresponding storage block number and storage block address in parallel according to the coordinates of each position within the local window, and synchronously reads the candidate features from multiple storage blocks within the same clock cycle.

[0007] Preferably, the local window is a rectangular neighborhood search window, and the width and height of the rectangular neighborhood search window are aligned with the arrangement of the storage blocks.

[0008] Preferably, the graph construction module adopts a low-dependency hierarchical graph construction mechanism, completing neighborhood retrieval and the construction of the local subgraph at each event ev=(x,y,t,p), including: When a new event ev=(x,y,t,p) arrives, a fixed-size spatial neighborhood search window is established with spatial coordinates (x,y) as the center. Through the fine-grained spatial tiling mapping mechanism, the candidate events in the spatial neighborhood search window are distributed and stored in different storage blocks. In a single cycle, the graph construction module can read candidate event information from multiple storage blocks in parallel and write the candidate event information into the neighbor buffer. During the process of writing candidate event information into the neighbor buffer, time selection and valid event filtering are completed simultaneously: the time difference dt=t-τ is calculated for each candidate event, and the threshold MAX_DT is used to determine whether the current candidate event meets the temporal correlation condition. At the same time, invalid events are filtered by combining the valid bit. Only candidate events that meet both the validity constraint and the time constraint are written into the neighbor buffer, thereby obtaining the neighbor set of the current event node.

[0009] Preferably, the graph construction module adopts a hierarchical writing strategy, dividing the neighbor buffer into multiple writing levels and pre-allocating slots: high-priority slots support direct parallel writing, medium-priority slots use round-robin allocation to achieve partial parallel writing, and low-priority slots use sequential padding to ensure writing integrity.

[0010] Preferably, the graph convolution module uses a non-redundant parallel graph convolution mechanism to obtain the target node update features, including: the graph convolution module simultaneously reads cached feature items of multiple neighbor nodes based on the valid neighbor node indices recorded in the neighbor buffer, using the multi-storage block parallel read mechanism of the graph feature storage module. And compute the corresponding geometric terms in parallel. Add the two together to generate a multi-path neighborhood message. A pipelined aggregation structure is used to process multi-path neighborhood messages. Perform aggregation operations, and then sequentially perform bias addition, quantization, and activation function processing on the aggregation result to obtain the updated feature vector of the target event node. .

[0011] Preferably, the neighborhood message The generation process is divided into two parts: neighbor feature terms and geometric terms, so that the neighborhood message Represented as Among them: neighbor feature items Determined solely by the characteristics of neighboring nodes, represented as It is unrelated to the target node. The learnable weight matrix for the feature terms; geometric terms Determined by the relative positional characteristics of the target node and its neighboring nodes, and changing with the target node. Let be the learnable weight matrix for the geometric terms.

[0012] Preferably, the graph convolution module updates the feature vector of the target node after completing the k-th convolution layer. Then, at the tail end of the pipeline, the feature terms required for the target node to be used as neighbor nodes in the next convolutional layer are immediately pre-computed. and feature terms The feature terms are written to the feature term cache of the graph feature storage module; when a neighbor node j is used in subsequent event convolution calculations, the graph convolution module directly reads the corresponding neighbor feature terms from the feature term cache. As the neighborhood message Generate input, while only considering geometric terms that change with the target node. Perform online calculations and and The neighborhood message is obtained by adding them together. .

[0013] Compared to existing technologies, this invention achieves an event processing latency of only 0.58 μs on the Xilinx ZCU102 platform, representing an average improvement of 27.59 times over the latest event-driven GNN accelerators, realizing true sub-microsecond real-time inference. Simultaneously, while maintaining comparable hardware resource consumption, the recognition accuracy is further improved. Thanks to highly parallel graph feature storage, low-dependency hierarchical graph construction, and a non-redundant parallel convolution mechanism, this invention maintains stable throughput and high energy efficiency even under continuous event streams, significantly outperforming existing synchronous or semi-asynchronous solutions. Furthermore, this invention is well-suited to the resource constraints of edge platforms, combining high performance, low latency, and high deployability, fully meeting the needs of real-time vision applications such as autonomous driving, intelligent monitoring, and robot navigation. Attached Figure Description

[0014] Figure 1This invention provides a system overview of the event-driven asynchronous graph neural network processing flow. The system takes the asynchronous event stream output by the event camera as input and processes each arriving event instantly without waiting for frames or fixed batches. The system consists of three asynchronous, collaborative stages: graph storage, graph construction, and graph convolution. The graph storage stage caches historical events and their graph features using a unified spatiotemporal representation. It improves local window readout efficiency through rectangular neighborhood windows and fine-grained storage mapping, and reduces storage resource consumption by retaining only the latest historical events and using sparse-aware compressed storage. The graph construction stage reads historical events from the graph storage, filters neighbors based on their spatiotemporal relationships, and forms a neighbor set. Spatial selection employs parallel reading and row / column shifting to reduce rearrangement overhead, while temporal selection uses a hierarchical filtering strategy based on distance priority to balance throughput and resources between parallel and serial processing. The graph convolution stage uses the neighbor set and historical graph features as input to generate and aggregate neighborhood messages to update target node features. Convolution computation uses parallel processing of neighbor features to reduce computation time, pre-computes and caches reusable messages at the output, and reads and reuses them at the input to eliminate redundant computation. The three phases constitute an event-driven end-to-end asynchronous data flow, enabling low-latency, high-throughput inference for real-time edge vision.

[0015] Figure 2 As an example of spatial memory mapping, the figure shows: (a) illustrates coarse-grained spatial tiling mapping: coarse-grained spatial tiling mapping maps features of adjacent spatial locations to different depths of the same storage block, causing port conflicts during local window reading and requiring multiple cycles to complete; (b) illustrates fine-grained spatial tiling mapping: fine-grained spatial tiling mapping divides spatial locations into periodically repeating spatial blocks, so that features within any local window are distributed and stored in different storage blocks, thereby supporting parallel reading of multiple storage blocks.

[0016] Figure 3 This is an example of a spatial neighborhood search window. Irregular neighborhood windows, such as circular or rhomboid ones, result in port idleness and bandwidth waste when mapped to a regular storage array, leading to effective bandwidth being lower than the allocated bandwidth. This invention employs a regular rectangular neighborhood search window, strictly aligned with the storage ports, increasing bandwidth utilization from a minimum of approximately 51% to 100%. Simultaneously, the rectangular window's coverage area is consistent with that of the rhomboid window, thereby improving storage bandwidth utilization without losing neighborhood information.

[0017] Figure 4For an example of time depth compression for each spatial location, the figure shows: (a) a fixed-depth time first-in-first-out (T-FIFO) cache: the traditional fixed-depth time cache stores multiple entries of historical events for the same spatial location, causing the storage depth to increase cumulatively with the queue depth; (b) a time cache that only retains the latest events: the present invention adopts a caching strategy that only retains the latest historical events to avoid the accumulation of time redundancy, thereby significantly reducing the storage depth required for each spatial location.

[0018] Figure 5 For an example of time-depth compression for each BRAM block, the figure shows: (a) a fixed address storage structure: the fixed address storage method leads to a large number of idle storage units under the condition of sparse event space; (b) a sparse-aware compressed storage and index-assisted access structure: the present invention adopts a sparse-aware compressed storage method, organizes each storage block into a feature first-in-first-out queue written in the order of event arrival, saves only valid features, and combines an index buffer to record the queue depth index corresponding to each spatial position, realizing a two-step read path of "parallel index lookup - parallel feature retrieval", thereby improving storage utilization while avoiding the high latency caused by full queue scanning.

[0019] Figure 6 As an example of spatial selection strategy, the figure shows: (a) a serial time-division multiplexing spatial selection structure: serial processing uses time-division multiplexing to read neighborhood events, which has low resource overhead but high access latency; (b) a fully connected parallel spatial selection structure: fully parallel processing uses fully connected mapping to achieve fast reading, but requires a large number of multiplexers, resulting in excessive resource overhead; (c) a row-column shift two-step parallel spatial selection: this invention proposes a parallel two-step spatial selection strategy, first reading candidate events in parallel according to the physical port order, and then using the row-column cyclic mapping characteristics to complete the logical order rearrangement through row shift and column shift, thereby achieving low-latency parallel neighborhood reading under resource-constrained conditions.

[0020] Figure 7As an example of the time selection strategy, the figure shows: (a) illustrates the time selection priority: Candidate events are divided into high priority, medium priority, and low priority according to their spatial distance from the central event within the neighborhood window; among them, there are more medium priority candidate events and the writing slots are limited, so a round-robin allocation strategy is adopted to cyclically offset the selection of medium priority positions, that is, each time a group of adjacent positions are processed in parallel and the selection starting point is moved in a preset direction in the next round, so that each medium priority position gets an equal screening opportunity, thereby avoiding position bias and improving the time screening coverage; (b) illustrates serial time selection: Traditional serial writing requires dynamic calculation of the write address to avoid conflicts, resulting in serialization and high latency; (c) illustrates hierarchical time selection: This invention proposes a hierarchical time selection strategy, which utilizes the address independence of the neighbor buffer, uses fixed slots for conflict-free parallel writing of high priority candidate events, uses a round-robin method for partially parallel writing of medium priority candidate events, and uses serial traversal combined with a priority encoder to dynamically locate empty slots for low priority candidate events, thereby improving the parallelism of time screening and reducing the write dependency chain.

[0021] Figure 8 As an example of a graph convolutional architecture, the figure shows: (a) a sequential graph convolutional computation architecture: existing pipelined convolutional architectures use time-division multiplexing to perform message generation, aggregation, and bias-quantization-activation processing on a neighbor-by-neighbor basis, which has low resource consumption but limited parallelism; (b) a parallel graph convolutional computation architecture: parallel convolutional architectures use parallel processing in the message generation, aggregation, and bias-quantization-activation stages to reduce computation time, but have higher resource consumption; (c) a convolutional reordering computation architecture: this invention reorders the graph convolutional computation flow based on the parallel convolutional architecture, transferring the feature transformation terms that were originally repeatedly calculated on neighbor nodes in the message generation stage to the convolutional output for pre-computation, and using the pre-computed results Feature terms are written to a cache so that they can be directly read and reused at the input of the next layer of convolution, thereby reducing repeated multiplication and addition operations and multiplier overhead. At the same time, to avoid bit-width expansion caused by pre-computed feature terms in the cache, this invention introduces a quantization compression module at the feature term output to compress high-bit-width intermediate results into low-bit-width representations to reduce storage and bandwidth requirements. In addition, this invention implements geometric term calculation and bias-quantization-activation processing using a lookup table, reducing multiplication and addition operations and logic resource consumption, and enables pipelined reuse of the aggregation module in the output channel dimension, allowing the aggregation comparison tree and post-processing unit to be reused across channels in a time-division manner without reducing overall throughput, thereby significantly reducing hardware resource overhead while maintaining high throughput of parallel convolution.

[0022] Figure 9The figure shows an example of the computation process of reordered graph convolution. (a) illustrates the traditional graph convolution computation process: the traditional graph convolution computation process repeatedly transforms and calculates the features of neighbor nodes during the message generation stage, resulting in high computational redundancy in the shared neighbor scenario; (b) illustrates the reordered graph convolution computation process: the present invention pre-calculates and caches the neighbor feature terms required for the next layer at the convolution output end, and directly reads the cached feature terms at the convolution input end and only calculates the geometric terms online to complete message generation, thereby realizing feature term reuse and significantly reducing repeated calculations (by about 94%). Detailed Implementation

[0023] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0024] This invention discloses an event-driven asynchronous graph neural network (GNN) FPGA accelerator for real-time edge vision. Addressing the problems of low storage utilization, significant data access bottlenecks, limited parallelism, and excessive computational redundancy in traditional GNN accelerators for event vision scenarios, this invention proposes a novel architecture combining efficient graph feature storage, hierarchical graph construction, and redundancy-eliminating convolutional computation. By introducing parallel read / write optimization, low-dependency graph structure generation, and a reusable computation caching mechanism, this invention significantly reduces computational latency and improves overall throughput while maintaining controllable resource consumption, thereby achieving sub-microsecond real-time inference capabilities. This makes it suitable for applications requiring low-latency visual computation on edge devices, such as autonomous driving, intelligent monitoring, and robot navigation.

[0025] The FPGA accelerator disclosed in this invention is event-driven. When the event camera outputs an event ev=(x,y,t,p), the graph construction module first retrieves historical events in parallel within a preset spatial window and generates a neighbor buffer NB based on time constraints to form a local subgraph. Here, (x,y) are the spatial coordinates of the event node, t is the timestamp, and p is the polarity. Subsequently, the graph convolution module reads neighbor node features or cached feature items in parallel from the graph feature storage module based on the neighbor buffer NB, completes message generation, aggregation, and activation update, obtains the updated features of the target node, and writes the updated features back to the graph feature storage module for reuse in subsequent events. Finally, the inference result is output through the readout layer and the fully connected layer at the last layer of the network, thus forming an end-to-end closed-loop data flow of "event input—local graph construction—feature reading—convolution update—write back for reuse—result output". Based on this overall data flow, the embodiments of the present invention achieve efficient acceleration for event vision scenarios through efficient graph feature storage, low-dependency hierarchical graph construction, and parallel redundancy elimination convolution calculation techniques, solving the problems of low storage utilization, obvious data access bottlenecks, limited parallelism, and excessive computational redundancy in the prior art.

[0026] 1) High-parallelism graph feature storage Graph construction and the graph features generated after graph convolution need to be efficiently stored and readily available for retrieval. To achieve this goal, embodiments of the present invention design an efficient graph feature storage mechanism, specifically including the following aspects: 1) Storage Block Selection and Addressing: To improve the parallelism of neighborhood window access and reduce storage conflicts, this invention adopts a fine-grained spatial tiling mapping mechanism for graph feature storage. Using the spatial coordinates (x, y) of event nodes as input, the graph features are divided into repeating spatial blocks of fixed size in two-dimensional space. The positions in different spatial blocks are mapped to different storage blocks according to periodic rules, ensuring that candidate features in any neighborhood search window can be distributed across multiple storage blocks. Based on this mapping method, when the graph construction module or graph convolution module needs to read historical event features within a local window, it can generate corresponding storage block numbers and addresses in parallel based on the coordinates of each position within the window. Candidate features are then synchronously read from multiple storage blocks within the same clock cycle, avoiding the serial access bottleneck caused by a large amount of data concentrated in the same storage block under traditional mapping methods. Meanwhile, to further reduce port idleness and bandwidth waste caused by irregular windows, this invention adopts a rectangular neighborhood search window and aligns its width and height with the storage block arrangement, thereby improving the effective bandwidth utilization during the window readout process and bringing the storage bandwidth close to full-load working state, thereby reducing the feature access latency in the local mapping and convolution stages.

[0027] 2) Intra-block depth addressing: To reduce the storage depth occupied by graph features in the time and spatial dimensions, this invention proposes a depth addressing and compressed storage mechanism: In the time dimension, for each spatial location, only its latest historical events and corresponding features are retained, replacing the traditional fixed-depth time series queue storage method that continuously accumulates on the time axis, thereby significantly reducing the feature storage depth required for each spatial location; In the spatial dimension, considering the highly sparse nature of the event stream, this invention adopts a sparse-aware compressed storage method, organizing each storage block into a first-in-first-out feature queue written according to the arrival order of events, storing only valid event features, avoiding the waste of depth caused by reserving storage space for a large number of pixel locations without events. To ensure rapid location and parallel reading of neighborhood features even under compressed storage, this invention further sets up an index buffer to record the compressed queue depth index corresponding to each spatial location. When it is necessary to read the historical features of a certain spatial location, the graph feature storage module first reads the index buffer in parallel to obtain the depth address of that location within the storage block, and then accesses the corresponding storage block and reads the feature data accordingly. This achieves a balance between "compressed storage reducing depth occupancy" and "parallel access ensuring bandwidth", improving storage resource utilization and reducing memory access latency.

[0028] (ii) Construction of low-dependency hierarchical graph One of the key steps in event-driven asynchronous GNNs is to quickly construct a local neighborhood subgraph for each arriving event, i.e., to determine the adjacency relationship between the current event node and historical event nodes. However, in existing technologies, graph construction often requires sequential scanning or dynamic index updates, resulting in long control dependency chains, making parallelization difficult and causing high latency, which cannot meet the requirements of sub-microsecond event processing. To address this, this invention proposes a low-dependency hierarchical graph construction mechanism, enabling neighborhood retrieval and subgraph construction to be completed within a very short period after each event arrives. The technical solution includes: 1) Parallel Acquisition of Local Spatial Window and Candidate Events: When a new event ev=(x,y,t,p) arrives, this embodiment of the invention establishes a fixed-size spatial neighborhood search window (such as a rectangular window) centered on its spatial coordinates (x,y). Through the aforementioned row-column cyclic mapping mechanism, candidate events within the spatial window are distributed and stored in different storage blocks. The graph construction module can read candidate event information (including valid bit, timestamp τ, polarity p, etc.) from multiple storage blocks in parallel within a single cycle and write it into the local event window cache, thereby achieving high-throughput parallel extraction of candidate events.

[0029] 2) Time Selection and Valid Event Filtering: During the writing of candidate events into the neighbor buffer NB, this embodiment of the invention simultaneously performs time selection and valid event filtering: For each candidate event, the time difference dt = t - τ is calculated (where t is the timestamp of the current target event (or the current reference event), τ is the event timestamp of the candidate event, and dt represents the time interval between the candidate event and the current target event, used to characterize the timeliness of the candidate event), and it is determined whether it meets the time-series association condition based on the threshold MAX_DT. Simultaneously, invalid events are filtered using the valid bit. Only candidate events that simultaneously meet the validity constraint and the time constraint are written into the neighbor buffer NB, thus obtaining the neighbor set of the current event node. To avoid the control dependency and write conflicts caused by traditional dynamic allocation of write positions, this invention adopts a hierarchical write strategy, dividing the neighbor buffer into multiple write levels and pre-allocating slots: high-priority slots support direct parallel writing, medium-priority slots use round-robin allocation to achieve partial parallel writing, and low-priority slots use sequential padding to ensure write integrity. This strategy significantly shortens the write dependency chain, improves the parallelism of time filtering and neighbor writing, and thus avoids the serial bottleneck caused by traditional sequential traversal.

[0030] (iii) Redundant Parallel Graph Convolution The core step of event-driven asynchronous graph neural network inference is graph convolution computation. Its goal is to aggregate neighborhood features and update the feature vector of the target event node based on the local subgraph structure formed by the current event node and its neighboring nodes. In existing event GNN FPGA convolution implementations, feature transformations are repeatedly computed due to neighbor sharing, and convolution computation and memory access suffer from long dependency chains and bandwidth bottlenecks, resulting in limited overall parallelism and high inference latency. Therefore, this invention proposes a non-redundant parallel graph convolution computation mechanism to reduce redundant computations and improve computational throughput, thereby meeting the real-time inference requirements of event vision scenarios. The technical solution includes: 1) Parallel Pipeline Graph Convolution Computation Structure: The graph convolution module reads the cached feature items of multiple neighbor nodes simultaneously using the multi-block parallel read mechanism of the graph feature storage module, based on the valid neighbor node indices recorded in the neighbor buffer NB. And compute the corresponding geometric terms in parallel. Add the two together to generate a multi-path neighborhood message. Subsequently, the aggregation unit uses a pipelined aggregation structure to perform aggregation operations (such as maximum value aggregation or summation aggregation) on multiple messages, and sequentially performs bias addition, quantization, and activation function processing on the aggregation result to obtain the updated feature vector of the target event node. Updated feature vectors On the one hand, the feature storage module writes back to the graph for reuse in subsequent event convolutions. On the other hand, the inference results are aggregated and sent to the fully connected module through the readout layer in the last layer of the network, thereby realizing continuous, low-latency, and high-throughput parallel convolution pipeline execution of the event stream.

[0031] 2) Convolution computation decomposition: To reduce redundant computations in the convolution message generation stage, this invention decomposes the neighborhood message generation process into two parts: neighbor feature terms and geometric terms, so that the message can be represented as... Among them: neighbor feature items Determined solely by the characteristics of neighboring nodes, it can be represented as This term is independent of the target node, so it will be calculated repeatedly when different target nodes share the same neighbor node, which is the main source of convolution redundancy. The learnable weight matrix for the feature terms; geometric terms Based on the relative positional characteristics of the target node and its neighboring nodes (such as...) , A decision can be expressed as This parameter must be calculated in real time as it changes with the target node. This represents the learnable weight matrix for the geometric terms. Through this decomposition method, embodiments of the present invention can separate reusable neighbor feature terms from geometric terms that must be computed online, providing a foundation for subsequent feature term caching and reuse, and redundant parallel convolution computation.

[0032] 3) Feature item cache reuse: To eliminate neighbor feature items To avoid repeated multiplication and addition calculations, this embodiment of the invention employs a convolution calculation reordering and feature term cache reuse mechanism: after completing the k-th convolution and obtaining the target node, the feature vector is updated. Subsequently, the graph convolution module immediately pre-computes the features required for the target node to serve as a neighbor node in the next convolutional layer at the tail end of the pipeline. and feature terms The feature terms are written to the feature term cache of the graph feature storage module; when a neighbor node j is needed in subsequent event convolution calculations, the convolution module directly reads the corresponding neighbor feature terms from the feature term cache. As input for message generation, no repetition is required. The multiplication and addition operations are performed only on geometric terms that vary with the target node. Perform online calculations and and Adding them together yields neighborhood messages. This allows the feature transformation terms of the same neighbor node to be calculated only once and reused by multiple target nodes, achieving redundant computation in the convolutional message generation stage. This significantly reduces repeated multiplication and addition operations on neighbor feature terms, improves the effective utilization of parallel computing units, reduces convolutional inference latency, and enhances overall throughput performance.

[0033] The designed event-driven asynchronous GNN accelerator was deployed on the Xilinx ZCU102 MPSoC platform, and the design and synthesis were completed using Vitis HLS 2022.2 and Vivado 2022.2. The system operating frequency was 200 MHz. In this embodiment of the invention, the event stream of the N-CARS dataset was pre-compiled to generate a static event graph for GNN training. 15,422 samples were divided into training and validation sets at 85% and 15% respectively. After training, inference evaluation was performed on the test set in the form of a dynamic event graph. Experimental results show that the above-described scheme disclosed in this embodiment of the invention achieves a single-event processing latency of 0.58 μs on the ZCU102 platform, achieving an average speedup of 27.59× compared to state-of-the-art event-driven GNN accelerators, while maintaining comparable resource consumption and achieving higher recognition accuracy. This verifies the efficiency and deployability of the technical solution disclosed in this embodiment of the invention in real-time event vision scenarios.

Claims

1. An event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision, characterized in that, The system includes a graph construction module, a graph feature storage module, and a graph convolution module. When the event camera outputs an event ev=(x,y,t,p), the graph construction module retrieves historical events in parallel within a preset spatial window and generates a neighbor buffer based on time constraints to form a local subgraph, which is then stored in the graph feature storage module. Here, (x,y) represents the spatial coordinates of the event node, t is the timestamp, and p is the polarity. The graph convolution module reads neighbor node features or cached feature items in parallel from the graph feature storage module based on the neighbor buffer, completes message generation, aggregation, and activation update, and obtains the target node update features. While the target node update features output the inference results through the readout layer and the fully connected layer, the graph convolution module writes the target node update features back to the graph feature storage module for reuse in subsequent events.

2. The event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision as described in claim 1, characterized in that, The graph features generated by the graph construction module and the graph convolution module are stored in the graph feature storage module using a graph feature storage mechanism. This mechanism includes: storing the graph features using a fine-grained spatial tiling mapping mechanism: taking the spatial coordinates (x, y) of the event node as input, dividing the graph features into repeating spatial blocks of a fixed size in two-dimensional space, and mapping the positions in different spatial blocks to storage blocks of different graph feature storage modules according to periodic rules, ensuring that candidate features in any neighborhood search window are distributed across multiple storage blocks; and employing a depth-addressing and compressed storage mechanism. In the time dimension, only the latest historical events and corresponding features are retained for each spatial location; in the spatial dimension, a sparse-aware compression storage method is adopted, and each storage block is organized into a first-in-first-out queue of features written in the order of event arrival, storing only valid event features; an index buffer is set to record the compression queue depth index corresponding to each spatial location. When it is necessary to read the historical features of a certain spatial location, the graph feature storage module first reads the index buffer in parallel to obtain the depth address of the current spatial location in the storage block, and then accesses the corresponding storage block and reads the feature data according to the obtained depth address.

3. The event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision as described in claim 2, characterized in that, Based on the fine-grained spatial tiling mapping mechanism, when the graph construction module or the graph convolution module reads the historical event features within a local window, it generates the corresponding storage block number and storage block address in parallel according to the coordinates of each position within the local window, and synchronously reads the candidate features from multiple storage blocks within the same clock cycle.

4. The event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision as described in claim 3, characterized in that, The local window adopts a rectangular neighborhood search window, and the width and height of the rectangular neighborhood search window are aligned with the arrangement of the storage blocks.

5. An event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision as described in claim 2, characterized in that, The graph construction module employs a low-dependency hierarchical graph construction mechanism. Upon the arrival of each event ev=(x,y,t,p), neighborhood retrieval and the construction of the local subgraph are completed. This includes: when a new event ev=(x,y,t,p) arrives, a fixed-size spatial neighborhood search window is established centered on the spatial coordinates (x,y). Through the fine-grained spatial tiling mapping mechanism, candidate events within the spatial neighborhood search window are distributed and stored in different storage blocks. Within a single cycle, the graph construction module can read candidate event information from multiple storage blocks in parallel and write the candidate event information into the neighbor buffer. During the process of writing candidate event information into the neighbor buffer, time selection and valid event filtering are simultaneously completed: the time difference dt=t-τ is calculated for each candidate event, and the threshold MAX_DT is used to determine whether the current candidate event meets the temporal correlation condition. Simultaneously, invalid events are filtered using the valid bit, and only candidate events that simultaneously meet the validity constraint and time constraint are written into the neighbor buffer, thereby obtaining the neighbor set of the current event node.

6. The event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision as described in claim 5, characterized in that, The graph construction module adopts a hierarchical writing strategy, which divides the neighbor buffer into multiple writing levels and pre-allocates slots: high-priority slots support direct parallel writing, medium-priority slots use round-robin allocation to achieve partial parallel writing, and low-priority slots use sequential padding to ensure writing integrity.

7. The event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision as described in claim 1, characterized in that, The graph convolution module uses a non-redundant parallel graph convolution mechanism to obtain the target node update features, including: the graph convolution module simultaneously reads cached feature items of multiple neighbor nodes based on the valid neighbor node indices recorded in the neighbor buffer, using the multi-storage block parallel read mechanism of the graph feature storage module. And compute the corresponding geometric terms in parallel. Add the two together to generate a multi-path neighborhood message. A pipelined aggregation structure is used to process multi-path neighborhood messages. Perform aggregation operations, and then sequentially perform bias addition, quantization, and activation function processing on the aggregation result to obtain the updated feature vector of the target event node. .

8. The event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision as described in claim 7, characterized in that, The neighborhood message The generation process is divided into two parts: neighbor feature terms and geometric terms, so that the neighborhood message Represented as Among them: neighbor feature items Determined solely by the characteristics of neighboring nodes, represented as It is unrelated to the target node. The learnable weight matrix for the feature terms; geometric terms Determined by the relative positional characteristics of the target node and its neighboring nodes, and changing with the target node. Let be the learnable weight matrix for the geometric terms.

9. An event-driven asynchronous graph neural network FPGA accelerator for real-time edge vision as described in claim 7, characterized in that, The graph convolution module completes the k-th convolution layer and obtains the updated feature vector of the target node. Then, at the tail end of the pipeline, the feature terms required for the target node to be used as neighbor nodes in the next convolutional layer are immediately pre-computed. and feature terms The feature terms are written to the feature term cache of the graph feature storage module; when a neighbor node j is used in subsequent event convolution calculations, the graph convolution module directly reads the corresponding neighbor feature terms from the feature term cache. As the neighborhood message Generate input, while only considering geometric terms that change with the target node. Perform online calculations and and The neighborhood message is obtained by adding them together. .

Images