Lightweight target trajectory prediction method based on twin-pulse neural network

Abstract

This invention discloses a lightweight target trajectory prediction method based on a Siamese spiking neural network, belonging to the field of target trajectory prediction. The invention includes: constructing sample pairs comprising template frames and search frames; constructing a Siamese spiking neural network model comprising a template branch, a search branch, an adaptive masking module, a spiking neural Transformer module, and an output prediction layer, wherein the template branch and the search branch share weights; the adaptive masking module is used to align the output features of the two branches and generate masks and feature enhancements; the spiking neural Transformer module is used to obtain the global dependencies of the enhanced features and outputs the current position of the target and the endpoint of the tracking path through the output prediction layer to generate the target's tracking plan path; training the constructed model based on the training data constructed from the sample pairs to obtain a model for obtaining the target's tracking plan path. This invention achieves a dual improvement in accuracy and efficiency by dynamically selecting key pulses and optimizing the network structure.

Description

Technical Field

[0001] This invention belongs to the field of target trajectory prediction, specifically relating to a lightweight target trajectory prediction method based on Siamese spiking neural networks. Background Technology

[0002] Target trajectory prediction is of paramount importance in the fields of autonomous driving and robotics. Although significant progress has been made in methods based on traditional artificial neural networks, these methods typically involve more than 100 million model parameters and rely on computationally intensive multiply-accumulate (MAC) operations, which severely hinders their deployment on edge devices.

[0003] Inspired by the neural mechanisms of the human brain, spiking neural networks (SNNs) hold promise as a low-power alternative to traditional artificial neural networks. However, current applications of SNNs still face two major challenges: first, a performance gap exists between them and traditional artificial neural networks; and second, their training costs are high, making it difficult to balance accuracy and efficiency.

[0004] Event cameras possess excellent temporal resolution, dynamic range, energy efficiency, and pixel bandwidth, while spiking neural networks (SNNs) are naturally suited to event data generated by event cameras due to their ability to transmit information through discrete pulse signals. Therefore, combining the two makes them an ideal choice for event-based target trajectory prediction tasks. However, existing research often employs a hybrid architecture of "spiking neural network + traditional artificial neural network," which neither fully leverages the low-power advantage of spiking neural networks nor overcomes performance bottlenecks. Furthermore, the unoptimized network structure further reduces energy efficiency and limits accuracy.

[0005] Current research has applied spiking neural networks to event-based target trajectory prediction tasks and has made some progress. However, these studies have obvious limitations: on the one hand, most adopt a hybrid architecture of "spiking neural network + traditional artificial neural network", which does not make full use of the inherent energy efficiency advantage of spiking neural network; on the other hand, the network structure design lacks targeted optimization. For example, fixed template matching cannot adapt to changes in target scale, and redundant pulse transmission leads to excessive computational load, ultimately limiting the balance between trajectory prediction performance and efficiency. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a lightweight target trajectory prediction method based on Siamese spiking neural networks. This method is a lightweight target trajectory prediction scheme based on Siamese neural networks and dynamic masking mechanisms. By dynamically selecting key pulses and optimizing the network structure, it achieves a dual improvement in accuracy and efficiency.

[0007] The technical solution adopted in this invention is a lightweight target trajectory prediction method based on Siamese spiking neural networks, which includes the following steps:

[0008] Step 1, construct sample pairs;

[0009] Using keyframe sequences of spatiotemporally non-uniformly sampled data acquired by an event camera as the input data source, a dynamic random sampling strategy is employed to generate sample pairs within a preset time window; each sample includes a template frame and a search frame.

[0010] Step 2: Construct a Siamese spiking neural network model, which includes two weight-shared template branches and a search branch. The template and search branches have identical network structures, employing a lightweight backbone network based on spiking neural convolutional blocks. The template branch receives template frames from sample pairs as input, used to extract template features. The search branch receives search frames from sample pairs as input, used to extract search features. An adaptive masking module is connected after the template and search branches to achieve feature alignment and mask generation between template and search features. Based on the generated mask, the search features are adaptively enhanced, and the enhanced features are output to the spiking neural Transformer module. The spiking neural Transformer module obtains global dependencies in the feature maps corresponding to the enhanced features based on spiking neural convolutional blocks and a self-attention mechanism, calculating correlations using pulse timestamp information. Finally, the current position of the target and the endpoint of the tracking path are output through the output prediction layer to generate the target's tracking plan path.

[0011] Step 3: Construct a training dataset based on the sample pairs generated in Step 1, train the model parameters of the Siamese spiking neural network model, and obtain the Siamese spiking neural network model for obtaining the tracking plan path of the target when the preset training convergence conditions are met (such as the number of training times reaching the preset upper limit or the training loss function value converging).

[0012] Furthermore, the template frame is selected from the keyframe sequence that meets the motion edge features with preset conditions, while the search frame is located after the template frame. Randomly selected within a time interval; where, This is the default value.

[0013] Furthermore, the template frame is selected as the first frame in the keyframe sequence.

[0014] Furthermore, the lightweight backbone network includes a first stacked structure consisting of several first stacked modules cascaded together, each consisting of a downsampling layer, a first stacking unit, and a second stacking unit. Each first stacking unit includes three spiking neural convolutional blocks, and each second stacking unit includes two spiking neural convolutional blocks. Each spiking neural convolutional block sequentially includes a LIF neuron layer, a grouped convolutional layer, and a batch normalization layer. In the first stacking unit, a residual connection is established between the output of the first spiking neural convolutional block and the output of the downsampling layer of the first stacking unit, and a residual connection is established between two adjacent spiking neural convolutional blocks. In the second stacking unit, a residual connection is established between two spiking neural convolutional blocks.

[0015] Furthermore, the kernel size of the grouped convolutional layers in the spiking neural convolutional block is set to 3×3.

[0016] Furthermore, the adaptive masking module includes two first downsampling layers, which are used to process template features and second downsampling layers are used to process search features;

[0017] Following the first downsampling layer are the following layers in sequence: average pooling layer, connection layer, first convolutional layer, first batch normalization layer, ReLU activation function, LIF neuron layer, second convolutional layer, second batch normalization layer, Sigmoid activation function, and adaptive enhancement layer; and the search features processed by the second downsampling layer. Directly fed into the connection layer and adaptive enhancement layer;

[0018] The two downsampling layers use grouped convolution, and the kernel size of both convolutional layers is set to 1×1.

[0019] The connection layer adjusts the feature dimension of the template features after average pooling to match the spatial dimension of the downsampled search features, thus aligning the template and search features spatially. Then, the template and search features are concatenated along the channel dimension. The concatenated features are then fed into the first convolutional layer, sequentially passing through the first batch of normalization layers, ReLU activation function, LIF neuron layer, second convolutional layer, second batch of normalization layers, and a sigmoid activation function to generate a mask that distinguishes the target from the background. ;

[0020] Adaptive enhancement layer based on mask and learnable parameters Perform feature enhancement and output enhanced features:

[0021]

[0022] in, Indicates enhanced features, It represents the Hadamardi (or Hadama) stack.

[0023] Furthermore, the mask In the diagram, 1 corresponds to the effective target area, and 0 corresponds to the redundant background area.

[0024] Furthermore, learnable parameters for: ,in, Indicates learnable weights, This represents the Sigmoid activation function.

[0025] Furthermore, the spiking neural Transformer module sequentially includes a second stacked structure consisting of several cascaded second stacked modules, each composed of a downsampling layer, a third stacked unit, a self-crossing attention module, and a fourth stacked unit. The third stacked unit includes three spiking neural convolutional blocks, and the second stacked unit includes two spiking neural convolutional blocks. Each spiking neural convolutional block sequentially includes a LIF neuron layer, a grouped convolutional layer, and a batch normalization layer. In the third stacked unit, a residual connection is established between the output of the first spiking neural convolutional block and the output of the downsampling layer of the first stacked unit, and a residual connection is established between adjacent spiking neural convolutional blocks. In the fourth stacked unit, a residual connection is established between two spiking neural convolutional blocks.

[0026] The technical solution provided by this invention brings at least the following beneficial effects:

[0027] This invention integrates an event camera, a spiking neural network, and a Siamese network, using addition-only operations to replace traditional multiplication-accumulation calculations, thus aligning with the sparse characteristics of event data. Furthermore, this invention introduces a dynamic masking mechanism to suppress background noise in real time; simultaneously, it employs grouped convolution techniques to compress the model size, achieving a lightweight design.

[0028] In this invention, features of template data and search data are extracted based on a constructed lightweight backbone network (such as a lightweight grouped convolution extraction backbone network), providing high-quality input for subsequent feature processing. To address the issue of spiking neural networks being susceptible to redundant pulse interference, high-frequency responses (such as edge and texture features) in the target region are dynamically selected to suppress background noise. Simultaneously, the region of interest can be adaptively adjusted according to changes in target scale and occlusion (such as expanding / shrinking the mask range), overcoming the robustness deficiencies of traditional fixed template matching. Furthermore, the input channel is divided into multiple independent groups through grouped convolution, with each group performing convolution operations separately, reducing the number of parameters and computational load. Finally, combined with the discretized feature transfer characteristics of pulse coding, dual "space-time" compression is achieved, further improving energy efficiency. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the structure of the twin spiking neural network model used in an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the lightweight backbone network used in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the adaptive mask module used in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the structure of the pulse nerve Transformer module used in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0035] This invention provides a lightweight target trajectory prediction method based on Siamese spiking neural networks (Siamese network architecture, a Siamese network structure using shared weights), an event camera, and a spiking neural network. The event camera, with its microsecond-level temporal resolution, can dynamically perceive scene changes and exhibits superior performance in scenes where traditional frame-based cameras perform poorly due to motion blur. The spiking neural network, with its sparse pulse (0 / 1) computational paradigm, naturally aligns with the sparse nature of event data: replacing multiplication and accumulation operations with only addition operations reduces energy consumption to levels far below those of traditional artificial neural networks. This synergistic computational effect, combined with the low-power characteristics of the event camera itself, lays a solid foundation for building a high-energy-efficiency trajectory prediction system. The Siamese network structure reduces the number of parameters through a weight-sharing mechanism and forces the learning of cross-modal universal feature representations to further improve the model's generalization ability, especially when the event data distribution shifts. A tracking plan path is generated based on the target's current position and the endpoint of the tracking path from the corner prediction output.

[0036] The weight sharing in Siamese network structures not only significantly reduces the number of model parameters but also enables the model to learn generalized feature representations of input pairs, thereby achieving efficient similarity measurement in spatiotemporal sequences. However, its implementation requires processing high-dimensional feature maps and employing complex self-attention mechanisms, resulting in enormous computational resource consumption.

[0037] To address this challenge, this invention integrates spiking neural networks and Siamese network structures to construct an energy-efficient Siamese spiking neural network model. The core advantage of spiking neural networks lies in their biologically inspired, pulse-driven computational paradigm: the spiking neuron, as the basic computational unit of the spiking neural network, simulates the dynamic changes in the membrane potential of biological neurons, abstracting the "integration-firing" mechanism in a concise manner while retaining key biological characteristics such as the refractory period. This embodiment employs leak-integration-firing (LIF) neurons, whose dynamic behavior can be described by the following differential equation:

[0038]

[0039] in, Represents membrane potential. This represents the resting potential (the potential of a spiking neuron when it is not activated). R represents the membrane time constant (which controls the rate of potential decay), and R represents the resistivity. Indicates the synaptic input current. Indicates time.

[0040] Spiking neural networks inherently possess the ability to encode precise temporal information from spikes. For spatiotemporally non-uniformly sampled data driven by event cameras, the temporal sequence and dynamic characteristics of spikes can explicitly model the microsecond-level temporal continuity between frames. Specifically, the Siamese spiking neural network model of this invention captures the temporal evolution of target motion by quantifying the dynamic correlation between the template frame and the search frame (i.e., by utilizing the phase difference of spikes), such as in scenarios involving occlusion or appearance changes. This mechanism enhances the robustness of the network model to complex temporal patterns.

[0041] In this invention, the spatiotemporally irregular keyframe sequence captured by an event camera is used as the input data source for the Siamese spiking neural network model. Specifically, the spatiotemporally non-uniformly sampled keyframe sequence acquired by the event camera is used as the input data source. This input data source is based on a dynamic random sampling strategy within a predefined time window (e.g., ...). Sample pairs generated from timestamp intervals, where, This represents the preset window scale parameter. Each generated sample pair includes a template frame and a search frame; the template frame is preferentially selected from frames in the sequence with significant motion edge features, and the search frame is placed after the template frame. The sample frame is randomly selected within a preset time interval to ensure that there is a spatiotemporal continuity constraint between sample pairs. In this embodiment, the template frame for each sample pair is the first frame in the pair, and the search frame is the second frame, thereby ensuring that the spatiotemporal continuity constraint is met between frames.

[0042] like Figure 1As shown, in this embodiment of the invention, the Siamese spiking neural network model includes two lightweight backbone networks with shared weights, constructed based on spiking neural convolutional blocks (SNN convolutional blocks). One lightweight backbone network receives a template frame as input, and the other receives a search frame to extract template and search features. An adaptive mask module is connected after the lightweight backbone network to achieve feature alignment, mask generation, and enhancement. Then, a spiking neural Transformer module (i.e., an SNN Transformer module) captures global dependencies in the feature map, introducing bio-inspired sparse coding / temporal modeling capabilities to enhance the understanding of temporal information of moving targets. Specifically, the spiking neural Transformer module implements a temporal self-attention mechanism on the pulse sequence, converting input labels into query (Qs), key (Ks), and value (Vs) pulses, and calculating relevance using pulse timestamp information. The spiking neural Transformer module can be implemented based on a self-cross-attention (SDSA) module. Finally, by outputting the target's current position and the endpoint of the tracking path (i.e., predicting the target's endpoint after a specified time t from the current position) by the output prediction layer, a tracking plan path for the target is generated, thereby filtering out the target's final trajectory from the multiple generated tracking plan paths based on trajectory matching.

[0043] After constructing the Siamese spiking neural network model, a training dataset is built based on the generated sample pairs (including template frames and search frames). The model parameters are then trained. When the preset training convergence conditions are met (e.g., the number of training iterations reaches a preset upper limit or the training loss function value converges), a Siamese spiking neural network model for obtaining the target tracking plan path is obtained. The prediction task of the output prediction layer can be set as: target location confidence and target classification result, etc. Optionally, the loss function during training can be set to include both classification loss and regression loss. For example, based on the predicted bounding boxes, L1 loss and generalized intersection-union (IoU) loss are used for bounding box regression. The overall loss function can be expressed as: ,in, For the total training loss, This represents the classification loss calculated based on the output target category of the output prediction layer. This represents the intersection-union ratio (IoU) loss based on the bounding box of the target. Represents the regression loss based on the coordinates of the bounding box of the target. This indicates the preset intersection-union loss weights. This represents the preset regression loss weights.

[0044] In one embodiment, such as Figure 2As shown, the lightweight backbone network used in this embodiment of the invention is a lightweight grouped convolutional extraction backbone network. Its structure includes a first stacked structure consisting of several first stacked modules cascaded together, each composed of a downsampling layer, a first stacked unit, and a second stacked unit. In this embodiment, the number of stacked first stacked modules is set to 2. The first stacked unit includes three spiking convolutional blocks, and the second stacked unit includes two spiking convolutional blocks. Each spiking convolutional block sequentially includes a LIF neuron layer and a grouped convolutional layer. The stacked unit consists of a convolutional layer (with a 3×3 kernel) and a batch normalization (BN) layer. In the first stacked unit, a residual connection is established between the output of the first spiking convolutional block and the output of the downsampling layer of the first stacked unit. That is, the output of the downsampling layer and the output of the first spiking convolutional block are added together and used as the input of the next spiking convolutional block. A residual connection is also established between two adjacent spiking convolutional blocks. In the second stacked unit, a residual connection is established between two spiking convolutional blocks. The downsampling layer of each stacked module uses grouped convolution, combined with a batch normalization layer and a ReLU activation function (preferably ReLU6) to adapt to the characteristics of the spiking network. In this embodiment, the lightweight backbone network used to process the template frame is called the template branch, and the lightweight backbone network used to process the search frame is called the search branch.

[0045] In one embodiment, such as Figure 3As shown, the adaptive masking module used in this embodiment of the invention includes two first downsampling layers. The first downsampling layer is used to process the template features output by the lightweight backbone network, and the second downsampling layer is used to process the search features output by the lightweight backbone network. The first downsampling layer is followed by, in sequence, an average pooling layer, a connection layer, a first convolutional layer, a first batch normalization layer, a ReLU activation function, a LIF neuron layer, a second convolutional layer, a second batch normalization layer, a Sigmoid activation function, and an adaptive enhancement layer. The search features processed by the second downsampling layer are directly fed into the connection layer and the adaptive enhancement layer. The two downsampling layers use grouped convolutions, followed by batch normalization layers and ReLU activation functions. The convolutional kernels of the two convolutional layers can both be set to 1×1. The average pooling layer compresses the feature dimension of the downsampled template features to 1×1×C, where C is the preset number of channels. Compared to traditional max pooling, this method retains the mean statistical information of the feature map and avoids feature bias caused by local highlight features. The connection layer adjusts the feature dimension of the template features after average pooling to match the spatial dimension of the downsampled search features, achieving complete alignment between the template features and the search features in spatial dimension. Then, the template features and the search features are concatenated by channel dimension, and the concatenated features are fed into the subsequent network layer. The adaptive masking module introduced in this invention dynamically enhances key region features (such as the target region) and suppresses redundant information such as background by learning spatial attention or feature selection mechanisms, thereby improving the discriminative power of the features. The adaptive enhancement layer is used to achieve adaptive enhancement of the target features, further improving the robustness and adaptability of the module; its final enhanced feature output is:

[0046]

[0047] in, Indicates the final enhancement feature, It is a learnable parameter. Indicates the input features of the search region. This is the mask generated by the mask predictor, which is the output of the first Sigmoid activation function in the network structure of the adaptive mask module described above. In this embodiment, the output value is mapped to the 0-1 range through the Sigmoid activation function to generate a binary mask that can accurately distinguish between the target and the background, where 1 corresponds to the effective area of ​​the target and 0 corresponds to the redundant area of ​​the background. Represents the Hadamard product. Learnable parameters. It can be represented as: ,in, These are learnable weights. This represents the Sigmoid activation function.

[0048] Spiking neural networks (NNs), relying on pulse firing mechanisms to transmit information, are highly susceptible to severe interference from redundant pulses in the background. These redundant pulses not only contain a large amount of meaningless background noise but also mix in low-frequency feature pulses unrelated to the target. On the one hand, this forces the subsequent attention module to perform redundant calculations on a massive number of invalid pulses during feature similarity calculations, significantly increasing the model's computational load and inference latency. On the other hand, it interferes with the attention mechanism's focus on the target's core features, leading to spurious responses during feature matching and directly causing a decrease in target localization accuracy. This severely restricts the application performance of NNs in high-precision target detection, trajectory prediction, and other tasks. To address this, this invention specifically designs an adaptive masking module, whose core function is positioned as "inserting a lightweight intervention mechanism after feature extraction and before attention interaction." This precise positioning targets the key feature processing nodes of the NN, efficiently filtering redundant background pulses at the source to solve the core interference problem. Furthermore, through an extremely streamlined module architecture design, it avoids introducing additional computational burdens and parameter overhead, ultimately achieving the optimal balance between target localization accuracy and computational efficiency, perfectly meeting the dual requirements of lightweight and high performance for NNs.

[0049] In the critical pulse selection stage, this invention uses an adaptive masking module to initially fuse template features and search features, leveraging their semantic association to achieve preliminary differentiation between the target and background regions. Subsequently, the module focuses on retaining effective pulses in the target region—especially high-frequency edge pulses that characterize the target contour and detailed texture; these pulses are the core basis for subsequent target matching and localization. Simultaneously, through thresholding and semantic filtering mechanisms, noise pulses and low-frequency redundant pulses in the background region are thoroughly eliminated, fundamentally reducing the computational data volume of the subsequent attention module and laying the foundation for improving model computational efficiency. In this embodiment, the adaptive masking module is inserted after feature extraction in the template and search branches, before the self-crossing attention-based pulse neural Transformer module. This placement offers dual technical advantages:

[0050] (1) At this time, the template features and search features have completed the initial feature screening and dimension regularization through the backbone network. The proportion of target information in the feature layer has been significantly increased. The mask can be directly applied to the high-value effective feature layer without modifying the feature extraction logic of the backbone network. This minimizes the changes to the original network structure and reduces the adaptation cost and training difficulty of module integration.

[0051] (2) Filtering redundant information before attention interaction can directly avoid the meaningless similarity measurement calculation of background noise features by the subsequent self-cross attention module. This design greatly reduces the size of the matching matrix and the amount of invalid computation in the attention calculation process, while avoiding the interference of redundant features on the attention weight allocation, so that the attention mechanism can focus more accurately on the core features of the target, taking into account both efficiency improvement and accuracy optimization.

[0052] In this embodiment, the adaptive masking module achieves dynamic and precise filtering of redundant pulses through four interconnected steps: template guidance, feature fusion, mask generation, and adaptive enhancement. Each step incorporates a lightweight design philosophy, strictly controlling computational overhead while ensuring filtering effectiveness. The specific workflow and technical details are as follows:

[0053] 1) Template-guided mask generation: Average pooling is used to process template features to fully preserve the global semantic information and overall structural features of the template. Compared to max pooling, average pooling can extract global statistical information of template features in a balanced manner, covering key information such as the overall contour and spatial distribution of the target; while max pooling often overemphasizes local highly activated feature points, easily ignoring the global structure and contextual relationships of the target, which may lead to local biases in the generated mask. Therefore, average pooling is chosen as the core operation for template feature extraction, providing a more comprehensive and reliable guiding basis for subsequent mask generation.

[0054] 2) Feature Fusion: Due to spatial differences between the template features after average pooling and the search features, direct feature interaction can lead to dimensionality mismatch. To address this, the module performs copying and spatial expansion operations on the average pooled template features. Through an adaptive dimensionality adjustment algorithm, it precisely matches the spatial dimensionality parameters of the search features, achieving complete spatial alignment between the template-guided features and the search features. This design ensures that subsequent masks can accurately target the corresponding regions of the search features, avoiding filtering failures caused by dimensionality misalignment.

[0055] 3) Mask Generation: An extremely simplified lightweight mask predictor is employed, consisting of only two basic convolutional blocks, achieving mask generation without a complex network structure. The first convolutional block uses a 1×1 kernel, achieving dimensionality reduction and feature fusion of the channel dimension without changing the feature space dimension, effectively controlling the number of parameters. A batch normalization layer follows the convolution, accelerating model training convergence and improving the stability of the feature distribution. A ReLU activation function is applied after the first convolutional block to enhance the non-linear expressive power of the features. The second convolutional block also uses a 1×1 convolution as its core, combined with a batch normalization layer to optimize the feature output. Finally, a Sigmoid activation function maps the output value to the 0-1 range, generating a binary mask that accurately distinguishes the target from the background, where 1 corresponds to the effective target region and 0 corresponds to the redundant background region. In other words, this invention uses the global features of the template frame as a guide, fusing them with search features to generate a mask, achieving the purpose of filtering redundant background pulses and enhancing target features.

[0056] 4) Adaptive Feature Enhancement: To prevent the mask from excessively removing weak edge features of the target when filtering redundant pulses, thus compromising the integrity of the target features, a learnable weight parameter is introduced. This parameter is dynamically adjusted during model training based on the task scenario and data characteristics. It balances the contribution of the original search features and the masked search features through a weighted summation method—preserving the accuracy of the target features after mask filtering while also ensuring the integrity of weakly effective information in the original features. Compared to traditional masking mechanisms (such as fixed masks or global self-attention masks), the lightweight advantage of the adaptive masking module of this invention is significant, specifically reflected in:

[0057] (a) Very few parameters: For example, for an input with 128 channels and a reduction ratio of 16, the module has only 2057 parameters in total, far fewer than the tens of thousands of parameters in a global self-attention masking scheme.

[0058] (b) Low computational overhead: Single-frame mask generation is faster than other methods.

[0059] (c) Non-intrusive design: No modification is required to the backbone network or attention module, making it easy to integrate into other Siamese frameworks and demonstrating strong adaptability.

[0060] To further advance the lightweight implementation of spiking neural networks and meet the stringent requirements of real-time performance and hardware compatibility in target trajectory prediction tasks, this invention employs a lightweight design throughout its grouped convolutional approach. Its technical objectives have a clear priority order, focusing on reducing model parameter size, computational complexity, and memory usage, rather than prioritizing direct improvements in detection or accuracy. This design philosophy precisely addresses the pain points of spiking neural networks in embedded devices and edge computing scenarios. Furthermore, by optimizing the grouping strategy, it cleverly adapts to the parallel computing characteristics of graphics processing units (GPUs) and neuromorphic chips, leveraging the efficient release of hardware computing power to further enhance model inference speed, achieving a dual optimization of "algorithm lightweighting" and "hardware compatibility."

[0061] To address the bottlenecks of traditional target trajectory prediction models, such as large number of parameters, high memory consumption, and slow inference speed, this invention proposes a lightweight overall solution based on the core concept of "spatiotemporal dual compression." This solution deeply integrates the spatial dimension compression advantages of grouped convolution with the temporal dimension simplification characteristics of pulse coding, constructing a collaborative and efficient lightweight technology system. Pulse coding captures the temporal variation patterns of features, releasing pulse signals only at key temporal nodes to eliminate redundant information in the temporal dimension; grouped convolution performs structured simplification of the spatial channel dimension of the feature map. The two complement each other, breaking through the performance bottlenecks of traditional models from both spatiotemporal dimensions. Technically, the core of the grouped convolution lies in breaking the fully connected mapping relationship between the input and output channels in standard convolution, reconstructing traditional dense connections into sparse grouped connections, and significantly reducing computational costs through structured decomposition of the channel dimension.

[0062] In traditional standard convolution operations, a single convolution kernel needs to traverse all channels of the input feature map, integrating and calculating the feature information from all channels before outputting the corresponding feature. While this fully connected approach ensures comprehensive feature fusion, the number of parameters and computational cost increase exponentially with the number of input and output channels—assuming the number of input channels is... The number of output channels is The kernel size is The standard convolution has 100,000 parameters. The computational complexity increases in tandem with the size of the parameter model. The core optimization logic of grouped convolution is to equally divide the input and output channels, decomposing them into... There are 3 independent channel groups (i.e., each group has 10 independent input channels). The number of output channels in each group is Convolution operations are strictly limited to independent operations within each group: each output channel group is generated solely by convolving its corresponding input channel group, with no feature interaction or parameter sharing between groups. This significantly reduces the number of parameters in grouped convolutions. The parameter size and computational complexity are both reduced to that of standard convolution. .

[0063] In one embodiment, the pulse neural Transformer module used in this embodiment is as follows: Figure 4 As shown, similar to the lightweight backbone network, a self-crossing attention module (i.e., SDSA attention module) is added between the two stacked units. In this embodiment, the spiking neural Transformer module includes a second stacked structure consisting of several cascaded second stacked modules composed of downsampling layers, third stacked units, self-crossing attention modules, and fourth stacked units. In this embodiment, the number of stacks of the second stacked modules is set to 2. The third stacked unit includes three spiking neural convolutional blocks, and the second stacked unit includes two spiking neural convolutional blocks. Each spiking neural convolutional block includes, in sequence, a LIF neuron layer, a group convolutional layer (Groupconv, the convolutional kernel can be set to 3×3), and a batch normalization layer (BN). In the third stacked unit, a residual connection is set between the output of the first spiking neural convolutional block and the output of the downsampling layer of the first stacked unit, and a residual connection is set between two adjacent spiking neural convolutional blocks. In the fourth stacked unit, a residual connection is set between two spiking neural convolutional blocks.

[0064] In this embodiment, the default number of groups for all downsampling layers in the grouped convolution can be set to 2 for channel splitting, in order to reduce parameters and computational load.

[0065] In the network model constructed in this invention, the design of grouped convolution is intended to simultaneously adapt to the parallel computing characteristics of graphics processing units and neuromorphic chips, while maintaining compatibility with the discrete features of spiking neural networks to prevent information loss. Its core advantage lies in effectively combining the spatial compression provided by grouped convolution with the temporal compression achieved by the pulse coding of spiking neural networks, thus realizing "spatiotemporal dual compression".

[0066] In summary, this embodiment addresses the core technical bottleneck of "difficulty in balancing accuracy and real-time performance" prevalent in the current field of target trajectory prediction. Specifically, when enhancing model feature representation capabilities to improve accuracy in complex scenarios, the increase in parameter size and computational load often leads to a decrease in inference speed, failing to meet the demands of real-time trajectory prediction. To address this, a lightweight target trajectory prediction method based on Siamese neural networks is proposed. This method forms a collaborative optimization system through multi-dimensional technological innovation, systematically breaking through the performance constraints of traditional solutions. Its core technological advantages and application value are specifically reflected in the following two aspects:

[0067] 1) Introducing a dynamic masking mechanism to efficiently suppress background interference and improve positioning accuracy: A targeted adaptive masking module is designed to accurately embed key nodes after feature extraction and before attention interaction. This mechanism generates an adaptive mask through template guidance, intelligently selecting effective impulse features in the target region and powerfully filtering redundant background information and noise interference in complex scenes, avoiding interference from invalid features in subsequent attention calculations. This design not only solves the target positioning offset problem caused by background noise but also maintains the module's lightweight characteristics, significantly improving the accuracy and stability of target positioning in complex environments without increasing additional computational burden.

[0068] 2) A lightweight strategy combining grouped convolutions achieves a significant improvement in model efficiency: By deeply integrating the sparse connectivity of grouped convolutions, a lightweight architecture adapted to Siamese neural networks is constructed. This strategy significantly reduces the parameter size and computational complexity to that of standard convolutions by splitting the fully connected relationships between the input and output channels. Furthermore, leveraging the independent computation characteristics within a group, it perfectly aligns with the parallel computing mechanism of graphics processing units (GPUs) and neuromorphic chips. This design significantly reduces the number of model parameters and shortens inference time, while effectively avoiding the feature representation capability degradation problem that may occur during the lightweighting process, providing a solid guarantee for the real-time deployment of the model.

[0069] The method proposed in this embodiment achieves a breakthrough in both accuracy improvement and real-time performance optimization in target trajectory prediction tasks. It not only meets the demand for high-precision trajectory prediction in complex scenarios, but also has the characteristics of being lightweight and easy to deploy. It can be widely adapted to practical application scenarios with limited hardware resources, such as embedded devices and real-time monitoring, and has important theoretical reference value and engineering practice significance.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0071] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A lightweight target trajectory prediction method based on Siamese spiking neural networks, characterized in that, Includes the following steps: Step 1, construct sample pairs; Using keyframe sequences of spatiotemporally non-uniformly sampled data acquired by an event camera as the input data source, a dynamic random sampling strategy is employed to generate sample pairs within a preset time window; each sample includes a template frame and a search frame. Step 2: Construct a Siamese spiking neural network model, which includes two weight-shared template branches and a search branch. The template and search branches have identical network structures, employing a lightweight backbone network based on spiking neural convolutional blocks. The template branch receives template frames from sample pairs as input, used to extract template features. The search branch receives search frames from sample pairs as input, used to extract search features. An adaptive masking module is connected after the template and search branches to achieve feature alignment and mask generation between template and search features. Based on the generated mask, the search features are adaptively enhanced, and the enhanced features are output to the spiking neural Transformer module. The spiking neural Transformer module obtains global dependencies in the feature maps corresponding to the enhanced features based on spiking neural convolutional blocks and a self-attention mechanism, calculating correlations using pulse timestamp information. Finally, the current position of the target and the endpoint of the tracking path are output through the output prediction layer to generate the target's tracking plan path. Step 3: Construct a training dataset based on the sample pairs generated in Step 1, train the model parameters of the Siamese spiking neural network model, and obtain the Siamese spiking neural network model for obtaining the tracking plan path of the target when the preset training convergence condition is met. The adaptive masking module includes two downsampling layers: the first downsampling layer is used to process template features, and the second downsampling layer is used to process search features. Following the first downsampling layer are the following layers in sequence: average pooling layer, connection layer, first convolutional layer, first batch normalization layer, ReLU activation function, LIF neuron layer, second convolutional layer, second batch normalization layer, Sigmoid activation function, and adaptive enhancement layer; and the search features processed by the second downsampling layer. Directly fed into the connection layer and adaptive enhancement layer; The two downsampling layers use grouped convolution, and the kernel size of both convolutional layers is set to 1×1. The connection layer adjusts the feature dimension of the template features after average pooling to match the spatial dimension of the downsampled search features, thus aligning the template and search features spatially. Then, the template and search features are concatenated along the channel dimension. The concatenated features are then fed into the first convolutional layer, sequentially passing through the first batch of normalization layers, ReLU activation function, LIF neuron layer, second convolutional layer, second batch of normalization layers, and a sigmoid activation function to generate a mask that distinguishes the target from the background. ; Adaptive enhancement layer based on mask and learnable parameters Perform feature enhancement and output enhanced features: ； in, Indicates enhanced features, It represents the Hadamardi (or Hadama) stack.

2. The lightweight target trajectory prediction method based on Siamese spiking neural networks as described in claim 1, characterized in that, The template frame is selected from the keyframe sequence that meets the motion edge features with preset conditions, while the search frame is located after the template frame. Randomly selected within a time interval; where, This is the default value.

3. The lightweight target trajectory prediction method based on Siamese spiking neural networks as described in claim 1, characterized in that, The template frame is selected as the first frame in the keyframe sequence.

4. The lightweight target trajectory prediction method based on Siamese spiking neural network as described in claim 1, characterized in that, The lightweight backbone network includes a first stacked structure consisting of several first stacked modules cascaded together, each consisting of a downsampling layer, a first stacking unit, and a second stacking unit. Each first stacking unit includes three spiking convolutional blocks, and each second stacking unit includes two spiking convolutional blocks. Each spiking convolutional block sequentially includes a LIF neuron layer, a grouped convolutional layer, and a batch normalization layer. In the first stacking unit, a residual connection is established between the output of the first spiking convolutional block and the output of the downsampling layer of the first stacking unit, and a residual connection is established between two adjacent spiking convolutional blocks. In the second stacking unit, a residual connection is established between two spiking convolutional blocks.

5. The lightweight target trajectory prediction method based on Siamese spiking neural networks as described in claim 1, characterized in that, mask In the diagram, 1 corresponds to the effective target area, and 0 corresponds to the redundant background area.

6. The lightweight target trajectory prediction method based on Siamese spiking neural networks as described in claim 1, characterized in that, Learnable parameters for: ,in, Indicates learnable weights, This represents the Sigmoid activation function.

7. The lightweight target trajectory prediction method based on Siamese spiking neural networks as described in claim 1, characterized in that, The spiking neural Transformer module sequentially includes a second stacked structure consisting of several cascaded second stacked modules, each composed of a downsampling layer, a third stacked unit, a self-crossing attention module, and a fourth stacked unit. The third stacked unit includes three spiking neural convolutional blocks, and the second stacked unit includes two spiking neural convolutional blocks. Each spiking neural convolutional block sequentially includes a LIF neuron layer, a grouped convolutional layer, and a batch normalization layer. In the third stacked unit, a residual connection is established between the output of the first spiking neural convolutional block and the output of the downsampling layer of the first stacked unit, and a residual connection is also established between adjacent spiking neural convolutional blocks. In the fourth stacked unit, a residual connection is established between two spiking neural convolutional blocks.

8. The lightweight target trajectory prediction method based on Siamese spiking neural networks as described in claim 4 or 7, characterized in that, The kernel size of the grouped convolutional layers in the spiking neural convolution block is set to 3×3.

Images