A target tracking method based on event data and spiking neural network

Abstract

The application discloses a target tracking method based on event data and a pulse neural network, and comprises the following steps: acquiring event data, and performing time dimension compression processing on the event data to obtain sample data; adjusting the structure of a SiamFC network into a pulse neural network, pre-training the SiamFC network by using the sample data, and then migrating the obtained weight and bias to the pulse neural network to optimize the pulse neural network; acquiring real-time event data, performing time step adjustment processing to obtain search data, and performing inference and comprehensive similarity estimation on the search data and template data by using the optimized pulse neural network to obtain a real-time target tracking result. The target tracking method is optimized by the trained SiamFC pulse neural network, and the pulse neural network is accelerated based on an FPGA, so that the real-time performance and precision of the target tracking of the pulse neural network are improved under low power consumption.

Description

Technical Field

[0001] This invention belongs to the field of target tracking technology, specifically relating to a target tracking method based on event data and a spiking neural network. Background Technology

[0002] Target tracking is a crucial component of intelligent video surveillance, widely used in intelligent transportation, real-time monitoring, and human-computer interaction. Traditional sensors and tracking algorithms are based on camera image frames, which suffer from drawbacks such as high redundancy, high latency, and high data volume. High-speed and accurate tracking is often difficult to achieve under complex environmental conditions, facing challenges such as lighting, deformation, occlusion, scale changes, image blurring, and rapid motion. Current target tracking algorithms mainly employ two methods: correlation filters and deep learning.

[0003] Fully-convolutional Siamese networks (SiamFC) have a paired network structure, specifically, they have two inputs: a baseline template and candidate samples to be selected. In single-object tracking tasks, the baseline template is the object to be tracked, typically the target object in the first frame of a video sequence, while the candidate samples are the image search regions in each subsequent frame. The SiamFC's task is to find the candidate region in each subsequent frame that is most similar to the template in the first frame, which is the target in that frame. This allows us to track a single target. Compared to other fully convolutional network tracking algorithms, SiamFC has a smaller network structure, requires less computation, and also offers faster tracking speed.

[0004] Spike neural networks (SNNs) are third-generation neural networks characterized by temporal dynamics in their neuronal nodes, homeostatic-plastic synaptic structures, and function-specific network loops. In SNNs, neurons are not activated in every iteration, but rather when their membrane potential reaches a specific threshold. When a neuron is activated, it generates a signal that is transmitted to other neurons, raising or lowering their membrane potential. This mimics the firing of pulses by biological neurons, offering advantages such as low power consumption and high speed not found in traditional artificial neural networks, thus overcoming the shortcomings of deep learning methods used for target tracking. Methods for converting artificial neural networks into spiking neural networks can fully utilize existing artificial neural network training algorithms, enabling the training of deeper networks.

[0005] Hardware platforms capable of supporting spiking neural networks can be broadly categorized into two types: customized FPGA accelerators for specific models and neuromorphic computers suitable for general-purpose models. FPGAs enable rapid hardware functional verification and evaluation, and allow for rapid design iteration with fewer constraints on neural networks, making them a popular choice for many hardware designs. Furthermore, a successful FPGA implementation can serve as a preliminary step in custom chip design, providing a reference for chip design. Summary of the Invention

[0006] To address the technical problems of existing target tracking methods, such as high computational load and low real-time performance, the present invention aims to provide a target tracking method based on event data and spiking neural networks. By optimizing the spiking neural network with trained SiamFC and accelerating the spiking neural network based on FPGA, the real-time performance and accuracy of target tracking by spiking neural networks are improved with low power consumption.

[0007] To achieve the above-mentioned objectives, an embodiment provides a target tracking method based on event data and a spiking neural network, comprising the following steps:

[0008] Acquire event data and compress the event data into time-dimension event frames to obtain sample data;

[0009] The SiamFC network is structurally modified to become a spiking neural network. After pre-training the SiamFC network with sample data, the resulting weights and biases are transferred to the spiking neural network to optimize it.

[0010] Real-time event data is acquired and the time step is adjusted to obtain search data. An optimized spiking neural network is used to perform inference and comprehensive similarity estimation on the search data and template data to obtain real-time target tracking results.

[0011] Preferably, the event data is acquired asynchronously via a dynamic visual sensor, and the event data e includes the position information (x, y) of each pixel, the event polarity p, and the time t, represented as e = [x, y, t, p]. T The polarity p of an event includes positive and negative events;

[0012] The process of compressing the event data in the time dimension of event frames includes:

[0013] Event data within a given time period is compiled into an event packet, and the target pixel points (x) within the event packet are counted. k y k The number of positive events and the number of negative events Event data of different polarities are stored in different channels to obtain two-dimensional data containing the number of positive events and the number of negative events.

[0014] Preferably, the step of cropping the SiamFC network into a pattern consistent with a spiking neural network includes: setting the output of each convolutional layer in the SiamFC network to positive using an absolute value function;

[0015] Set the bias of the convolution operation in each convolutional layer of the SiamFC network to 0;

[0016] The max pooling operation in the SiamFC network is changed to evaluate the firing rate of neurons at each time step, that is, to calculate the absolute impulse firing rate accumulated over time, and to allow the impulses of neurons with the maximum impulse firing rate in a pooling window to pass through based on the absolute impulse firing rate.

[0017] Preferably, the absolute pulse firing rate is calculated in the following manner:

[0018] f s (t)=f s (t-1)+x s (t)(Tt)

[0019] Among them, f s (t) represents the firing rate of neuron s at time t, x s (t) indicates whether neuron s fired a pulse at time t, x s (t) = 1 indicates that the payment is made, x s (t) = 0 indicates that no firing occurs. If the neuron fires a pulse, the firing rate of the neuron will be updated; otherwise, it will not be updated. T represents the length of the time window required for one inference.

[0020] Preferably, the step of transferring the obtained weights and biases to the spiking neural network to optimize the spiking neural network includes:

[0021] At each channel dimension, the weights and biases are normalized using the maximum activation value, expressed by the formula:

[0022]

[0023]

[0024] Where i and j represent dimension subscripts, and l represents the layer number. This represents the weights of layer l in the i and j dimensions, achieved by using the maximum activation value on each channel. For non-first-level cells, the normalized activation values ​​must be multiplied by . The input is restored to its value before normalization in the previous layer, and then normalized in this layer to obtain the normalized weights. This represents the bias of layer l in the j-th dimension. Indicates the bias after normalization;

[0025] The normalized weights and biases are used to transfer the algorithm to spiking neural networks to optimize them.

[0026] Preferably, the time step of the real-time event data is adjusted, including: preset the time step and the number of time steps N, and accumulate the real-time event data in each time step according to the time step. If the number of events in the preset N time steps is greater than 0, the pixel value is marked as 1; otherwise, the pixel value is marked as 0, and the search data is obtained.

[0027] Preferably, an optimized spiking neural network is used to perform inference and comprehensive similarity estimation on the search data and template data, including:

[0028] The search data and template data are respectively input into two branches of the optimized spiking neural network. After forward inference calculation, the activation function of the last layer of each branch accumulates the membrane potential within a given time without triggering a peak. At the same time, two pulse feature maps corresponding to the two data are obtained. The potential similarity estimate is calculated by the correlation between the membrane potentials corresponding to the two pulse feature maps, which is expressed by the formula:

[0029]

[0030] in, This represents the pulse feature map corresponding to the template data at time step t. M represents the pulse feature map corresponding to the search data at time step t. P (z, x) represents the potential similarity estimate;

[0031] Simultaneously, the temporal correlation between the two pulse feature maps is calculated, and temporal similarity estimates are matched at each time step, expressed by the formula:

[0032]

[0033] Where τ represents the response period;

[0034] In the conversion from SiamFC network to spiking neural network, the error is inversely proportional to the time. Combining latent similarity estimation and temporal similarity estimation yields a comprehensive similarity estimate, expressed by the formula:

[0035]

[0036] Among them, f spike (z, x) represents the comprehensive similarity estimate, and T represents the time period.

[0037] Preferably, the inference process of the spiking neural network is accelerated by an FPGA, and the spiking neural network is subjected to fixed-point quantization to convert it into data that can be input into the FPGA, including:

[0038] By v(t)=V(t)·2 β As the neuron state voltage, W i =w i ·2 β As synaptic weights, v thr =V thr ·2 β The floating-point to vertex transformation is performed as the threshold of the neuron, β represents the scaling factor, V(t) represents the original voltage of the neuron, and w i Representing the original weights of neurons, the formula is transformed into the following fixed-point form:

[0039]

[0040] If v(t)≥v thr The pulse is emitted and the reset is v(t) = v(t) - v thr

[0041] If v(t) < v min Reset v(t) = 0

[0042] Where, x i (t-1) represents the pulse input value of the i-th neuron in the previous layer at time t-1, where the value of i is the number of neurons in the previous layer.

[0043] Preferably, when the inference process of the spiking neural network is accelerated by an FPGA, it includes two parts: data transmission and network computation. The data transmission part is used to transmit network transmission pulse data from the PC to the FPGA and transmit the network transmission pulse results calculated by the FPGA back to the PC for display.

[0044] The network computing part is implemented on the FPGA. After the network inputs pulse data, it outputs pulse data after a specified number of calculation layers. The network computing is concentrated inside the pulse convolution layer, which is divided into a pulse convolution overall control module, a shift module, a computing unit control module, a pulse computing unit array, and a membrane potential calculation module.

[0045] The pulse convolution overall control module is used to control the data calculation inside the pulse convolution layer. The pulse calculation unit array is composed of pulse calculation units. Each pulse calculation unit is used to realize the weight accumulation of a single convolution kernel based on pulse data. The calculation unit control module is used to control the weight accumulation of a single convolution kernel. The shift module is used to control the shift of pulse data between pulse calculation units in the pulse calculation unit array. The membrane potential calculation module is used to calculate the membrane potential based on the cumulative weights of all convolution kernels inside the pulse convolution layer.

[0046] Preferably, each convolutional kernel is equipped with one pulse calculation unit. The input pulse data is stored in the register inside the pulse calculation unit. Each pulse data contains the pulse information of the neurons of the number of channels. When the control signal output by the calculation unit control module indicates the start of calculation, all channels are traversed. Based on whether the pulse of the corresponding channel in the register is 1 or 0, it is selected whether to accumulate the input weight value to the weight sum. After all channels are traversed, the accumulated weight sum is output to the membrane potential calculation module, and the pulse data in the register is sent to other pulse calculation units.

[0047] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0048] Different processing methods are applied to the input event data for training and testing. During training, the event data stream is compressed into two-dimensional data according to polarity to preserve the accuracy of artificial network training. During testing, the event data stream is re-divided according to time steps to reduce redundant data, preserve the impulse characteristics of event data, and better match the spiking neural network.

[0049] The method of converting traditional artificial neural networks into spiking neural networks is adopted. The SiamFC network is pre-trained to avoid the problem of non-differentiation at the pulse firing point of the spiking neural network during training. Furthermore, the mature training methods of artificial neural networks can be used to make the converted spiking neural network results more accurate.

[0050] Using FPGAs for hardware acceleration of spiking neural networks allows for rapid design iterations with fewer constraints on the neural network. The hardware architecture improves speed and reduces memory consumption compared to implementing the same algorithm on a CPU. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1This is a flowchart of the target tracking method based on event data and spiking neural networks provided in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the SiamFC network structure provided in an embodiment of the present invention;

[0054] Figure 3 This is a flowchart of target tracking using a spiking neural network provided in an embodiment of the present invention;

[0055] Figure 4 This is a diagram illustrating the structure and implementation process of the pulse calculation unit provided in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the structure of the pulse calculation unit array provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0058] To address the technical problems of high computational cost and low real-time performance in traditional target tracking methods, this invention provides a target tracking method based on event data and a spiking neural network. The method includes input processing of event data, construction of the spiking neural network, and comprehensive similarity estimation analysis. Event data, after compression of the time dimension and adjustment of the time step, is input into both the SiamFC network to be trained and the spiking neural network to be inferred. The spiking neural network architecture is derived from a trimmed SiamFC network, using normalized weights obtained from the pre-trained SiamFC network. An FPGA hardware acceleration architecture is designed to accelerate the spiking neural network. This method can track target images from event data in real time, is suitable for high-speed moving targets, and offers low power consumption and high accuracy.

[0059] Figure 1 This is a flowchart of a target tracking method based on event data and a spiking neural network provided in an embodiment of the present invention. Figure 1 As shown, the target tracking method based on event data and spiking neural networks provided in this embodiment includes the following steps:

[0060] S110: Acquire event data and perform time-dimension compression processing on the event data to obtain sample data.

[0061] In this embodiment, a dynamic vision sensor is used to acquire asynchronous event data streams. The dynamic vision sensor perceives the light intensity change information at each pixel. When the light intensity change range exceeds a certain threshold, the vision sensor can independently output the position information (x, y), event polarity p, and time t of each pixel, represented as four-dimensional data e = [x, y, t, p]. T The event polarity p includes positive and negative events. Using event data streams acquired by dynamic vision sensors as input data, it is better suited for high-speed motion scenes and offers higher real-time performance compared to traditional images.

[0062] After obtaining the event data, it is preprocessed to construct sample data for training the SiamFC network. The event data stream for training is input into the SiamFC network, and the event data undergoes temporal compression processing, converting the four-dimensional data into two-dimensional event frames. Specifically, this includes selecting event data within a time period to form an event packet, and counting the target pixel points (x, y) in the event packet. k y k The number of positive events and the number of negative events Event data of different polarities are stored in different channels to obtain two-dimensional data containing the number of positive events and the number of negative events as sample data.

[0063] S120, the SiamFC network is structurally adjusted to transform it into a spiking neural network.

[0064] In this embodiment, the SiamFC network is used for target tracking, and its specific structure and parameters are shown in Table 1 and... Figure 2 As shown:

[0065] Table 1

[0066]

[0067]

[0068] Where Conv2d represents a convolutional layer, ReLU represents a neuron layer, Maxpooling represents a max pooling layer, Z represents a template frame, and X represents a search frame. size represents the batch size, and stirde represents the time step.

[0069] In this embodiment, the SiamFC network is structurally adjusted to become a spiking neural network, mainly through the following adjustments:

[0070] First, the weights and biases obtained from convolution operations may output negative values, and the activation function may produce negative activation values. Spiking neural networks cannot represent negative weights and activation values. To address this issue, an absolute value function is added after each convolutional layer, ensuring that the output of each convolutional layer is non-negative.

[0071] Second, a bias is obtained after each convolution operation, but spiking neural networks cannot achieve a constant bias input during the accumulation of membrane potential. To address this issue, all biases can be set to 0.

[0072] Third, max pooling in convolutional networks selects the largest feature value within a constant-size window, while spiking neural networks cannot perform non-linear max pooling. In spiking neural networks, max pooling allows the pulses of neurons with the highest firing rate within a pooling window to pass through. Therefore, at each time step, the firing rate of neurons needs to be evaluated. This invention calculates the absolute firing rate accumulated over time, and the firing rate formula is as follows:

[0073] f s (t)=f s (t-1)+x s (t)(Tt)

[0074] Among them, f s (t) represents the firing rate of neuron s at time t, x s (t) indicates whether neuron s fired a pulse at time t, x s (t) = 1 indicates that the payment is made, x s (t) = 0 indicates that no firing occurs. If the neuron fires a pulse, the firing rate of the neuron will be updated; otherwise, it will not be updated. T represents the length of the time window required for one inference.

[0075] In this embodiment, the impulses of neurons with the highest impulse firing rate are allowed to pass through a pooling window based on their absolute impulse firing rate. After each inference operation, the neuron's firing rate is reset to 0, with later-arriving impulses having a smaller impact on the firing rate. Furthermore, a neuron firing later may have a higher firing rate than a neuron firing earlier if it fires multiple impulses.

[0076] S130: After pre-training the SiamFC network using sample data, the obtained weights and biases are transferred to the spiking neural network to optimize the spiking neural network.

[0077] In this embodiment, the template data (template image) and sample data (image of the region to be searched) are 127×127 and 255×255 in size, respectively. They are input into the corresponding branch of the SiamFC network for end-to-end forward inference to obtain a confidence score map. The target location is then determined from the confidence score map. One path of the SiamFC network consists of 5 convolutional layers, with max pooling layers after the first and second layers. The final features obtained from the two convolutions are then subjected to further convolution calculations for similarity evaluation. The higher the similarity, the better the target tracking performance. The SiamFC network is pre-trained using sample data to obtain pre-trained weights and biases.

[0078] In this embodiment, after pre-training the SiamFC network, the obtained weights and biases are transferred to the spiking neural network to optimize it. Specifically, at the channel dimension of each layer, the weights and biases are normalized using the maximum activation value, as expressed by the formula:

[0079]

[0080]

[0081] Where i and j represent dimension subscripts, and l represents the layer number. This represents the weights of layer l in the i and j dimensions, achieved by using the maximum activation value on each channel. For non-first-level cells, the normalized activation values ​​must be multiplied by . To restore the input to its value before normalization in the previous layer, we then perform normalization in this layer; otherwise, the information passed on will become increasingly smaller, resulting in normalized weights. This represents the bias of layer l in the j-th dimension. This represents the normalized bias. The normalized weights and biases are used to transfer the optimization of the spiking neural network.

[0082] S140: Obtain real-time event data and adjust the time step to obtain search data.

[0083] In this embodiment, the event stream data used for inference is input into the converted SiamSNN network. The spiking neural network has a high compatibility with the asynchronous event stream data format, eliminating the need to compress the event data dimension; only the time step size and number of time steps need to be redefined. Specifically, a preset time step size (e.g., 0.1ms) and a preset number of time steps N (e.g., N=50) are used. Real-time event data within each time step is accumulated according to the preset time step size. If the number of events within the preset N time steps is greater than 0, the pixel value is marked as 1; otherwise, the pixel value is marked as 0, resulting in search data with newly defined time steps.

[0084] S150 utilizes an optimized spiking neural network to perform inference and comprehensive similarity estimation on the search data and template data to obtain real-time target tracking results.

[0085] In this embodiment, when using an optimized spiking neural network to perform reasoning on search data and template data, a comprehensive similarity estimation is employed to unify latent similarity estimation and temporal similarity estimation. Specifically, as shown... Figure 3 As shown, the search data (search graph) and template data (template graph) are input into two branches of the optimized spiking neural network, respectively. After forward inference calculation, the activation function of the last layer of each branch accumulates the membrane potential within a given time without triggering a peak, thus obtaining two spiking feature maps corresponding to the two data points. The potential similarity estimate is calculated by the correlation between the membrane potentials corresponding to the two spiking feature maps, expressed by the formula:

[0086]

[0087] in, This represents the pulse feature map corresponding to the template data at time step t. M represents the pulse feature map corresponding to the search data at time step t. P (z, x) represents the potential similarity estimate;

[0088] Simultaneously, the temporal correlation between the two pulse feature maps is calculated, and temporal similarity estimates are matched at each time step, expressed by the formula:

[0089]

[0090] Where τ represents the response period;

[0091] In the conversion from SiamFC network to spiking neural network, the error is inversely proportional to the time. Combining latent similarity estimation and temporal similarity estimation yields a comprehensive similarity estimate, expressed by the formula:

[0092]

[0093] Among them, f spike (z, x) represents the comprehensive similarity estimate, and T represents the time period.

[0094] In this embodiment, to improve computational efficiency, the inference process of the aforementioned spiking neural network is accelerated by an FPGA. Therefore, the spiking neural network needs to be fixed-point quantized to convert it into data that can be input into the FPGA. Specifically, this includes:

[0095] By v(t)=V(t)·2 β As the neuron state voltage, W i =w i·2 β As synaptic weights, v thr =V thr ·2 β The floating-point to vertex conversion is performed using the neuron threshold. β represents the scaling factor, which can take different values ​​to represent different compression quantizations. For example, a β value of 3 indicates 8-bit quantization to reduce on-chip storage space. V(t) represents the original neuron voltage, and w... i Representing the original weights of neurons, the formula is transformed into the following fixed-point form:

[0096]

[0097] If v(t)≥v thr The pulse is emitted and the reset is v(t) = v(t) - v thr

[0098] If v(t) < v min Reset v(t) = 0

[0099] Where, x i (t-1) represents the pulse input value of the i-th neuron in the previous layer at time t-1, where the value of i is the number of neurons in the previous layer.

[0100] In this embodiment, when the inference process of the spiking neural network is accelerated by an FPGA, it includes two parts: data transmission and network computation. The data transmission part is used to transmit network transmission pulse data from the PC to the FPGA and transmit the network transmission pulse results calculated by the FPGA back to the PC for display.

[0101] The network computing part is implemented on the FPGA. After the network inputs pulse data, it outputs pulse data after a specified number of calculation layers. The network computing is concentrated inside the pulse convolution layer, which is divided into a pulse convolution overall control module, a shift module, a computing unit control module, a pulse computing unit array, and a membrane potential calculation module.

[0102] The pulse convolution overall control module is used to control the data calculation inside the pulse convolution layer. The pulse calculation unit array is composed of pulse calculation units. Each pulse calculation unit is used to realize the weight accumulation of a single convolution kernel based on pulse data. The calculation unit control module is used to control the weight accumulation of a single convolution kernel. The shift module is used to control the shift of pulse data between pulse calculation units in the pulse calculation unit array. The membrane potential calculation module is used to calculate the membrane potential based on the cumulative weights of all convolution kernels inside the pulse convolution layer.

[0103] In this embodiment, one pulse calculation unit (SCU) is set up for each convolution kernel. The pulse calculation unit selects whether to accumulate the input weights into the weight sum based on whether the pulse of the corresponding channel in the input register of the pulse convolution kernel is 1 or 0. Specifically, as shown in the example... Figure 4 As shown, the input pulse data is stored in the register inside the pulse calculation unit. Each pulse data contains pulse information of the neurons of the number of channels. When the control signal output by the calculation unit control module indicates the start of calculation, all channels are traversed. Based on whether the pulse of the corresponding channel in the register is 1 or 0, it is selected whether to accumulate the input weight value to the weight sum. After all channels have been traversed, the accumulated weight sum is output to the membrane potential calculation module, and the pulse data in the register is sent to other pulse calculation units.

[0104] In this embodiment, a pulsed convolution parallel computation structure is also designed, employing a pulsed convolution kernel computation array and a shift register. The convolution kernel is a 5*5 window, with one SCU placed at each position of the convolution kernel, for a total of 25 SCUs. The SCUs in each row are connected sequentially from right to left. The rightmost SCU receives pulses from the outside. Whenever an SCU completes its computation and outputs the sum of weights for all output channels, it sends the pulses latched in its internal register to the neurons on the left. The weights of all SCUs are then retrieved from the weight storage unit and distributed to all SCUs. The sum of weights calculated by all SCUs is then aggregated and output to the membrane potential calculation module.

[0105] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An event data and spiking neural network based target tracking method, characterized in that, The method comprises the following steps: Obtaining event data and performing time dimension compression processing on the event data to obtain sample data; Adjusting the structure of the SiamFC network to convert it into a spiking neural network, including: setting the output of each convolutional layer in the SiamFC network to be positive through an absolute value function; setting the bias of the convolution operation in each convolutional layer in the SiamFC network to be 0; changing the maximum pooling operation in the SiamFC network into evaluating the firing rate of neurons at each time step, i.e., calculating the absolute pulse firing rate accumulated over time, and allowing the pulses of the neuron with the maximum pulse firing rate in a pooling window to pass based on the absolute pulse firing rate, wherein the absolute pulse firing rate is calculated by the following method: f s (t) = f s (t-1) + x s (t)(T-t) where f s (t) denotes the firing rate of neuron s at time t, x s (t) denotes whether neuron s fired a spike at time t, x s (t) = 1 denotes firing, x s (t) = 0 denotes no firing, the firing rate of neuron s is updated if neuron s fired a spike, otherwise it is not updated, T denotes the length of the time window required for one inference; After pre-training the SiamFC network using the sample data, the obtained weights and biases are migrated to the spiking neural network to optimize the spiking neural network, including: in the channel dimension of each layer, the maximum activation value is used to normalize the weights and biases, and the normalized weights and biases are migrated to the spiking neural network to optimize the spiking neural network; Obtaining real-time event data and performing time step adjustment processing to obtain search data, and using the optimized spiking neural network to perform inference and comprehensive similarity estimation on the search data and template data to obtain real-time target tracking results.

2. The event data and spiking neural network based target tracking method of claim 1, wherein, The event data is asynchronously collected by a dynamic visual sensor, and the event data e includes position information (x, y) of each pixel point, event polarity p, and time t, and is expressed as e=[x, y, t, p] T The event polarity p includes positive events and negative events. The time dimension compression processing on the event data comprises: Event data within a given time period is compiled into an event packet, and the target pixel points (x) within the event packet are counted. k ,y k The number of positive events and the number of negative events Event data of different polarities are stored in different channels to obtain two-dimensional data containing the number of positive events and the number of negative events.

3. The event data and spiking neural network based target tracking method of claim 1, wherein, In the channel dimension of each layer, the maximum activation value is used to normalize the weights and biases, which can be expressed by the following formula: where i and j represent the dimension index, and l represents the layer index, represents the weight of the l-th layer in the i,j-th dimension, which is obtained by using the maximum activation value on each channel is normalized, and for the non-first layer, the normalized activation value must be multiplied by to restore the input to the value before the previous layer is normalized, and then the normalization of the current layer is performed to obtain the normalized weight represents the bias of the l-th layer in the j-th dimension, represents the normalized bias.

4. The event data and spiking neural network based target tracking method of claim 1, wherein, The time step adjustment on the real-time event data comprises: presetting a time step and a time step number N, accumulating the real-time event data in each time step according to the time step, marking the pixel value as 1 if the number of events in the preset N time steps is greater than 0, otherwise marking the pixel value as 0, and obtaining the search data.

5. The event data and spiking neural network based target tracking method of claim 1, wherein, Using the optimized spiking neural network to perform inference and comprehensive similarity estimation on the search data and template data comprises: The search data and the template data are input into two branch paths of the optimized spiking neural network, and the forward inference calculation is performed, the last layer activation function of each branch path accumulates the membrane potential in a given time and does not trigger the peak value, and two pulse feature maps corresponding to the two data are obtained at the same time, and the correlation between the membrane potentials corresponding to the two pulse feature maps is used to calculate the potential similarity estimation, which can be expressed by the following formula: wherein, denotes the pulse feature map corresponding to the template data at time step t, denotes the pulse feature map corresponding to the search data at time step t, M P (z, x) denotes the latent similarity estimate; The time correlation between the two pulse feature maps is also calculated, and the time similarity estimation is matched at each time step, which can be expressed by the following formula: Wherein, τ represents the response period; During the conversion of the SiamFC network into the spiking neural network, the error is inversely proportional to the time, and the comprehensive similarity estimation is obtained by combining the potential similarity estimation and the time similarity estimation, which can be expressed by the following formula: where f spike (z, x) represents the integrated similarity estimate, T represents the time period.

6. The event data and spiking neural network based target tracking method of claim 1, wherein, The inference process of the spiking neural network is accelerated by FPGA, and the spiking neural network is fixed-point quantized to convert the data that can be input into the FPGA, including: By setting v(t) = V(t) · 2 β As the neuron state voltage, W i = w i · 2 β As the synaptic weight, v thr = V thr · 2 β As the neuron threshold, the conversion from floating point to fixed point is performed, β represents the certificate of the scaling factor, V(t) represents the neuron original voltage, w i represents the neuron original weight, the formula is converted into the following fixed point form: If v(t) > v thr , fire pulse and reset v(t) = v(t) - v thr if v(t) < v min , reset v(t) = 0 wherein x i (t-1) represents the pulse input value of the i-th neuron of the previous layer at the t-1 time, and i has a value range of the number of neurons of the previous layer.

7. The event data and spiking neural network based target tracking method of claim 6, wherein, When the inference process of the pulse neural network is accelerated by the FPGA, the inference process includes data transmission and network calculation, wherein the data transmission part is used for transmitting the network pulse data from the PC to the FPGA, and transmitting the network pulse result calculated by the FPGA to the PC for display; The network calculation part is implemented on the FPGA, and after the network input pulse data, the pulse data is calculated and outputted through the specified number of layers, and the network calculation is concentrated in the pulse convolution layer, which is divided into a pulse convolution overall control module, a shift module, a calculation unit control module, a pulse calculation unit array, and a membrane potential calculation module; The pulse convolution overall control module is used for controlling the data calculation in the pulse convolution layer, the pulse calculation unit array is formed by the pulse calculation unit array, each pulse calculation unit is used for realizing the weight accumulation of a single convolution kernel based on the pulse data, the calculation unit control module is used for controlling the weight accumulation of the single convolution kernel, the shift module is used for controlling the shift of the pulse data between the pulse calculation units in the pulse calculation unit array, and the membrane potential calculation module is used for performing the membrane potential calculation based on the accumulated weight sum of all the convolution kernels in the pulse convolution layer.

8. The event data and spiking neural network based target tracking method of claim 7, wherein, One pulse calculation unit is arranged for each convolution kernel, the input pulse data is stored in the register in the pulse calculation unit, each pulse data contains the pulse information of the neurons in the channel, when the control signal outputted by the calculation unit control module indicates that the calculation starts, all channels are traversed, whether the input weight value is accumulated to the weight sum is selected according to whether the pulse in the corresponding channel in the register is 1 or 0, after the traversal of all channels is completed, the accumulated weight sum is outputted to the membrane potential calculation module, and the pulse data in the register is sent to other pulse calculation units.

Images