A high frame rate target tracking method based on deep impulse neural network

Abstract

This invention relates to a high frame rate target tracking method based on a deep spiking neural network. The method involves segmenting target events on an embedded platform within a given time period into an n-bin voxel grid, generating positive and negative event frames. These frames are then input into a YOLO neural network for spatial feature extraction, outputting the position and category information of all targets in each frame, forming spatial features. The positive and negative event frames are then input into a spiking convolutional neural network in chronological order for temporal feature extraction, outputting temporal information related to the tracked target. A target tracking network is constructed, and after spatiotemporal feature fusion of the spatial and temporal information, regression and classification calculations are performed to output the position and category of the tracked target at different times. This invention achieves fast and efficient target tracking on embedded, low-power platforms, effectively solving the problems of high energy consumption and low efficiency in high frame rate target recognition networks under complex environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision target tracking technology, and relates to a high frame rate target tracking method based on a deep spiking neural network. Background Technology

[0002] High frame rate target tracking and identification systems have a wide range of applications in the military field, particularly in complex battlefield environments. Thanks to the rise of big data-based machine learning technology in the information age, intelligent high frame rate target tracking technology has become a crucial component of current and future weapon systems. Therefore, theoretical and applied research on intelligent high frame rate target tracking is of significant theoretical and practical importance.

[0003] Object tracking is an important branch of computer vision. It involves selecting the location of a target within an input video and continuously tracking it. Object tracking is more complex than object detection, as it requires tracking a specific target across multiple frames of the video, necessitating the extraction of inter-frame information. The main challenges of object tracking stem from complex and dynamic scenes and rapidly changing targets, such as deformation, movement, or rotation of the target; blurring due to rapid target movement; color changes caused by varying lighting conditions; and target occlusion and re-enhancing. Therefore, methods such as correlation filtering and deep learning are widely used to address these challenges. However, current technologies still suffer from high energy consumption and low efficiency in high-frame-rate object recognition networks operating in complex environments. Summary of the Invention

[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose a high frame rate target tracking method based on deep spiking neural networks to achieve fast and efficient target tracking on embedded, low-power platforms. This effectively solves the problems of high energy consumption and low efficiency of high frame rate target recognition networks in complex environments and provides a feasible solution for novel detection technology.

[0005] The solution of this invention is: to propose a high frame rate target tracking method based on a deep spiking neural network, comprising the following steps:

[0006] The embedded platform target events within a given time period are segmented into an n-bin voxel grid. For each voxel grid, a positive event frame E is generated by recording the spatial locations of all positive and negative events that occur during this period. p and negative event frame E n The positive event refers to the activation event that occurs when the membrane potential of a neuron exceeds the threshold, and the negative event refers to the activation event that occurs when the membrane potential of a neuron drops below the threshold.

[0007] All positive event frames E obtained p and negative event frame E n The YOLO neural network is input to extract spatial features and outputs the location and category information of all targets in each frame of the image, which together form spatial features.

[0008] All positive event frames E obtained p and negative event frame E n The data is input into a pulsed convolutional neural network in chronological order to extract temporal features, and the output is temporal information related to the tracked target.

[0009] A target tracking network is constructed, including a spatiotemporal feature fusion module, a classifier, and a regressor. The spatiotemporal feature fusion module performs spatiotemporal feature fusion on the spatial features and the temporal information and outputs the results to the classifier and regressor. After regression calculation and classification calculation, the position and category of the tracked target at different times are output.

[0010] Furthermore, the YOLO neural network adopts a binarized and quantized YOLOv3-Tiny neural network, and the network structure consists of 24 network layers, including 13 convolutional layers, 6 pooling layers, 2 YOLO prediction layers, 2 feature fusion layers and 1 upsampling layer;

[0011] The YOLO neural network takes an image as input. The first part of the network consists of five convolutional layers and four pooling layers stacked alternately. It then splits into two branches. The first branch consists of five convolutional layers and two pooling layers stacked alternately to obtain shallow features. The second branch obtains deep features through a feature fusion layer. These deep features are then fused with the shallow features from the first branch through another feature fusion layer after passing through one convolutional layer and one upsampling layer. Finally, two convolutional layers are used to extract features. At the end of each branch, there is a YOLO prediction layer to calculate and output the position and category of the tracked target.

[0012] After training to obtain a usable YOLO model, the model's weight parameters are binarized, and a weight threshold is set. Weight parameters greater than the weight threshold are set to 1, and those less than the weight threshold are set to -1.

[0013] Furthermore, the YOLO neural network performs spatial feature extraction, including:

[0014] Binarization convolution is performed on the input image and intermediate features, where the intermediate features represent all feature maps obtained after convolution and pooling.

[0015] Binarization activation value quantization calculation is performed on intermediate features;

[0016] The target's location information is obtained by calculating the coordinates of the bounding box of the predicted target location, and the target's category information is obtained by calculating the category probability of the bounding box.

[0017] Furthermore, the binarized convolution calculation of the input image and intermediate features specifically involves:

[0018] The original convolution kernel weight matrix is ​​binarized, converting the original floating-point weights into binary weights.

[0019] The original convolution operation is broken down into 9 groups of binary convolution operations, which are then computed simultaneously.

[0020] The output feature maps of the nine groups of binary convolution operations are fused together to obtain the final convolution result.

[0021] Furthermore, the pulsed convolutional neural network includes three convolutional-pulse layers, each of which includes a convolutional layer and a pulsed layer based on the LIF model. The convolutional layer converts pulses into membrane potentials, which are then passed as input to the pulsed layer.

[0022] Furthermore, the pulsed convolutional neural network performs temporal feature extraction, including:

[0023] S301, The convolutional layer will generate positive event frames E p and negative event frame E n The peak value of activation events is converted into membrane potential; where the membrane potential V of the l-th convolutional layer neuron at time t is... t,l Represented as:

[0024] V t,l =H t-1,l +C(Z t,l-1 ),

[0025] Z t,l =f(V t,l -V th ),

[0026] H t,l =(αV t,l (1-Z) t,l ),

[0027]

[0028] Where f(·) is the Heaviside step function; V th α is the membrane potential threshold; α is the neuron's leakage factor; ∨ is the element-wise OR operator; C represents the convolutional layer with a kernel size of k×k; Z t,l H is the action potential of the l-th layer neuron at time t; t,l It is the cumulative membrane voltage of the neuron at time t; It is a positive event frame at time t; These are the positive and negative frames at time t;

[0029] S302. At the last time n, calculate the membrane potential corresponding to each time step in the last convolutional-pulse layer. The summation average as a time feature F T ;

[0030]

[0031] Where ψ is an operator consisting of batch normalization and a ReLU activation function, and C1 is the l-th convolutional layer;

[0032] S303. Perform channel-level quantization on the convolutional layer, extract the maximum and minimum values ​​of the parameters in the convolutional layer channel by channel, and use the maximum and minimum value quantization method to quantize the parameters of each channel from 32-bit floating point type to 8-bit or 2-bit integer type.

[0033] Furthermore, the spatiotemporal feature fusion module includes a temporal attention module, a cross-domain integrator, and a spatial attention module.

[0034] Furthermore, the output includes the position and category of the tracked target at different times, including:

[0035] S401. The temporal attention module generates temporal attention features based on the input temporal information. The temporal attention features are features calculated using a self-attention mechanism.

[0036] S402. The cross-domain integrator takes temporal attention features and spatial features as input, optimizes the temporal attention features using a self-attention mechanism, and then adds the spatial features and the optimized temporal attention features element-wise to obtain the fused spatial features F. G 'as follows:

[0037] F G ′=χF T ′+F G ,χ=σ(M(A([F G ,F T ′])))

[0038] Among them, F G For the spatial features of the input; F T ' represents the optimized temporal attention feature; M is a three-layer perceptron operator with a linear input layer, a ReLU activation function, and a linear output layer; A represents adaptive average pooling; χ is the intermediate feature matrix of the fusion process; σ is the sigmoid function;

[0039] S403, the spatial attention module will fuse the spatial features F G As input, the spatial features are further enhanced by the self-attention mechanism, and the enhanced spatial features are output. The enhanced spatial features are then input into the classifier and regressor for classification and regression calculations, respectively, and the position and category of the tracked target at different times are output.

[0040] Furthermore, the pulsed convolutional neural network also includes a compression module for performing tensor decomposition and quantization compression;

[0041] The tensor decomposition involves transforming the feature values ​​and weights in the pulsed convolutional neural network into tensor form and decomposing the tensors. The product of the decomposed sub-tensors is then used to approximate the original weights.

[0042] The quantization compression involves converting the elements in the decomposed subtensors from floating-point numbers to fixed-precision integers, and then encoding and storing the quantized integer values ​​in a more compact format.

[0043] Furthermore, the process of transforming the feature values ​​and weights in the pulsed convolutional neural network into tensor form and then decomposing the tensors specifically involves:

[0044] The neuron connection weight matrix of each layer in a spiking convolutional neural network is represented as a multidimensional tensor.

[0045] A high-order singular value decomposition algorithm is used to decompose the multidimensional tensor to extract feature values ​​and reduce data complexity, while preserving time dynamic information.

[0046] The decomposed tensors are then mapped back to the spiking convolutional neural network to sparsely represent the weights of the spiking convolutional neural network and update the weights in the neural network.

[0047] The advantages of this invention compared to the prior art are:

[0048] (1) The high frame rate moving target detection algorithm based on deep spiking neural networks in this embodiment of the invention treats video frames as independent images, uses image target detection algorithms to obtain detection results, and uses the temporal and contextual information of the video to correct the high frame rate detection results. Through the research on the target tracking architecture of deep spiking neural networks, fast and efficient target tracking on embedded platforms is achieved, effectively solving the problem of low accuracy of high frame rate target recognition networks in complex environments.

[0049] (2) The present invention extracts target tracking information for embedded platforms based on deep spiking neural networks. Through tensor decomposition and quantization compression, the computational complexity of the network and the space consumed by storage parameters are significantly reduced, minimizing the consumption of storage and computing resources in the network. The deep compression technology based on tensor quantization reduces the number of parameters in the deep spiking neural network, making the originally complex deep spiking neural network lightweight. Attached Figure Description

[0050] Figure 1 This is a flowchart of a high frame rate target tracking method based on a deep spiking neural network according to an embodiment of the present invention;

[0051] Figure 2 This is an architecture diagram of a high frame rate target tracking method based on a deep spiking neural network according to an embodiment of the present invention. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] Example 1

[0054] This embodiment is achieved through the following technical solution: A high frame rate target tracking method based on a deep spiking neural network is proposed, such as... Figure 1 As shown, it includes the following steps:

[0055] Step 1: Divide the target events of the embedded platform within a given time period into an n-bin voxel grid. For each voxel grid, generate positive event frames and negative event frames by recording the spatial locations of all positive and negative events that occur during this period. Positive events refer to activation events that occur when the membrane potential of a neuron exceeds a threshold. This activation usually indicates that the neuron has received sufficient stimulation to produce output. Negative events refer to activation events that occur when the membrane potential of a neuron drops below a threshold. This activation usually indicates that the neuron's output is suppressed.

[0056] Step 2: Obtain all positive event frames E p and negative event frame E n The YOLO neural network is input to extract spatial features and outputs the location and category information of all targets in each frame of the image, which together form spatial features.

[0057] In step two, the YOLO neural network adopts a binary quantized YOLOv3-Tiny neural network, and the network structure consists of 24 network layers, including 13 convolutional layers, 6 pooling layers, 2 YOLO prediction layers, 2 feature fusion layers and 1 upsampling layer.

[0058] The YOLO neural network takes an image as input. The first part of the network consists of five convolutional layers and four pooling layers stacked alternately. It then splits into two branches. The first branch consists of five convolutional layers and two pooling layers stacked alternately to obtain shallow features. The second branch obtains deep features through a feature fusion layer. These deep features are then fused with the shallow features from the first branch through another feature fusion layer after passing through a convolutional layer and an upsampling layer. Finally, two convolutional layers are used to extract features. At the end of each branch, there is a YOLO prediction layer to calculate and output the position and category of the tracked target.

[0059] After obtaining a usable YOLO model through training, the model's weight parameters are binarized, and a weight threshold is set. Weight parameters greater than the weight threshold are set to 1, and those less than the weight threshold are set to -1.

[0060] Spatial feature extraction is performed using a YOLO neural network, outputting the location and category information of all targets in each frame of the image. The specific method is as follows:

[0061] S201. Perform binarized convolution calculation on the input image and intermediate features, wherein the intermediate features represent all feature maps obtained after convolution and pooling.

[0062] In this embodiment, step S201 specifically involves: first, binarizing the original convolution kernel weight matrix to convert the original floating-point weights into binary weights. Then, splitting the original convolution operation into 9 groups of binarized convolution operations and performing calculations simultaneously. Finally, fusing the output feature maps of the 9 groups of binarized convolution operations to obtain the final convolution result.

[0063] S202, Perform binarization activation value quantization calculation on intermediate features;

[0064] S203. Calculate the coordinates of the bounding box of the predicted target location to obtain the target's location information, and calculate the category probability of the bounding box to obtain the target's category information.

[0065] Step 3: Obtain all positive event frames E p and negative event frame E n The data is input into a pulsed convolutional neural network in chronological order to extract temporal features, and the output is temporal information related to the tracked target.

[0066] The pulsed convolutional neural network includes three convolutional-pulse layers, each of which includes a convolutional layer and a pulsed layer based on the LIF model. The convolutional layer converts pulses into membrane potentials, which are then passed as input to the pulsed layer.

[0067] Temporal feature extraction is performed using a pulsed convolutional neural network, as follows:

[0068] S301, The convolutional layer will generate positive event frames E p and negative event frame E n The peak value of activation events is converted into membrane potential; where the membrane potential V of the l-th convolutional layer neuron at time t is... t,l Represented as:

[0069] V t,l =H t-1,l +C(Z t,l-1 ),

[0070] Z t,l =f(V t,l -V th ),

[0071] H t,l =(αV t,l (1-Z) t,l ),

[0072]

[0073] Where f(·) is the Heaviside step function; V th α is the membrane potential threshold; α is the neuron's leakage factor; ∨ is the element-wise OR operator; C represents the convolutional layer with a kernel size of k×k; Z t,l H is the action potential of the l-th layer neuron at time t; t,l It is the cumulative membrane voltage of the neuron at time t; It is a positive event frame at time t; These are the positive and negative frames at time t;

[0074] S302. At the last time n, calculate the membrane potential corresponding to each time step in the last convolutional-pulse layer. The summation average as a time feature F T ;

[0075]

[0076] Where ψ is an operator consisting of batch normalization and a ReLU activation function, and C1 is the l-th convolutional layer;

[0077] S303. Perform channel-level quantization on the convolutional layer, extract the maximum and minimum values ​​of the parameters in the convolutional layer channel by channel, and use the maximum and minimum value quantization method to quantize the parameters of each channel from 32-bit floating point type to 8-bit or 2-bit integer type.

[0078] like Figure 1 As shown, in this embodiment, the pulse convolutional neural network also includes a compression module for performing tensor decomposition and quantization compression;

[0079] The tensor decomposition involves transforming the feature values ​​and weights in the pulsed convolutional neural network into tensor form and decomposing the tensors. The product of the decomposed sub-tensors is then used to approximate the original weights.

[0080] In this embodiment, tensor decomposition specifically involves: first, representing the neuron connection weight matrix of each layer in the spiking convolutional neural network as a multidimensional tensor; then, using a high-order singular value decomposition algorithm, decomposing the multidimensional tensor to extract feature values ​​and reduce data complexity while retaining important temporal dynamic information; finally, mapping the decomposed tensor back to the deep spiking neural network to sparsely represent the weights of the spiking convolutional neural network and update the weights in the neural network.

[0081] The quantization compression involves converting the elements in the decomposed subtensors from floating-point numbers to fixed-precision integers, encoding the quantized integer values ​​in a more compact format, and storing each value with fewer bits.

[0082] Step 4: Construct a target tracking network, including a spatiotemporal feature fusion module, a classifier, and a regressor. The spatiotemporal feature fusion module performs spatiotemporal feature fusion on the spatial features and the temporal information and outputs the results to the classifier and regressor. After regression calculation and classification calculation, the position and category of the tracked target at different times are output.

[0083] In step four, the spatiotemporal feature fusion module includes a temporal attention module, a cross-domain integrator, and a spatial attention module.

[0084] The target tracking network outputs the position and category of the tracked target at different times, as follows:

[0085] S401. The temporal attention module generates temporal attention features based on the input temporal information. The temporal attention features are features calculated using a self-attention mechanism.

[0086] S402. Design a cross-domain integrator. The cross-domain integrator takes temporal attention features and spatial features as input, optimizes the temporal attention features using a self-attention mechanism, and then adds the spatial features and the optimized temporal attention features element-wise to obtain the fused spatial features F. G 'as follows:

[0087] F G ′=χF T ′+F G ,

[0088] χ=σ(M(A([F G ,F T ′])))

[0089] Among them, FG For the spatial features of the input; F T ' represents the optimized temporal attention feature; M is a three-layer perceptron operator with a linear input layer, a ReLU activation function, and a linear output layer; A represents adaptive average pooling; χ is the intermediate feature matrix of the fusion process; σ is the sigmoid function;

[0090] S403, the spatial attention module will fuse the spatial features F G As input, the spatial features are further enhanced by the self-attention mechanism, and the enhanced spatial features are output. The enhanced spatial features are then input into the classifier and regressor for classification and regression calculations, respectively, and the position and category of the tracked target at different times are output.

[0091] Figure 2 This is an architecture diagram of the high frame rate target tracking method based on a deep spiking neural network in this embodiment. First, the input positive event frame E... p and negative event frame E n As the membrane potential input of the spiking neural network, the temporal feature points are extracted and channel-level low-bit quantization is performed to output the temporal information of the target tracking network. At the same time, the YOLO neural network extracts spatial features and outputs the spatial features of the target tracking network. Through spatiotemporal feature fusion, tensor decomposition, and quantization compression, real-time tracking of high frame rate targets is achieved.

[0092] In this embodiment, a cross-domain integrator design is used to fuse spatial features and temporal information. An attention map is generated based on the input spatial features and temporal information and applied to the temporal cues. Therefore, low-entropy information in the temporal domain can be suppressed from both spatial and temporal perspectives. Then, the cross-domain integrator adds the input spatial features and the optimized temporal features element-wise. For the input spatial features F... G Optimized temporal attention features F T If ′, then the result after fusion is:

[0093] F G ′=χF T ′+F G ,

[0094] χ=σ(M(A([F G ,F T ′])))

[0095] Where M is a three-layer perceptron operator with a linear input layer, a ReLU activation function, and a linear output layer; A represents adaptive average pooling; χ is the intermediate feature matrix of the fusion process; and σ is the sigmoid function.

[0096] This invention can effectively perform synchronous video monitoring and multimedia recording, and effectively track targets using computer vision algorithms. It possesses the ability to verify and confirm test results and to be traceable afterward. This improves testing efficiency and coverage, makes the testing process traceable, and ensures the reliability of measurements.

[0097] The overall flowchart of this embodiment is as follows: Figure 1 As shown, computer vision technology based on SNN provides spatiotemporal information for tracking targets in this automated framework. Furthermore, through model compression optimization techniques such as tensor compression, the number and size of model parameters are greatly reduced, the model running speed and resource consumption are greatly reduced, and the overall performance of the model is improved, making high frame rate target tracking technology on embedded platforms more automatic and efficient.

[0098] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

[0099] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A deep-pulse neural network-based high-frame-rate target tracking method, characterized in that, Includes the following steps: The embedded platform target events within a given time period are divided into an n-bin voxel grid. For each voxel grid, a positive event frame is generated by recording the spatial locations of all positive and negative events that occur during this period. and negative event frames The positive event refers to the activation event that occurs when the membrane potential of a neuron exceeds the threshold, and the negative event refers to the activation event that occurs when the membrane potential of a neuron drops below the threshold. All positive event frames obtained and negative event frames The YOLO neural network is input to extract spatial features and outputs the location and category information of all targets in each frame of the image, which together form spatial features. All positive event frames obtained and negative event frames The data is input into a pulsed convolutional neural network in chronological order to extract temporal features, and the output is temporal information related to the tracked target. A target tracking network is constructed, including a spatiotemporal feature fusion module, a classifier, and a regressor. The spatiotemporal feature fusion module performs spatiotemporal feature fusion on the spatial features and the temporal information and outputs the result to the classifier and the regressor. After regression calculation and classification calculation, the position and category of the tracked target at different times are output. The YOLO neural network performs spatial feature extraction, including: Binarization convolution is performed on the input image and intermediate features, where the intermediate features represent all feature maps obtained after convolution and pooling. Binarization activation value quantization calculation is performed on intermediate features; The target's location information is obtained by calculating the coordinates of the bounding box of the predicted target location, and the target's category information is obtained by calculating the category probability of the bounding box.

2. The high frame rate target tracking method based on a deep spiking neural network according to claim 1, characterized in that, The YOLO neural network described above uses a binarized and quantized YOLOv3-Tiny neural network. The network structure consists of 24 network layers, including 13 convolutional layers, 6 pooling layers, 2 YOLO prediction layers, 2 feature fusion layers, and 1 upsampling layer. The YOLO neural network takes an image as input. The first part of the network consists of five convolutional layers and four pooling layers stacked alternately. It then splits into two branches. The first branch consists of five convolutional layers and two pooling layers stacked alternately to obtain shallow features. The second branch obtains deep features through a feature fusion layer. These deep features are then fused with the shallow features from the first branch through another feature fusion layer after passing through one convolutional layer and one upsampling layer. Finally, two convolutional layers are used to extract features. At the end of each branch, there is a YOLO prediction layer to calculate and output the location and category of the tracked target. After training to obtain a usable YOLO model, the model's weight parameters are binarized, and a weight threshold is set. Weight parameters greater than the weight threshold are set to 1, and those less than the weight threshold are set to -1.

3. The high frame rate target tracking method based on a deep spiking neural network according to claim 1, characterized in that, The specific steps of performing binary convolution calculation on the input image and intermediate features are as follows: The original convolution kernel weight matrix is ​​binarized, converting the original floating-point weights into binary weights. The original convolution operation is broken down into 9 groups of binary convolution operations, which are then computed simultaneously. The output feature maps of the nine groups of binary convolution operations are fused together to obtain the final convolution result.

4. The high frame rate target tracking method based on a deep spiking neural network according to claim 1, characterized in that, The pulsed convolutional neural network includes three convolutional-pulse layers, each of which includes a convolutional layer and a pulsed layer based on the LIF model. The convolutional layer converts pulses into membrane potentials, which are then passed as input to the pulsed layer.

5. A high frame rate target tracking method based on a deep spiking neural network according to claim 4, characterized in that, The pulsed convolutional neural network performs temporal feature extraction, including: S301, The convolutional layer will generate positive event frames. and negative event frames The peak value of the activation event is converted into the membrane potential; among which, the first A convolutional layer neuron at time membrane potential at time Represented as: in It is a Heaviside step function; The membrane potential threshold; It is a leakage factor of neurons; It is the element-wise OR operator; This represents a convolutional layer with a kernel size of k × k; It is time Time l Action potentials of layer neurons; It is time The cumulative membrane voltage of the neuron at that time; It is time Positive event frames at the time; It is time Positive and negative frames at the time; S302, at the last time At this point, calculate the membrane potential corresponding to each time step in the last convolutional-pulse layer. The summation average as a time feature ; in, It is a batch normalization and a Operators composed of activation functions, It is the first l One convolutional layer; S303. Perform channel-level quantization on the convolutional layer, extract the maximum and minimum values ​​of the parameters in the convolutional layer channel by channel, and use the maximum and minimum value quantization method to quantize the parameters of each channel from 32-bit floating point type to 8-bit or 2-bit integer type.

6. The high frame rate target tracking method based on a deep spiking neural network according to claim 5, characterized in that, The spatiotemporal feature fusion module includes a temporal attention module, a cross-domain integrator, and a spatial attention module.

7. A high frame rate target tracking method based on a deep spiking neural network according to claim 6, characterized in that, The output includes the position and category of the tracked target at different times, including: S401. The temporal attention module generates temporal attention features based on the input temporal information. The temporal attention features are features calculated using a self-attention mechanism. S402. The cross-domain integrator takes temporal attention features and spatial features as input, optimizes the temporal attention features using a self-attention mechanism, and then adds the spatial features and the optimized temporal attention features element-wise to obtain the fused spatial features. as follows: in, Spatial features as input; The optimized temporal attention features; M is a three-layer perceptron operator with a linear input layer, a ReLU activation function, and a linear output layer; Represents adaptive average pooling; It is the intermediate feature matrix of the fusion process; It is the sigmoid function; S403, the spatial attention module will integrate the spatial features. As input, the self-attention mechanism is used to enhance the discriminativeness of spatial features, and the enhanced spatial features are output. The enhanced spatial features are then input into the classifier and regressor for classification and regression calculations, respectively, and the position and category of the tracked target at different time points are output.

8. A high frame rate target tracking method based on a deep spiking neural network according to claim 5, characterized in that, The pulsed convolutional neural network also includes a compression module for performing tensor decomposition and quantization compression; The tensor decomposition involves transforming the feature values ​​and weights in the pulsed convolutional neural network into tensor form and decomposing the tensors, then replacing the original weights with the product of the decomposed sub-tensors. The quantization compression involves converting the elements in the decomposed subtensors from floating-point numbers to fixed-precision integers, and then encoding and storing the quantized integer values ​​in a more compact format.

9. A high frame rate target tracking method based on a deep spiking neural network according to claim 8, characterized in that, The process of transforming the feature values ​​and weights in the pulsed convolutional neural network into tensor form and then decomposing the tensors specifically involves: The neuron connection weight matrix of each layer in a spiking convolutional neural network is represented as a multidimensional tensor. A high-order singular value decomposition algorithm is used to decompose the multidimensional tensor to extract feature values ​​and reduce data complexity, while retaining time dynamic information. The decomposed tensors are then mapped back to the spiking convolutional neural network to sparsely represent the weights of the spiking convolutional neural network and update the weights in the neural network.

Images