An event camera based pulse neural network tracking method and system

Abstract

The application provides an event camera-based pulse neural network tracking method and system. Multi-time step event data and low-resolution grayscale images output by an event camera are acquired, and the event data are combined with time sequence and polarity to form a multi-channel event tensor; dynamic event features in the event tensor and static image features in the grayscale images are extracted respectively, and feature fusion is performed through channel splicing and convolution operation; the fused features are input into a super-resolution decoder, up-sampling of the images is realized through transposed convolution operation, and a high-resolution moving target image is generated; the high-resolution image is input into a pulse neural network feature extraction module based on LIF neurons, features of a target template and a search area are extracted, and a target position is determined through cross-correlation operation to realize tracking. The application can effectively improve the spatial resolution of the event camera output image and the target positioning accuracy, and realizes robust target tracking in a complex dynamic environment on a low-power edge device.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and brain-like perception, belonging to the interdisciplinary fields of image processing, event vision and target tracking, and particularly to a spiking neural network tracking method and system based on an event camera. Background Technology

[0002] With the rapid development of artificial intelligence and neuromorphic computing technologies, event visual perception, as a crucial component of high-dynamic vision systems, has demonstrated significant advantages in fields such as drones, autonomous driving, and industrial inspection. Event cameras, by asynchronously outputting pixel-level brightness change event streams, possess characteristics such as microsecond-level response, high dynamic range, low power consumption, and low data redundancy, providing a new technical path for real-time vision tasks in complex environments. Among related technologies, a multimodal perception and processing system has been constructed through the collaborative operation of event data and grayscale images. Specifically, this technology covers the entire process from event stream acquisition, multi-timestep event frame construction, multimodal feature extraction to target reconstruction and tracking, including key steps such as event feature extraction, image feature enhancement, feature fusion, super-resolution decoding, and a tracking mechanism driven by a spiking neural network (SNN).

[0003] However, existing event camera visual tracking methods directly use low-resolution images and sparse event streams for target perception, failing to fully exploit their dynamic information in the temporal dimension. This can lead to blurred target contours, loss of detail, and decreased positioning accuracy. Furthermore, in complex scenarios such as drastic lighting changes and multiple overlapping targets, tracking drift and a significantly increased failure rate may occur, impacting the tracking stability and robustness of UAV platforms under high dynamic and low-power constraints. In addition, traditional methods lack effective modeling of spatiotemporal consistency and edge preservation capabilities during feature fusion and reconstruction, making it difficult to meet the dual requirements of real-time performance and computational efficiency for edge computing devices. The existing technical problems are as follows:

[0004] (1) Under constraints such as low power consumption and high dynamic range, traditional target tracking methods based on frame images are difficult to adapt to the application requirements of mobile platforms such as UAVs, and lack tracking robustness and real-time performance.

[0005] (2) The event stream and grayscale image frame spatial resolution output by the existing event camera are low, resulting in the loss of image details and making it difficult to meet the requirements of image quality and perception accuracy in high dynamic vision tasks such as target tracking.

[0006] (3) Sparse event data contains rich dynamic information in the time dimension, but existing methods are unable to effectively extract and utilize its spatiotemporal features, which limits the further improvement of target perception and tracking accuracy. Therefore, it is urgent to study how to deeply mine the spatiotemporal dynamic characteristics of event data. Summary of the Invention

[0007] The present invention aims to at least partially solve one of the technical problems in the related art.

[0008] Therefore, the first objective of this invention is to propose a spiking neural network tracking method based on an event camera, construct an information fusion mechanism with temporal consistency modeling capability, achieve high-quality reconstruction of low-resolution images, and effectively improve the accuracy and robustness of target tracking.

[0009] The second objective of this invention is to propose a spiking neural network tracking system based on an event camera.

[0010] To achieve the above objectives, a first aspect of the present invention proposes a spiking neural network tracking method based on an event camera, comprising:

[0011] S1, acquire multi-timestep event data and low-resolution grayscale image output by the event camera, and combine the event data into a multi-channel event tensor according to time order and polarity;

[0012] S2, extract the dynamic event features from the event tensor and the static image features from the grayscale image respectively, and perform feature fusion through channel splicing and convolution operations to form a spatiotemporally consistent feature representation;

[0013] S3 inputs the fused features into the super-resolution decoder, and upsamples the image through transposed convolution to generate a high-resolution moving target image;

[0014] S4. Input the high-resolution image into the LIF neuron-based spiking neural network feature extraction module to extract the features of the target template and the search area, and determine the target position through cross-correlation operation to achieve tracking.

[0015] In one embodiment of the present invention, the step of acquiring multi-timestep event data and low-resolution grayscale images output by the event camera, and combining the event data into a multi-channel event tensor according to time order and polarity, includes:

[0016] S11, read the event images corresponding to the time steps from the ON and OFF subdirectories respectively, a total of 8 images, and stitch them together in chronological order into a tensor of shape [8, H, W], where 4 time steps correspond to positive and negative polarity event images respectively;

[0017] S12, normalize all image data, and normalize the intensity values ​​of event images and grayscale images to the range of [0, 1] to enhance the convergence stability and generalization ability of the model.

[0018] In one embodiment of the present invention, the step of extracting dynamic event features from the event tensor and static image features from the grayscale image, and fusing features through channel concatenation and convolution operations to form a spatiotemporally consistent feature representation, further includes:

[0019] S21 extracts dynamic event features through three consecutive two-dimensional convolution operations. Each convolution is followed by batch normalization and ReLU activation function, and the final output is a feature map with 64 channels.

[0020] S22 extracts static image features through a parallel multi-scale convolutional structure, including three sets of convolutional channels with kernel sizes of 3×3, 5×5 and 7×7. After concatenation, the key region response is enhanced through an attention mechanism.

[0021] In one embodiment of the present invention, the step of inputting the fused features into the super-resolution decoder and upsampling the image through a transposed convolution operation to generate a high-resolution moving target image further includes:

[0022] S31 uses transposed convolution with a kernel size of 4, a stride of 2, and padding of 1 to enlarge the image space size to twice its original size.

[0023] S32 enhances nonlinear modeling capabilities through batch normalization and ReLU activation functions, and outputs a 1-channel grayscale image through the final convolutional layer.

[0024] In one embodiment of the present invention, it further includes:

[0025] S5, perform image quality assessment on the high-resolution moving target image, use the Sobel operator to calculate the image gradient magnitude map, and constrain gradient sparsity through L1 norm to further optimize the image edge preservation capability.

[0026] To achieve the above objectives, a second aspect of the present invention provides a spiking neural network tracking system based on an event camera, comprising:

[0027] The event data acquisition module is used to acquire multi-time step event data and low-resolution grayscale images output by the event camera, and combine the event data into a multi-channel event tensor according to time order and polarity.

[0028] The feature extraction and fusion module is used to extract dynamic event features from the event tensor and static image features from the grayscale image, respectively, and to fuse the features through channel splicing and convolution operations to form a spatiotemporally consistent feature representation.

[0029] The image super-resolution reconstruction module is used to input fused features into the super-resolution decoder, and to achieve image upsampling through transposed convolution operation to generate high-resolution moving target images;

[0030] The target tracking feature processing module is used to input high-resolution images into the LIF neuron-based spiking neural network feature extraction module, extract the features of the target template and the search area, and determine the target position through cross-correlation calculation to achieve tracking.

[0031] The method and system of this invention achieve high-quality super-resolution reconstruction of moving targets under the constraint of low spatial resolution event camera data, significantly improving the accuracy and robustness of target tracking, while maintaining low power consumption and real-time performance.

[0032] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0034] Figure 1 This is a flowchart of a spiking neural network tracking method based on an event camera according to an embodiment of the present invention;

[0035] Figure 2 This is a system architecture diagram according to an embodiment of the present invention;

[0036] Figure 3 This is a data processing flowchart of a system according to an embodiment of the present invention;

[0037] Figure 4 This is a structural diagram of a spiking neural network tracking system based on an event camera according to an embodiment of the present invention. Detailed Implementation

[0038] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] The following description, with reference to the accompanying drawings, describes a pulse neural network tracking method and system based on an event camera according to an embodiment of the present invention.

[0041] Example 1

[0042] Figure 1 This is a flowchart of a spiking neural network tracking method based on an event camera according to an embodiment of the present invention, as shown below. Figure 1 As shown, it includes:

[0043] S1: Acquire multi-timestep event data and low-resolution grayscale images output by the event camera, and combine the event data into a multi-channel event tensor according to time order and polarity.

[0044] Specifically, this step aims to acquire multi-timestep event data and low-resolution grayscale images output by the event camera, and combine the event data into a multi-channel event tensor according to time order and polarity. This is a key step in the data preprocessing and feature input construction of the entire algorithm system.

[0045] In one embodiment of the present invention, the event camera outputs an event stream asynchronously. Each event includes a timestamp, pixel coordinates, and polarity (ON / OFF), reflecting the dynamic process of pixel brightness changes. To facilitate subsequent spiking neural network (SNN) processing, the event data needs to be aggregated at fixed time steps (T=4) to form event frames. Within each time step, the event data is divided into an ON event map and an OFF event map, representing the event distribution of brightness increases and decreases, respectively. By reading and stitching the ON and OFF event images of the four time steps in chronological order, an event tensor of shape [8, H, W] is finally formed, where the eight channels correspond to the positive and negative polarity event maps of the four time steps, and H and W are the height and width of the image. This tensor structure can effectively preserve the temporal dynamic characteristics of the events, providing a unified input format for subsequent feature extraction and information fusion.

[0046] Furthermore, the construction of the event tensor depends on the setting of the time step T. In this invention, T=4, meaning that every 4 time steps constitute a processing unit, ensuring the continuity and representativeness of event information in the time dimension. The event image normalization process employs a standard image intensity normalization method, mapping the pixel values ​​of all event images and grayscale images to the range [0,1] to unify the intensity distribution and improve the stability and generalization ability of the model training. The event tensor has 8 channels, and its spatial size is consistent with the original image, facilitating channel stitching and fusion with image features in subsequent modules.

[0047] This step is applicable to real-time target tracking tasks on UAV platforms. Event cameras, due to their high dynamic range and low latency, are particularly suitable for use in scenarios with drastic lighting changes and fast-moving targets. By synchronously processing event data and grayscale images and constructing a multi-channel event tensor, high-quality, structured input data is provided for subsequent moving target super-resolution reconstruction and SNN tracking algorithms, thereby improving the system's perception capabilities and tracking robustness in complex environments.

[0048] This step, through structured processing of event data, aligns the event stream with image frames in both time and space, laying the foundation for multimodal feature fusion. The multi-channel event tensor not only preserves the temporal dynamics of the events but also enhances the ability to discriminate motion directions through polarity separation, contributing to improved accuracy and stability of subsequent network modeling of target motion trajectories.

[0049] Furthermore, S1 includes:

[0050] S11: Read the event images corresponding to the time steps from the ON and OFF subdirectories, for a total of 8 images, and stitch them together in chronological order into a tensor of shape [8, H, W], where 4 time steps correspond to positive and negative polarity event images respectively.

[0051] Specifically, this step involves reading the event images for the corresponding time steps from the ON and OFF subdirectories of the event camera and stitching them together in chronological order into a tensor of shape [8, H, W], containing four time steps, each corresponding to a positive and negative polarity event image. This step is a key data preprocessing step in the entire moving target super-resolution reconstruction and target tracking algorithm, and its technical implementation is based on the asynchronous output mechanism of the event camera and the multi-time step event frame construction strategy.

[0052] In some implementations, the event camera outputs an event stream asynchronously, with each event recording its pixel location, polarity (ON or OFF), and timestamp. To facilitate neural network processing, the event stream is aggregated over fixed time steps (T=4) to generate event frames. Each time step contains two polarity event maps, representing the event density maps for brightness increases (ON) and decreases (OFF) within that time window. Therefore, each time step corresponds to two event images, for a total of four time steps, resulting in eight event images.

[0053] The specific operation is as follows: First, based on the timestamp information from the event camera, the event data is divided into four consecutive time windows, each with a length of T = 4 milliseconds (adjustable according to actual task requirements). Within each time window, the pixel counts of ON and OFF events are counted, generating corresponding event images. Event images are typically represented as single-channel grayscale images, with pixel values ​​reflecting the frequency or intensity of the event, usually ranging from [0, 255]. To ensure consistency of model input, all event images are normalized to the [0, 1] interval, conforming to the input standards of deep learning models.

[0054] Furthermore, the eight event images are concatenated channel by channel in chronological order to form a four-dimensional tensor of shape [8, H, W], where 8 represents the number of time-polarity channels, and H and W are the height and width of the image, respectively. This tensor serves as the input to the event feature extractor for subsequent feature extraction and fusion processing.

[0055] The technical value of this step lies in constructing multi-time-step event frame tensors by structuring event data, providing the neural network with a temporally consistent input representation, thereby effectively capturing the target's motion trajectory and changing trends over time. In target tracking tasks, this method can enhance the model's ability to perceive fast-moving targets, improve tracking robustness and real-time performance, and is particularly suitable for complex scenarios with drastic changes in lighting and high target speeds.

[0056] S12, normalize all image data, and normalize the intensity values ​​of event images and grayscale images to the range of [0, 1] to enhance the convergence stability and generalization ability of the model.

[0057] Specifically, this step aims to normalize the intensity values ​​of the event image and the grayscale image, mapping them uniformly to a standardized numerical range of [0, 1], thereby improving the convergence stability and generalization ability of the subsequent spiking neural network (SNN) model. In some implementations, this normalization operation is based on the min-max normalization method.

[0058] In practical implementation, considering that event images output by event cameras are typically sparse binary or count-type data (such as ON / OFF event counts), their intensity value range may be much smaller than the 0-255 range of grayscale images. Therefore, the normalization operation needs to be adapted to the numerical characteristics of different modal images. For example, the intensity value of an event image may be 0 to a number of event counts (such as 0-100), while the grayscale image is an 8-bit integer of 0-255. During the normalization process, channel-level or batch-level normalization strategies can be optionally adopted to ensure that the model has consistent perception capabilities under different input conditions.

[0059] Furthermore, normalization plays a crucial role in this invention. Since event images and grayscale images differ significantly in temporal resolution, dynamic response mechanisms, and intensity distribution, failure to normalize them can lead to gradient oscillations and slow convergence during model training due to uneven input data distribution. By normalizing all image data to the [0, 1] interval, not only is the model's sensitivity to input scale reduced, but the efficiency of feature extraction modules (such as event feature extractors and image feature extractors) in fusing multimodal data is also improved.

[0060] In practical applications, this step is typically completed during the image preprocessing stage, suitable for edge computing environments on UAV platforms. Normalized image data, used as input to subsequent convolutional neural networks, helps improve the model's robustness in complex lighting and dynamic scenes. Especially in low-resolution, high-noise environments, it effectively enhances the recognizability of target contours, providing a stable data foundation for super-resolution reconstruction and tracking of moving targets.

[0061] S2, extract the dynamic event features from the event tensor and the static image features from the grayscale image respectively, and perform feature fusion through channel splicing and convolution operations to form a spatiotemporally consistent feature representation.

[0062] Specifically, this step involves the fusion of event tensors and grayscale image features, a crucial step in achieving spatiotemporal consistency modeling of moving targets. In some implementations, the dynamic event features output by the event feature extractor and the static image features extracted by the image feature extractor each have dimensions of [B, 64, H, W], where 64 is the number of feature channels, and H and W are the height and width of the image, respectively. To achieve effective integration of multimodal information, the event features and image features are first concatenated along the channel dimension to form a fused feature tensor with 128 channels, i.e., [B, 128, H, W]. This concatenation operation preserves the temporal dynamic characteristics of the event data and the spatial structure information of the image data, providing a foundation for subsequent feature interaction.

[0063] Furthermore, to reduce feature dimensionality and improve fusion efficiency, a 1×1 convolution operation is used to compress the channels of the concatenated features. The 1×1 convolution has the advantages of small parameter count and high computational efficiency, enabling cross-channel information integration and feature mapping. In this invention, the output channel number of the 1×1 convolution is set to 64, consistent with the original channel number of event features and image features, thereby achieving feature space alignment and efficient information fusion. Batch normalization and ReLU activation function are applied after the convolution operation to enhance the model's non-linear expressive power and accelerate the training process.

[0064] Optionally, to improve the training stability and feature representation capability of the model, a residual connection mechanism is introduced, which adds the original stitched features to the features processed by 1×1 convolution element-wise to form the final fused features. This fused feature not only retains the temporal sensitivity of the event data but also incorporates the structural information of the image data, thereby constructing a feature representation with spatiotemporal consistency and providing high-quality input for subsequent super-resolution reconstruction and target tracking.

[0065] In practical applications, this step is suitable for real-time target tracking systems on UAV platforms, especially in high-dynamic, low-light, or fast-moving scenarios, where it can effectively improve the robustness and accuracy of target perception. Through this fusion mechanism, the model can more accurately capture the target's trajectory and contour information when processing low-resolution images and sparse event streams, thereby significantly improving tracking performance.

[0066] Furthermore, S2 includes:

[0067] S21 extracts dynamic event features through three consecutive two-dimensional convolutional operations. Each convolutional layer is followed by batch normalization and ReLU activation function, and the final output is a feature map with 64 channels.

[0068] Specifically, this step is a key processing step in the event feature extractor. Its technical implementation principle is based on deep convolutional neural networks to extract and enhance the spatiotemporal features of event data layer by layer. The event data output by the event camera records pixel brightness changes in the form of asynchronous pulses, possessing high temporal resolution and low data redundancy. To effectively model the dynamic information in the event stream, this step employs three consecutive layers of two-dimensional convolutional operations, followed by batch normalization (BN) and the ReLU activation function after each convolutional layer to enhance the model's nonlinear expressive power and accelerate the training process. The input event tensor has dimensions [B, 8, H, W], where the 8 channels correspond to the ON / OFF event maps at 4 time steps, B is the batch size, and H and W are the height and width of the image.

[0069] In practical implementations, the parameter configuration of each 2D convolutional layer can be adjusted according to the actual task requirements. In some implementations, the first convolutional layer has a kernel size of 3×3, a stride of 1, 8 input channels, and 16 output channels; the second convolutional layer also has a kernel size of 3×3, 16 input channels, and 32 output channels; the third convolutional layer has a kernel size of 3×3, 32 input channels, and 64 output channels. All convolutional layers use zero padding (padding=1) to maintain the spatial size of the feature map. Batch normalization is performed after each convolutional layer, and its standardized parameters (mean and variance) are dynamically updated during training, which helps alleviate the gradient vanishing problem and improve model stability. The ReLU activation function introduces non-linearity, enhancing the model's ability to fit complex event patterns.

[0070] The feature map output by this step has a size of [B, 64, H, W] and 64 channels, which can fully express the motion trajectory, edge changes, and temporal dynamic features in the event data. In practical applications, this module is deployed on an UAV edge computing platform to process multi-time-step event data acquired by the event camera in real time, providing high-dimensional, highly discriminative dynamic feature representations for subsequent image reconstruction and target tracking. Through this structural design, this invention effectively improves the feature extraction efficiency and representation quality of event data, laying a solid foundation for achieving high-precision, low-power moving target tracking.

[0071] S22 extracts static image features through a parallel multi-scale convolutional structure, including three sets of convolutional channels with kernel sizes of 3×3, 5×5 and 7×7. After concatenation, the key region response is enhanced through an attention mechanism.

[0072] In this invention, the image feature extractor employs a parallel multi-scale convolutional structure to extract static texture features and structural information from low-resolution grayscale images. This structure uses convolutional kernels with different receptive fields (3×3, 5×5, and 7×7) to capture local features at different scales in the image, thereby enhancing the model's ability to perceive target contours, edges, and details. In some implementations, each convolutional channel uses a standard convolution operation, with the input being a single-channel grayscale image tensor of size [B, 1, H, W], where B represents the batch size, and H and W are the image height and width. The number of output channels for each convolutional layer is consistent at 64 to ensure alignment of channel dimensions for subsequent feature fusion.

[0073] Specifically, 3×3 convolutional kernels have a smaller receptive field, suitable for extracting local edges and high-frequency texture features in images; 5×5 convolutional kernels can capture medium-scale structural information, such as the contours and local shapes of objects; and 7×7 convolutional kernels have a larger receptive field, used to extract global contextual information, helping to enhance the model's understanding of the overall structure of the object. All convolutional operations in each channel use the same stride (stride=1) and padding (padding=same) to maintain consistent spatial dimensions of the output feature maps. After the convolutional operation, an attention mechanism (such as the SE attention module or CBAM) is introduced to weightedly fuse features at different scales, highlighting the responses of key regions and improving the discriminative power of feature representation.

[0074] This step plays a crucial role in the overall technical solution. The multi-scale image features and event features output from this step are stitched together and dimensionality reduced in the feature fusion module, providing complementary structural and textural information for subsequent super-resolution reconstruction. By introducing a multi-scale convolutional structure, the model can effectively preserve the structural details of the target even in low-resolution images, thereby improving the quality of the reconstructed image and the robustness of target tracking. This design meets the comprehensive requirements of UAV platforms for lightweight design, low power consumption, and high precision, and has significant engineering practical value.

[0075] S3, the fused features are input into the super-resolution decoder, and the image is upsampled through transpose convolution operation to generate a high-resolution moving target image.

[0076] Specifically, in the technical solution of this invention, the step of "inputting the fused features into the super-resolution decoder and upsampling the image through transposed convolution to generate a high-resolution moving target image" is one of the core steps in realizing super-resolution reconstruction of moving targets. This step expands the spatial dimension of the fused multimodal feature map through the transposed convolution operation in deep learning, thereby recovering a high-resolution moving target image and providing high-quality visual input for subsequent target tracking.

[0077] In this step, the input dimensions of the fused features are [B, 64, H, W], where B represents the batch size, and H and W are the height and width of the feature map. The super-resolution decoder uses a single-layer transposed convolutional structure with the following configuration parameters: kernel size of 4×4, stride of 2, and padding of 1. This configuration doubles the spatial size of the output image in each dimension, resulting in an output size of [B, 1, H×2, W×2]. The transposed convolution operation achieves upsampling through backpropagation, essentially interpolating and expanding the information in the low-resolution feature map in the spatial domain to generate finer image details.

[0078] The transposed convolutional layer outputs a single channel, corresponding to a grayscale image. This layer is typically followed by batch normalization and ReLU activation to enhance the model's non-linear expressiveness and accelerate the training process. Furthermore, the super-resolution decoder has an upsampling scale of 2, meaning the output image has twice the resolution of the input image. In practical deployments, this module has low computational complexity, making it suitable for running on edge devices (such as drones) and meeting real-time and low-power requirements.

[0079] This step is primarily applied to target tracking tasks involving small targets in high-dynamic, low-light, or fast-moving scenarios. Since the images output by the event camera have low resolution, the super-resolution reconstruction in this step significantly improves the sharpness of the target contour and the ability to preserve details, thereby enhancing the robustness of target detection and tracking. Especially in complex scenarios such as small target recognition and occlusion recovery, high-resolution images can provide richer edge and texture information, helping to improve tracking accuracy.

[0080] This step achieves image upsampling through transposed convolution, effectively restoring the lost details in low-resolution images while preserving the dynamic information contained in the event data. Combined with the multimodal features output by the feature fusion module, the super-resolution decoder can generate high-resolution images with spatiotemporal consistency, providing high-quality input for subsequent spiking neural network (SNN) feature extraction and target localization, thereby improving the performance and stability of the entire tracking system.

[0081] Furthermore, S3 includes:

[0082] S31 uses a transposed convolution operation with a kernel size of 4, a stride of 2, and padding of 1 to enlarge the image space size to twice its original size.

[0083] S32 enhances nonlinear modeling capabilities through batch normalization and ReLU activation functions, and outputs a 1-channel grayscale image through the final convolutional layer.

[0084] Specifically, in some implementations, the transposed convolution operation in the super-resolution encoder is a key step in achieving low-resolution image upsampling. Its technical principle is based on the mathematical mechanism of deconvolution, also known as transposed convolution, which is used to enlarge the feature map in spatial dimensions, thereby restoring high-resolution details of the image. This step receives a fused feature map from the feature fusion module, with input feature dimensions of [B, 64, H, W], where B represents the batch size, and H and W are the height and width of the current feature map. The transposed convolutional layer is configured with a standard parameter combination of a 4×4 kernel size, a stride of 2, and padding of 1. This configuration enables upsampling of the image size by a factor of 2, i.e., an output size of [B, 64, 2H, 2W], providing a foundation for subsequent generation of high-resolution grayscale images.

[0085] In terms of specific operation, transposed convolution achieves spatial expansion through backpropagation convolution operations. Essentially, it maps each feature point in the low-resolution feature map to multiple locations in the high-resolution space, thereby performing interpolation and detail enhancement in the spatial dimension. In this invention, this layer uses a standard convolution kernel initialization method, and the weight parameters are set through He initialization to alleviate the gradient vanishing problem and improve training efficiency. The number of channels in the convolution kernel is 64, and the number of output channels also remains 64 to maintain consistency in the feature dimension and facilitate the integration of subsequent processing modules.

[0086] The stride (stride=2) of the transposed convolution determines the magnification factor of the image in the width and height directions, while padding (padding=1) ensures that the output size is twice the input size, rather than twice minus 2. This configuration conforms to common upsampling strategies in image super-resolution tasks (such as the 4×4 convolution kernel and 2x upsampling structure widely used in models like SRGAN and ESRGAN), effectively balancing computational complexity and image quality. Furthermore, batch normalization and the ReLU activation function are applied to the output of this layer to enhance non-linear expressiveness and improve the restoration of image details.

[0087] This transposed convolution operation, deployed on an edge computing platform, processes low-resolution images and event stream data captured by event cameras in real time to generate high-resolution images of moving targets, thereby improving the accuracy and robustness of target tracking. Especially in highly dynamic scenes, such as fast-moving targets, drastic lighting changes, or low-light environments, this step effectively recovers target edge and texture information, providing high-quality input for subsequent spiking neural network (SNN) feature extraction and correlation matching.

[0088] Furthermore, this step plays a connecting role in the overall technical solution and is a key link connecting feature fusion and image reconstruction. Through the upsampling operation of transposed convolution, the model can achieve high-resolution reconstruction of moving targets while maintaining computational efficiency, thereby significantly improving the accuracy and stability of target perception and meeting the strict requirements of the drone platform for real-time performance and low power consumption.

[0089] S4. Input the high-resolution image into the spiking neural network feature extraction module based on LIF neurons to extract the features of the target template and the search area, and determine the target position through cross-correlation operation to achieve tracking.

[0090] Specifically, this step involves inputting the high-resolution image into a spiking neural network (SNN) feature extraction module based on Leaky Integrate-and-Fire (LIF) neurons to extract the features of the target template and the search area, and determining the target position through cross-correlation operation, thereby achieving the tracking of moving targets. This step is the core perception and decision-making link in the entire tracking algorithm. Its technical implementation is based on a brain-inspired computing architecture and has the characteristics of low power consumption, high temporal sensitivity, and strong robustness.

[0091] In some implementations, the SNN feature extraction module adopts a convolutional-based spiking neural network structure. Its basic architecture借鉴了AlexNet的拓扑结构,包含五个卷积层,其中卷积核尺寸分别为11×11、5×5、3×3、3×3和3×3,输出通道数依次为96、256、384、384和256。各层卷积操作后均接入LIF神经元模型,用于模拟生物神经元的脉冲发放机制,从而实现对图像特征的时序编码与动态响应。

[0092] Furthermore, the feature extraction module adopts a Siamese network structure to separately extract the features of the target template image and the search area image of the current frame. The two branches share the weights of the convolutional layers to ensure the consistency of the feature space. The size of the extracted feature map is [B, 256, H', W'], where B is the batch size, and H' and W' are the spatial dimensions of the feature map. Subsequently, a response score map is generated through per-channel cross-correlation operation (Cross-Correlation).

[0093] In practical applications, this step is applicable to real-time target tracking tasks of small targets in complex dynamic environments. Especially in scenarios with drastic changes in lighting and fast target movement speeds, it can effectively improve the tracking robustness and stability. By introducing the spiking neuron model, not only the computational energy consumption of the model is reduced, but also the temporal perception ability of event data is enhanced, thus enabling efficient deployment on low-power edge devices. It should be noted that there is an unclear expression "借鉴了AlexNet的拓扑结构" in the original Chinese text. I have translated it as literally as possible. If there is a more accurate expression, it can be adjusted accordingly.

[0094] In summary, this step, which extracts spatiotemporal features from high-resolution images using a spiking neural network driven by LIF neurons and combines cross-correlation operations to achieve precise target location, is one of the key innovations of this invention in the fusion application of event vision and target tracking.

[0095] The target tracking method based on spiking neural networks and event cameras in this invention effectively improves the reconstruction quality and target tracking robustness of low-resolution event camera images, and achieves accurate perception and stable tracking of moving targets in high dynamic and low power consumption scenarios.

[0096] Also includes:

[0097] S5 performs image quality assessment on high-resolution moving target images, uses the Sobel operator to calculate the image gradient magnitude map, and constrains gradient sparsity through the L1 norm to further optimize the image edge preservation capability.

[0098] Specifically, in this invention, the step of image quality assessment for high-resolution moving target images involves using the Sobel operator to calculate the image gradient magnitude map and constraining gradient sparsity using the L1 norm to further optimize the image edge preservation capability. The core of this step lies in using image gradient information to quantitatively evaluate the edge structure of the reconstructed image and improving the structural fidelity of the image during super-resolution reconstruction through sparsity constraints.

[0099] The Sobel operator is a classic edge detection operator that extracts edge information by calculating the gradient magnitudes of an image in the horizontal and vertical directions. Specifically, the Sobel operator consists of two 3×3 convolutional kernels, used to calculate the gradient responses in the x and y directions, respectively. In this invention, the Sobel operator is applied to the reconstructed high-resolution image to generate a gradient magnitude map.

[0100] Furthermore, to enhance the sharpness of image edges and suppress noise, this invention introduces the L1 norm to impose sparsity constraints on the gradient magnitude map. The L1 norm has sparsification properties, which can cause the response of non-edge regions in the gradient magnitude map to approach zero, thereby highlighting true edge information.

[0101] This step is primarily used in UAV tracking systems to evaluate and optimize the quality of super-resolution reconstructed images. Event camera output images often have low resolution and are prone to edge information loss. However, by combining Sobel gradients with L1 sparsity constraints, the target edge structure can be effectively recovered, improving image discriminability against complex backgrounds. Especially in low-light, high-speed motion, or occluded scenarios, this method significantly enhances target boundary sharpness, thereby improving the positioning accuracy and stability of the tracking algorithm.

[0102] The technical benefits of this step are twofold: firstly, by calculating the gradient magnitude map, the edge quality of the image can be quantitatively evaluated; secondly, the L1 norm constraint helps suppress artifacts and noise in the reconstructed image, enhancing the sparsity and saliency of the edges. Overall, this step, within the unsupervised learning framework, provides the model with structure-aware auxiliary supervision signals, improving the reconstruction quality of moving target images and the performance of subsequent tracking algorithms.

[0103] Example 2

[0104] To implement the spiking neural network tracking method based on an event camera proposed in this invention, it is first necessary to acquire video data of small targets using an event camera. The event camera can accurately capture microsecond-level visual changes and can simultaneously output traditional image frames as well as ON (increased light intensity) and OFF (decreased light intensity) event data. To ensure the consistency of the neural network model input and training efficiency, the event data and grayscale images need to undergo standardized preprocessing. For the event data output by the event camera, temporal matching is performed with the image frames, and the data is divided into multi-time-step event frames at a fixed time step (T=4). The specific processing flow is as follows:

[0105] First, S101 acquires event data and image data based on the event camera, as follows:

[0106] The event images for the corresponding time steps are read from the ON and OFF subdirectories respectively, for a total of 8 images. They are then stitched together in chronological order into a tensor of shape [8, H, W], where the channel dimension is 4 time steps × 2 positive and negative polarity event images.

[0107] All image data (event images and grayscale images) were normalized to the range [0, 1] to unify the image intensity distribution and enhance the convergence stability and generalization ability of the model.

[0108] Secondly, in S102, super-resolution reconstruction of the moving target based on multi-time-step event data is performed, as follows:

[0109] It is understood that this invention aims to utilize sparse visual information captured by an event camera at high temporal resolution to perform super-resolution reconstruction of moving targets within a target time period. The overall framework mainly consists of an event feature extractor, an image feature extractor, a feature fusion module, a super-resolution decoder, and an autoencoder module. This architecture can effectively fuse multi-timestep event streams with low-resolution image frame information, achieving spatiotemporal consistency modeling and detailed restoration of moving targets. The overall framework flow is as follows: Figure 2 As shown.

[0110] The event feature extractor is used to extract discriminative spatial features from the event data output by the neuromorphic visual perception chip. The input to this module is an event tensor of dimensions [B, 8, H, W], where B represents the batch size, H and W represent the height and width of the image, respectively, and the 8 channels correspond to the positive and negative polarity event maps extracted at 4 time steps.

[0111] This module comprises three consecutive 2D convolutional layers, each followed by batch normalization and ReLU activation to enhance the network's non-linear expressive power and accelerate convergence. Ultimately, the module outputs a feature map with 64 channels and dimensions [B, 64, H, W], providing a rich semantic information foundation for subsequent image reconstruction and tracking.

[0112] The image feature extractor is used to extract static texture features and structural information from low-resolution grayscale images. The input is a single-channel image of size [B, 1, H, W]. To obtain multi-scale contextual information, this module introduces a parallel multi-scale convolutional structure, including three sets of convolutional channels with kernel sizes of 3×3, 5×5, and 7×7. The extracted features are concatenated and further enhanced with an attention mechanism to improve the response of key regions, enabling the network to more accurately capture edges and target regions in the image. The final output dimension is [B, 64, H, W], aligned with the event features in both spatial and channel dimensions.

[0113] The feature fusion module is used to fuse event features and image features, achieving deep integration of multi-source information. The specific process is as follows:

[0114] The event feature map and the image feature map are concatenated along the channel dimension to form a fused feature with dimensions [B, 128, H, W].

[0115] By using a 1×1 convolution operation, the channel dimension is compressed to 64, achieving efficient information fusion and dimensionality reduction;

[0116] A residual connection mechanism is introduced to add the input and the processed features, thereby improving training stability and enhancing the representation of key features.

[0117] This module effectively integrates dynamic event information with static image texture features, improving the overall feature representation capability and enhancing the model's ability to perceive moving targets.

[0118] Among them, a super-resolution encoder is used. To achieve upscaling and detail restoration of low-resolution images, this invention designs a structurally efficient upsampling module. This module receives fused features as input and generates high-resolution output through the following structure:

[0119] A single transposed convolution operation, with a kernel size of 4, a stride of 2, and padding of 1, enlarges the image spatial size to twice its original size.

[0120] Batch normalization and ReLU activation function are used to enhance nonlinear modeling capabilities;

[0121] The final convolutional layer outputs a 1-channel grayscale image with dimensions [B, 1, H×scale, W×scale], where scale represents the upsampling ratio.

[0122] This decoder balances image quality and computational efficiency, making it suitable for real-time deployment on edge platforms such as drones.

[0123] Among them, an autoencoder-decoder structure is introduced as an auxiliary training module to enhance the network's learning ability in unsupervised scenarios. This module mainly includes:

[0124] Encoder: Consists of two convolutional layers. The input image is compressed into 64-channel low-dimensional features by the encoder.

[0125] Decoder: symmetrical structure, also consisting of two convolutional layers, used to restore features to the original image size;

[0126] Output: The reconstructed low-resolution image with dimensions [B, 1, H, W], identical to the original image.

[0127] The training objective of this module is to minimize the reconstruction error between the input image and the reconstructed image, guide the main network to learn more robust feature representations, and still have good generalization performance in the absence of high-quality labeled data.

[0128] Among them, the loss function design. To realize the training optimization process of the moving target super-resolution reconstruction method proposed in this invention, a composite loss function system with multiple components working collaboratively is designed. Under the unsupervised learning framework, this system uses multi-source supervision signals to constrain and guide the model, improve the quality and structural fidelity of the reconstructed moving target, and ensure that the model has strong edge preservation ability and feature consistency. It mainly includes the following four aspects:

[0129] Autoencoder Reconstruction Loss. This loss term measures the autoencoder module's ability to reconstruct low-resolution input images. By guiding the encoder-decoder structure to learn the main texture and structural information of the image, it enhances the model's representational ability under unsupervised conditions. This term consists of two forms of standard error.

[0130] L1 loss. Employing pixel-level absolute error, it possesses strong edge-preserving capabilities and is defined as:

[0131]

[0132] in, This indicates that the image is reconstructed by an autoencoder. The input is a low-resolution image.

[0133] Mean Squared Error Loss (MSE). A measure of the average squared difference in overall pixel variation, defined as:

[0134]

[0135] Combining these two losses can simultaneously ensure the accuracy of the overall image structure and the ability to preserve details.

[0136] Consistency Loss. To ensure structural consistency between the super-resolution image generated by the main path and the autoencoder path, a consistency loss term is introduced. This term constrains the downsampled super-resolution image to maintain a high degree of similarity with the autoencoder-reconstructed image, and is defined as:

[0137]

[0138] in, This indicates a super-resolution reconstructed image. This indicates that a 2x downsampling operation is performed using methods such as bilinear interpolation. This loss term ensures consistency between the two information flow paths in the feature space, enhancing the model's ability to model image structure.

[0139] Smoothness Loss. To reduce artifacts and noise in super-resolution images, an image smoothness loss is introduced to encourage spatial continuity in the generated image. This term is achieved by calculating the gradient difference between adjacent pixels in the image and is defined as:

[0140]

[0141] This helps eliminate high-frequency noise in images, improving the naturalness and visual quality of reconstructed images.

[0142] Gradient loss. To further preserve key edge information in the image, gradient loss is introduced to enhance the sparsity of the image gradient magnitude map. This term can be calculated using the Sobel operator or the difference operator, and a sparsity constraint is imposed, defined as:

[0143]

[0144] Here, ▽ represents the gradient calculation operation of the image, and the L1 norm is used to improve the response of edge regions while suppressing meaningless detail noise.

[0145] Finally, the total loss function is expressed as a weighted summation:

[0146]

[0147] in, to The weight parameters for each loss term can be adjusted according to task requirements and training performance. This multi-task loss design comprehensively considers the image's content accuracy, structural consistency, edge preservation, and visual quality, and is one of the core optimization methods in the model training phase of this invention.

[0148] The network training and optimization strategy is described below. To train the entire image reconstruction model, this section uses the Adam optimizer for end-to-end training, with the following parameters set:

[0149] Initial learning rate: 0.0001;

[0150] Weight decay: 0.0001;

[0151] Momentum parameter: .

[0152] During training, the following strategies were used to improve model performance:

[0153] Reduce the learning rate to 0.1 times its original value every 20 epochs.

[0154] Finally, in S103, the implementation process of the tracking algorithm based on reconstructed moving targets is as follows:

[0155] This embodiment proposes a target tracking algorithm based on reconstructed moving target and Spiking Neural Network (SNN). The model utilizes the reconstructed high-resolution moving target image to improve the model's adaptability and robustness to small targets in complex environments. While maintaining temporal dynamic information, it significantly reduces computational overhead and is suitable for low-power edge devices.

[0156] The overall framework of the model is shown in Figure 3. The input includes two modalities: images and corresponding event data streams. These two modalities are input into the super-resolution moving target reconstruction module mentioned above to obtain high-resolution moving target images. Then, the spatial features of the template and the search branch are obtained through the SNN feature extraction module. The spatial features of the template and the search frame are then correlated to find the position with the maximum correlation, which is the location of the target.

[0157] Specifically, the SNN feature extraction module, loss function, and training strategy are as follows:

[0158] The feature extraction module employs a convolutional spiking neural network structure to extract features from both the target template image and the search region image. This network structure is based on a simplified AlexNet and consists of multiple convolutional layers, a spiking neuron model, a local response normalization (LRN) layer, and pooling layers. The parameters of the convolutional layers share weights in the two branches of the Siamese network to ensure consistency of the extracted features within the same spatial region.

[0159] The network structure is as follows, consisting of five convolutional layers, configured as follows:

[0160] The first convolutional kernel size is 11×11, the stride is 2, and the output channels are 96.

[0161] The second convolutional kernel size is 5×5, and the output channels are 256.

[0162] The kernel size of the third, fourth, and fifth convolutional layers is 3×3, and the output channels are 384, 384, and 256, respectively.

[0163] All convolutional layers are followed by a spiking neuron model, with some layers having additional pooling or LRN operations.

[0164] The extracted target template feature map and the search region feature map are cross-correlated channel by channel to generate a response score map. The position of the maximum value in the response score map is the predicted position of the target in the current frame.

[0165] The spiking neurons in the network adopt the classic LIF neuron model. The dynamic model of LIF can be described by three equations: charging, discharging, and resetting. The charging equation is as follows:

[0166]

[0167] in The membrane time is a constant. This represents the neuron membrane voltage at the previous moment. It is the external input at the current moment.

[0168] When the instantaneous membrane potential exceeds the threshold If a pulse is emitted when the pulse is present, it is emitted; otherwise, no pulse is emitted. The neuron firing process is as follows:

[0169]

[0170] in For step functions:

[0171]

[0172] When the spiking neuron fires a pulse, the voltage resets to... When the spiking neuron does not fire, the voltage remains unchanged. The reset equation for the spiking neuron is as follows:

[0173]

[0174] in This represents the membrane potential at the current moment.

[0175] Loss function and training strategy. This model uses a logistic regression loss function to optimize the accuracy of the response map. The loss function is defined as follows:

[0176]

[0177] in: This represents the predicted value at position iii in the response graph; For the label corresponding to this position, the central region is set as a positive sample (+1), and the region far from the center is set as a negative sample (-1); N represents the number of all pixels in the response map.

[0178] The pulse neural network tracking method based on an event camera in this invention effectively improves the reconstruction quality and target tracking robustness of low-resolution event camera images, and achieves accurate perception and stable tracking of moving targets in high dynamic and low power consumption scenarios.

[0179] Example 3

[0180] To achieve the above embodiments, such as Figure 4 As shown, this embodiment also provides a spiking neural network tracking system 10 based on an event camera, including:

[0181] The event data acquisition module 100 is used to acquire multi-time step event data and low-resolution grayscale images output by the event camera, and combine the event data into a multi-channel event tensor according to time order and polarity.

[0182] The feature extraction and fusion module 200 is used to extract dynamic event features from the event tensor and static image features from the grayscale image, respectively, and perform feature fusion through channel splicing and convolution operations to form a spatiotemporally consistent feature representation.

[0183] The image super-resolution reconstruction module 300 is used to input the fused features into the super-resolution decoder and achieve image upsampling through transposed convolution operation to generate a high-resolution moving target image;

[0184] The target tracking feature processing module 400 is used to input high-resolution images into the LIF neuron-based spiking neural network feature extraction module, extract the features of the target template and the search area, and determine the target position through cross-correlation operation to achieve tracking.

[0185] Furthermore, the event data acquisition module is also used for:

[0186] The event images corresponding to the time steps are read from the ON and OFF subdirectories, totaling 8 images. They are then stitched together in chronological order into a tensor of shape [8, H, W], where 4 time steps correspond to positive and negative polarity event images respectively.

[0187] All image data are normalized to unify the intensity values ​​of event images and grayscale images to the range of [0, 1], thereby enhancing the convergence stability and generalization ability of the model.

[0188] Furthermore, the feature extraction and fusion module is also used for:

[0189] Dynamic event features are extracted by three consecutive two-dimensional convolution operations. Each convolution is followed by batch normalization and ReLU activation function, and the final output is a feature map with 64 channels.

[0190] Static image features are extracted using a parallel multi-scale convolutional structure, including three sets of convolutional channels with kernel sizes of 3×3, 5×5, and 7×7. After concatenation, the response of key regions is enhanced through an attention mechanism.

[0191] Furthermore, the image super-resolution reconstruction module is also used for:

[0192] By using transposed convolution with a kernel size of 4, a stride of 2, and padding of 1, the image spatial size is enlarged to twice its original size.

[0193] The nonlinear modeling capability is enhanced by batch normalization and ReLU activation function, and a 1-channel grayscale image is output through the final convolutional layer.

[0194] Furthermore, it also includes:

[0195] The image quality assessment module is used to assess the image quality of the high-resolution moving target image. It uses the Sobel operator to calculate the image gradient magnitude map and constrains the gradient sparsity through the L1 norm to further optimize the image edge preservation capability.

[0196] The pulse neural network tracking system based on an event camera in this invention effectively improves the reconstruction quality and target tracking robustness of low-resolution event camera images, and achieves accurate perception and stable tracking of moving targets in high dynamic and low power consumption scenarios.

[0197] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0198] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A spiking neural network tracking method based on an event camera, characterized in that, include: S1: Acquire multi-timestep event data and low-resolution grayscale images output by the event camera, and combine the event data into a multi-channel event tensor according to time order and polarity; among them, read the event images of the corresponding time steps from the ON and OFF subdirectories, a total of 8 images, and stitch them into a tensor of shape [8, H, W] according to time order, with the 4 time steps corresponding to positive and negative polarity event images respectively; normalize all image data, and uniformly normalize the intensity values ​​of the event images and grayscale images to the range of [0, 1] to enhance the convergence stability and generalization ability of the model; S2 extracts dynamic event features from the event tensor and static image features from the grayscale image, respectively, and fuses these features through channel concatenation and convolution operations to form a spatiotemporally consistent feature representation. Specifically, dynamic event features are extracted through three consecutive layers of two-dimensional convolution operations, with batch normalization and ReLU activation functions applied after each convolution layer, ultimately outputting a feature map with 64 channels. Static image features are extracted through a parallel multi-scale convolution structure, including three sets of convolution channels with kernel sizes of 3×3, 5×5, and 7×7. After concatenation, an attention mechanism is used to enhance the response of key regions. S3, the fused features are input into the super-resolution decoder, and the image is upsampled through transposed convolution to generate a high-resolution moving target image. The transposed convolution operation, configured with a kernel size of 4, a stride of 2, and padding of 1, enlarges the image spatial size to twice its original size. Batch normalization and ReLU activation function enhance nonlinear modeling capabilities, and a 1-channel grayscale image is output through the final convolutional layer. The super-resolution decoder is trained using an autoencoder structure, which includes an encoder and a decoder. The training objective is to minimize the reconstruction error between the input low-resolution image and the autoencoder-reconstructed image. A composite loss function is used to optimize the super-resolution decoder, including at least the autoencoder reconstruction loss, consistency loss, and gradient loss. The consistency loss constrains the downsampled super-resolution image to maintain similarity to the autoencoder-reconstructed image, and the gradient loss calculates the image gradient magnitude using the Sobel operator and applies L1 norm sparsity constraints. S4, the high-resolution image is input into the LIF neuron-based spiking neural network feature extraction module to extract the features of the target template and the search area, and the target position is determined through cross-correlation operation to achieve tracking; wherein, the spiking neural network feature extraction module adopts a convolution-based spiking neural network structure, the basic architecture of which includes five convolutional layers, each of which is connected to a LIF neuron model to simulate the spiking mechanism of biological neurons.

2. The method as described in claim 1, characterized in that, Also includes: S5 performs image quality assessment on high-resolution moving target images, uses the Sobel operator to calculate the image gradient magnitude map, and constrains gradient sparsity through the L1 norm to further optimize the image edge preservation capability.

3. A pulse neural network tracking system based on an event camera, characterized in that, include: The event data acquisition module is used to acquire multi-timestep event data and low-resolution grayscale images output by the event camera, and combine the event data into a multi-channel event tensor according to time order and polarity. Specifically, it reads the event images corresponding to the time steps from the ON and OFF subdirectories, totaling 8 images, and stitches them into a tensor of shape [8, H, W] in time order, with the 4 time steps corresponding to positive and negative polarity event images respectively. All image data are normalized to unify the intensity values ​​of the event images and grayscale images to the range of [0, 1] to enhance the convergence stability and generalization ability of the model. The feature extraction and fusion module is used to extract dynamic event features from the event tensor and static image features from the grayscale image, respectively, and to fuse features through channel concatenation and convolution operations to form a spatiotemporally consistent feature representation. Specifically, dynamic event features are extracted through three consecutive layers of two-dimensional convolution operations, with batch normalization and ReLU activation function applied after each convolution layer, ultimately outputting a feature map with 64 channels. Static image features are extracted through a parallel multi-scale convolution structure, including three sets of convolution channels with kernel sizes of 3×3, 5×5, and 7×7. After concatenation, an attention mechanism is used to enhance the response of key regions. The image super-resolution reconstruction module is used to input fused features into the super-resolution decoder, and upsample the image through transposed convolution to generate a high-resolution moving target image. Specifically, the transposed convolution operation, configured with a kernel size of 4, a stride of 2, and padding of 1, enlarges the image spatial size to twice its original size. Batch normalization and ReLU activation function are used to enhance nonlinear modeling capabilities, and a 1-channel grayscale image is output through the final convolutional layer. The super-resolution decoder is trained using an autoencoder structure, which includes an encoder and a decoder. The training objective is to minimize the reconstruction error between the input low-resolution image and the autoencoder-reconstructed image. A composite loss function is used to optimize the super-resolution decoder, which includes at least the autoencoder reconstruction loss, consistency loss, and gradient loss. The consistency loss constrains the downsampled super-resolution image to maintain similarity with the autoencoder-reconstructed image, and the gradient loss calculates the image gradient magnitude map using the Sobel operator and applies L1 norm sparsity constraints. The target tracking feature processing module is used to input high-resolution images into the LIF neuron-based spiking neural network feature extraction module, extract the features of the target template and the search area, and determine the target position through cross-correlation operation to achieve tracking; wherein, the spiking neural network feature extraction module adopts a convolution-based spiking neural network structure, the basic architecture of which includes five convolutional layers, each of which is connected to an LIF neuron model to simulate the spiking mechanism of biological neurons.

4. The system as described in claim 3, characterized in that, Also includes: The image quality assessment module is used to assess the image quality of high-resolution moving target images. It uses the Sobel operator to calculate the image gradient magnitude map and constrains gradient sparsity through the L1 norm to further optimize the image edge preservation capability.

Images