RGB / event-based spatio-temporal feature fusion target tracking method

Abstract

The RGB / Event-based spatiotemporal feature fusion target tracking method disclosed by the application belongs to the technical field of computer vision and target tracking. The implementation method of the application is as follows: taking DiMPNet network as the basis for tracking, introducing a pulse neuron with a dynamic threshold and a membrane time constant as a front-end processing module of the event mode, while filtering out redundant noise in the event stream, enhancing the dense event cluster caused by the moving target. On this basis, a high-order spatiotemporal feature fusion module is constructed, the RGB and Event mode information is fully mined and deeply interacted in the frequency domain, spatial multi-scale and high-order channel statistics three dimensions, so as to narrow the modal difference of the double modal features in the unified ResNet-18 backbone network, realize the complementation and cooperation of the double modal information, and improve the robustness, accuracy and real-time performance of the target tracking system in the complex scene such as low light, high-speed motion and complex background.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and target tracking technology, and relates to a target tracking method based on spatiotemporal feature fusion of RGB / Event, which is particularly suitable for real-time robust tracking in challenging scenarios such as low light, high speed motion, and complex background. Background Technology

[0002] Object tracking, a fundamental and crucial task in computer vision, aims to continuously and accurately estimate the position and scale of an object in a video sequence, based on a bounding box given in the initial frame. In recent years, object tracking has demonstrated wide-ranging applications in numerous real-world scenarios, including robot perception, drone navigation, and autonomous driving.

[0003] Traditional target tracking methods primarily rely on RGB images as input, using convolutional neural networks (CNNs) to extract the target's appearance features for tracking. While these methods perform well in normal lighting and smooth motion scenarios, in complex environments such as low light, high-speed motion, motion blur, and background interference, the inherent frame rate limitations and insufficient dynamic range of RGB cameras often lead to a significant drop in tracking performance.

[0004] To improve tracking robustness in complex scenarios, multimodal fusion tracking methods have gained increasing attention. Event cameras, as a novel type of visual sensor, can asynchronously respond to changes in light intensity with microsecond-level temporal resolution, outputting event stream data. They offer advantages such as high dynamic range, no motion blur, and low power consumption. Combining event data with RGB images allows for the full utilization of the fine temporal motion information provided by the former and the rich spatial semantic features of the latter, creating a complementary effect and enhancing the environmental adaptability of the tracking system.

[0005] However, existing RGB-E tracking methods still face two major challenges: first, event data is an asynchronous sparse stream, making it difficult for traditional CNNs to directly and effectively extract its temporal dynamic features; second, there are significant differences between RGB and event modalities, and how to achieve efficient and deep feature fusion across modalities remains a key challenge. Currently, mainstream methods typically convert the event stream into pseudo-image frames before feeding them into CNNs. While this simplifies data representation, it loses the inherent continuous temporal information of the event data, and the local receptive field of CNNs also limits their ability to model long-range spatiotemporal dependencies.

[0006] Biologically inspired Spike Neural Networks (SNNs), as third-generation neural networks, have shown great potential in processing event stream data due to their unique asynchronous processing mechanism and temporal encoding capabilities. However, a systematic solution is still lacking for how to organically combine the temporal processing advantages of SNNs with the spatial feature extraction capabilities of CNNs, and to design an effective cross-modal fusion mechanism. Furthermore, existing fusion methods mostly focus on spatial or shallow feature interactions, failing to fully explore the complementary value of the temporal dimension and high-order statistical information in complex scenarios.

[0007] Therefore, fully exploring the spatiotemporal information in RGB-E dual-modal data and realizing an efficient cross-modal fusion target tracking method is of great significance for improving tracking accuracy, robustness, and real-time performance in dynamic and complex environments. Summary of the Invention

[0008] To address the limitations of existing RGB-E bimodal target tracking methods, such as insufficient utilization of temporal information, significant event stream noise interference, and shallow cross-modal fusion mechanisms, this invention aims to provide a spatiotemporal feature fusion-based target tracking method using RGB / Event. This method is based on a DiMPNet network and introduces spiking neurons with dynamic thresholds and membrane time constants as the front-end processing module for the event modality. This enhances the dense event clusters triggered by moving targets while filtering out redundant noise in the event stream. Furthermore, a high-order spatiotemporal feature fusion module achieves deep interaction between RGB and Event modal information across multiple dimensions, including frequency, space, and channels. This reduces the modal differences between bimodal features within a unified ResNet-18 backbone network, enabling complementarity and synergy of bimodal information and improving the robustness, accuracy, and real-time performance of the target tracking system in challenging scenarios such as low light, high-speed motion, and complex backgrounds.

[0009] The objective of this invention is achieved through the following technical solution.

[0010] The spatiotemporal feature fusion target tracking method based on RGB / Event disclosed in this invention includes the following steps:

[0011] Step 1: Synchronous acquisition, timing alignment, and preprocessing of RGB-E dual-modal data to obtain RGB template frames. and RGB search frame Event template frame and Event search frame .

[0012] Step 1.1: Synchronously acquire aligned RGB image and event stream data. The event stream data consists of a series of quadruple events. Composition, among which, and Represents the spatial coordinates of where the event occurred. This represents the timestamp when the event occurred. p represents the polarity of the event.

[0013] Step 1.2: To preserve the high-resolution temporal information of the event stream data and maintain time alignment with the RGB image, the exposure time T of the RGB image is evenly divided into n time windows. Within each time window, the event stream data is aggregated into event frames. The value of n typically ranges from 2 to 8. For the i-th event stream data... It does not distinguish between positive and negative polarities, and is located at the corresponding position where the event occurs. Pixel values ​​are accumulated. Finally, each RGB image generates n corresponding event frames. Each event frame has 1 image channel, the same resolution as the RGB frame, and the tracking target position corresponds to the RGB frame.

[0014] Step 1.3: Perform preprocessing on the RGB image obtained in Step 1.1 to obtain the RGB template frame. and RGB search frame Preprocess the n event frames corresponding to the RGB images obtained in step 1.2 to obtain the Event template frames. and Event search frame .

[0015] Step 2: Based on three learnable weight parameters , , Adjust dynamic threshold .right and Global average pooling and global max pooling are performed separately, and the mixture is fed into a fully connected network and mapped to the interval [0.5, 0.95] to dynamically generate the membrane time constant. Local neighborhood weighting is performed using 3×3 grouped convolution, based on... Update the membrane potential, and when the membrane potential exceeds the dynamic threshold Timely pulse delivery to complete the process and Temporal filtering and information enhancement yielded and .

[0016] Step 2.1: Dynamic threshold adjustment. Three learnable weight parameters. , , These are, respectively, event density weights Calculated using the global pixel average of the current event frame, reflecting the sparsity of the event. Motion clustering weights. By calculating the variance of events within a local 3×3 neighborhood, moving targets typically generate event clusters with high variance, thus this method can be used to assess the spatial clustering of events. Temporal variation weights. The dynamic threshold is calculated by averaging the absolute differences between the current frame and the previous frame, and is used to measure the intensity of dynamic changes in the scene. pass , , The threshold is obtained by weighting and summing the corresponding density, motion pattern, and temporal changes. This step generates a dynamic threshold adjustment based on information from the current frame and past frames, enabling neurons to remain sensitive when events are sparse and to improve their inhibitory ability when events are dense.

[0017] Step 2.2: Dynamic Membrane Time Constant Adaptation. A channel attention mechanism is introduced to generate the dynamic membrane time constant based on dual-path pooling features (max pooling and average pooling). value:

[0018]

[0019] in, and These are average pooling and max pooling, respectively. and For activation function, and These are the learnable weights.

[0020] This design allows neurons to adjust their membrane time constant according to their response characteristics. This allows for adaptive adjustment of the information decay rate, enhancing the model's adaptability to various motion modes.

[0021] Step 2.3: Temporal filtering and information enhancement. This is achieved by using a 3×3 grouped convolution kernel. and Local neighborhood information fusion is performed, with convolutional kernel weights initialized to 0.2 for the center pixel and 0.1 for surrounding pixels to enhance the spatial correlation of events. The weighted feature input is based on the spiking neurons of the leak integral-fire LIF model, whose membrane potential... It will increase as input events accumulate:

[0022]

[0023] in The time constant of the dynamic membrane. The timestamp of the current moment. The input at the current moment is used when the membrane potential exceeds the dynamic threshold. At that time, neurons fire impulses Subsequently, the membrane potential recovers according to the reset rule, and the neuron eventually outputs a pulse weight matrix with the same shape as the input. Multiplying the weight matrix element-wise with the original input can enhance the motion event and filter out noise, resulting in the enhanced version. and .

[0024] Step 3: The result obtained in Step 1 and and the result obtained in step two and They share the same ResNet-18 backbone network for feature extraction to obtain spatially aligned and semantically consistent bimodal feature representations. and .

[0025] Step one and The input is passed through a ResNet-18 network, sequentially through convolutional layers, batch normalization layers, and activation functions, to extract multi-scale spatial feature maps. Step two and Input the same ResNet-18 network to extract spatiotemporal feature maps containing temporal motion information. ResNet-18 contains outputs from multiple layers. The output of Layer 2 is selected for precise target localization, and the output of Layer 3 is selected for target recognition and contextual understanding.

[0026] Step 4: Fuse the dual-modal features to obtain .

[0027] After acquiring Layer 2 and Layer 3 features of RGB and event modalities, a spatiotemporal high-order feature fusion module is used to achieve deep interaction and complementary enhancement of bimodal information. This module employs a multi-level fusion strategy to perform high-order interaction in the frequency domain, spatial domain, and channel dimensions.

[0028] Step 4.1: Process the RGB feature map obtained in Step 3. Event Feature Map Perform two-dimensional fast Fourier transforms to obtain the frequency domain representation. and The element-wise product of the two modal features is calculated in the frequency domain to generate the joint frequency response. After that, Perform an inverse Fourier transform to restore the spatial domain and obtain a preliminary attention map. Then, normalize along the channel dimension to generate the final spatial attention weights. The obtained attention weights are then compared with... and Perform weighted modulation and output the modulated features. and .

[0029] Step 4.2: Feature Analysis and , respectively , , Three different convolutional kernels are used to extract features, capturing multi-scale information from local details to global context. The three outputs are then concatenated along the channel dimension and passed through a... Convolutional layers are fused to generate cross-modal attention weights. and .

[0030] Step 4.3: Using weights and Modulation characteristics and ,get , Modulation was achieved using a self-attention mechanism. , Modulated , , , and original features and Output via residual connection and .

[0031] Step 4.4: Use second-order or higher statistical layers to... and conduct Weighted sub-iteration:

[0032]

[0033]

[0034] in For the first The channel weights generated by the order, The input features for each iteration, ,

[0035] Step 4.5: Take the result obtained in Step 4.3 and step 4.4 Weighted after the next iteration Obtained through residual connection Event modality and Obtained through residual connection .Will and The parts are then pieced together to form the final fused feature. .

[0036] This fusion module captures global structural dependencies through frequency domain interaction, enhances local details through multi-scale context, and achieves semantic association through high-order channel statistical modeling, realizing deep complementarity and synergy between RGB and Event modalities in the three dimensions of time, space, and channel.

[0037] Step 5: The data is fed into the tracking head to obtain the final target tracking result, achieving spatiotemporal feature fusion target tracking.

[0038] Will The data is fed into a classifier, which performs cross-correlation between the target template features and the search region features to generate a classification score map. Then, the probability distribution of the target location is obtained by applying a Softmax or Sigmoid activation function.

[0039] Will The data is fed into a regressor, which modulates the initial target box using the predicted position offset. The bounding box is then filtered using the IoU score, and non-maximum suppression is used to remove redundant detections.

[0040] By selecting the candidate box with the highest classification score as the final tracking result, the center coordinates, height, and width of the bounding box corresponding to the regressor are output, and the target template features are updated to adapt to changes in template appearance.

[0041] Beneficial effects:

[0042] 1. Regarding the adaptive temporal filtering capability of event stream data, this invention discloses a spatiotemporal feature fusion target tracking method based on RGB / Event. By introducing a spiking neuron module with dynamic thresholds and membrane time constants, this invention can adaptively filter out noise and enhance events triggered by moving targets based on the local density, temporal changes, and motion patterns of the event stream. This mechanism effectively solves the problems of relatively sparse event data and large noise interference, enabling the network to focus on the target's motion region more quickly and accurately, reducing invalid computation, and thus achieving higher tracking accuracy and robustness with fewer iterations in complex dynamic scenarios.

[0043] 2. A multi-layered, multi-dimensional cross-modal feature deep fusion mechanism: The spatiotemporal feature fusion target tracking method based on RGB / Event disclosed in this invention constructs a high-order spatiotemporal feature fusion module, breaking through the limitations of traditional simple fusion methods, and achieving collaborative interaction in three dimensions: frequency domain, spatial multi-scale, and high-order channel statistics. This invention can not only fully explore the complementary information of RGB and event modalities from multiple levels such as global structure, local details, and semantic association, but also enhance the model's adaptability to challenges such as target deformation, occlusion, and illumination changes, thereby improving the accuracy and stability of tracking.

[0044] 3. A unified architecture design that balances efficiency and performance: The spatiotemporal feature fusion target tracking method based on RGB / Event disclosed in this invention uses a shared backbone network for dual-modal feature extraction and combines a lightweight adaptive module with an efficient fusion mechanism to significantly reduce model complexity and computational overhead while maintaining high tracking accuracy. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the framework of the spatiotemporal feature fusion target tracking method based on RGB / Event disclosed in this invention.

[0046] Figure 2 This is a schematic diagram illustrating the principle of the adaptive neuron module in the method of this invention.

[0047] Figure 3 This is a schematic diagram of the structure of the high-order spatiotemporal feature fusion module in the method of the present invention.

[0048] Figure 4 Figure (a) shows the quantitative tracking results of the method of the present invention and conventional target tracking methods on the VisEvent dataset, and Figure (b) shows the target tracking evaluation index PR curve.

[0049] Figure 5 This paper compares the qualitative tracking results of the method of this invention with those of conventional target tracking methods on the VisEvent dataset, including complex scenarios such as high-speed target movement and low-light conditions. Detailed Implementation

[0050] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This embodiment discloses a spatiotemporal feature fusion-based target tracking method based on RGB / Event. The experimental device was an RTX 4060 Ti graphics card, and the experimental platform was based on Python 3.8 and PyTorch 2.0.0 deep learning API. The proposed spatiotemporal feature fusion network was used as the tracker, and experiments were conducted on the public datasets VisEvent and FE108. Figure 1 As shown, taking a complex scenario where the target is a high-speed moving vehicle as an example, the specific implementation steps of the spatiotemporal feature fusion target tracking method based on RGB / Event disclosed in this embodiment are as follows:

[0052] Step 1: Input RGB / Event bimodal samples containing vehicles, and preprocess them to obtain RGB template frames and RGB search frames, Event template frames and Event search frames.

[0053] Step 1.1: First, process the original ".aedat4" files containing the target vehicle captured by the event camera in chronological order. The RGB data is directly extracted and saved as RGB frames. For the Event data, based on the timestamps of the RGB data, only the event stream data between the start and end timestamps is retained. The event stream data includes... There are four dimensions, where x and y represent the spatial coordinates of the event. t represents the timestamp when the event occurred. p represents the polarity of the event, i.e., whether the brightness change is enhanced or weakened, with +1 indicating enhanced brightness and -1 indicating weakened brightness.

[0054] Step 1.2: To preserve the fine-grained temporal information of the event stream and align it with the RGB frame time, the exposure time of each RGB image is divided into four time windows. Within each time window, the event stream is aggregated into event frames. For the i-th event stream data... Regardless of positive or negative polarity, all events occur at the corresponding locations. The pixel value is incremented by one. Finally, four event frames accumulated before and after the RGB frame form an event frame group. Each event frame in the event frame group has 1 image channel, the same resolution as the RGB frame, and the tracking target position corresponds to the RGB frame.

[0055] Step 1.3: To meet the requirements of network input standardization and training data diversity, systematic spatial transformation and data augmentation preprocessing are needed for the original RGB frames and corresponding Event frames. For the initial frames (RGB frame size 260×346×3, Event frame size 4×260×346×3), a random dithering strategy is used to perturb the target center position and bounding box scale, enhancing the model's adaptability to changes in target scale and appearance. The original Event frames are single-channel representations. To maintain structural consistency with the RGB input and facilitate subsequent convolution operations, each Event frame is copied three times in the channel dimension, reshaping it into a tensor of size 4×260×346×3. Then, the input image is centered, cropped, and sized to 288×288, and normalized preprocessing is performed to obtain data that meets the network input requirements. RGB template frame. and RGB search frame The size is 3×288×288, Event template frame and Event search frame The dimensions are 4×3×288×288, corresponding to 4 time steps, each time step being a three-channel 288×288 image.

[0056] Step 2: Perform temporal filtering and enhancement on the input Event data using the adaptive neuron module.

[0057] Event stream data is characterized by sparsity, asynchronicity, and spatiotemporal clustering. When the target's movement is gentle or the illumination changes only slightly, the generated event stream is sparse, and the effective information in the event frames is limited. When the target moves at high speed or the illumination changes drastically, the event stream is dense, but it is often accompanied by a lot of noise. However, events triggered by real moving targets are highly correlated in time and space, while noisy events exhibit random distribution characteristics.

[0058] To address the aforementioned characteristics, an adaptive neuron module is designed. Leveraging the high temporal resolution of the event data, it performs temporal filtering and feature enhancement on the input multi-time-step event data. For example... Figure 2 As shown, this module is based on input features Dynamically adjusting key neuron parameters—firing threshold With membrane time constant This enables the enhancement of moving target events and the suppression of noise.

[0059] Step 2.1: Dynamic threshold generation. The threshold of a neuron. It is not fixed, but dynamically adjusted according to the spatiotemporal characteristics of the input event, and is determined by three learnable weight parameters. , , Control. Event density weighting. The sparsity of events is reflected by calculating the global pixel average of the current event frame. Motion clustering weights. Spatial clustering of events is assessed by calculating the variance of events within a local 3×3 neighborhood (moving targets typically generate clusters of events with high variance). Temporal variation weights are also considered. The intensity of dynamic changes in the scene is measured by calculating the mean absolute difference between the current frame and the previous frame. The final dynamic threshold is then determined. pass , , It is obtained by weighted summation with the corresponding density, motion pattern, and temporal changes.

[0060] Step 2.2: Dynamic membrane time constant adaptation. Membrane time constant. This invention controls the rate at which neurons retain historical information. It employs a channel attention mechanism to dynamically generate the memory for each channel. Values. First, global average pooling is performed on the input feature maps respectively. With global max pooling The pooled features are concatenated and then passed through a two-layer fully connected network (using ReLU activation in the middle layer and Sigmoid activation in the output layer) to generate initial weights. and Finally, a linear mapping to the interval [0.5, 0.95] is performed to obtain the dynamically adapted membrane time constant. The specific calculation formula is as follows:

[0061]

[0062] This design allows different feature channels to adaptively adjust the information decay rate according to their response characteristics, enhancing the model's adaptability to various motion modes.

[0063] Step 2.3: Neighborhood weighting and pulse triggering mechanism. First, a 3×3 grouped convolution kernel is used to... and Local neighborhood information fusion is performed, with convolutional kernel weights initialized to 0.2 for the center pixel and 0.1 for surrounding pixels to enhance the spatial correlation of events. The weighted feature input is based on spiking neurons of the leaky integral-fire (LIF) model, whose membrane potential... It will increase as input events accumulate:

[0064]

[0065] in The dynamic membrane time constant obtained in step 2.2 is used when the membrane potential exceeds the dynamic threshold obtained in step 2.1. At that time, neurons fire impulses Subsequently, the membrane potential recovers according to the reset rule, and the neuron eventually outputs a pulse weight matrix with the same shape as the input. By multiplying the weight matrix element-wise with the original input, the motion event can be enhanced while noise is filtered out, resulting in the enhanced version. and :

[0066]

[0067]

[0068] This bio-inspired triggering mechanism effectively distinguishes between ordered motion and random noise: events triggered by continuously moving targets at multiple time steps accumulate membrane potential at the same pixel location, eventually reaching a threshold and outputting a pulse. Randomly distributed noise events, lacking spatiotemporal consistency, are unlikely to cause sustained accumulation of membrane potential and are therefore suppressed.

[0069] Through the aforementioned adaptive mechanism, this module can effectively filter the event stream without relying on prior parameters, significantly improving the detection accuracy and robustness of the subsequent tracking module for moving targets.

[0070] Step 3: Take the result from Step 1 and and the result obtained in step two and The data are fed into a shared backbone network for bimodal feature extraction to obtain spatially aligned and semantically consistent bimodal feature representations. and .

[0071] Step 3.1: Before feeding the data into the backbone network for feature extraction, the input data needs to be dimension-unified. For RGB inputs with shape (frame_num, batch, 3, 288, 288) and Event inputs with shape (4, frame_num, batch, 3, 288, 288), the frame_num and batch dimensions are merged to obtain RGB inputs with shape (frame_num×batch, 3, 288, 288) and Event inputs with shape (4, frame_num×batch, 3, 288, 288). In this embodiment, frame_num = 3 (number of randomly sampled images), and batch = 30 (training batch size).

[0072] Step 3.2: Reconstruct the RGB data from Step 3.1 and and the reconstructed Event data and The data are input into a ResNet-18 network with shared weights for feature extraction. The RGB branch is directly input into the network to extract multi-scale spatial features. Since the Event branch contains four time steps, to reduce computational complexity and extract stable spatiotemporal representations, the event frames are first averaged along the time dimension to obtain an aggregated event representation of frame_num×batch (3, 288, 288), which is then input into ResNet-18 to extract features.

[0073] Step 3.3: The ResNet-18 backbone network outputs feature maps at multiple levels. This method mainly utilizes the features of Layer 2 and Layer 3 to obtain multi-level feature output. Layer 2 is a mid-level feature map with 128 channels, offering high spatial resolution and containing rich geometric details and local structural information, suitable for precise target localization. Layer 3 is a mid-to-deep feature map with 256 channels, possessing stronger semantic abstraction capabilities and a larger receptive field, suitable for target category determination and contextual understanding.

[0074] Step 4: Use the high-order spatiotemporal feature fusion module to fuse the dual-modal features obtained in Step 3 to obtain... .

[0075] After obtaining the Layer 2 and Layer 3 features of RGB and Event modalities, this invention achieves deep interaction and complementary enhancement of dual-modal information in three dimensions—frequency domain, spatial domain, and channel domain—through a designed high-order spatiotemporal feature fusion module. This module employs a cascaded fusion strategy, such as... Figure 3 As shown.

[0076] Step 4.1: Frequency Domain Spatial Interaction. This involves processing the input RGB feature map... Event Feature Map Perform two-dimensional fast Fourier transforms to obtain its frequency domain representation. and The element-wise product of the two modal features is calculated in the frequency domain to generate the joint frequency response. After that, An inverse Fourier transform (IFFT) is performed to restore the spatial domain, resulting in a preliminary attention map. Then, normalization (Norm) is applied along the channel dimension to generate the final spatial attention weights. The obtained attention weights are then used to weight and modulate the original RGB and Event features, outputting the modulated features. and The specific formula is as follows:

[0077]

[0078]

[0079]

[0080] Step 4.2: Multi-scale feature extraction. For the input features... and , respectively , , Parallel branches of three different convolutional kernels extract features, capturing multi-scale information from local details to global context. The outputs of the three branches are then concatenated along the channel dimension and processed through a... Convolutional layers are fused to generate cross-modal attention weights. and .

[0081] Step 4.3: Using weights and Modulation characteristics and ,get , :

[0082]

[0083]

[0084] Modulation using a self-attention mechanism yields... , :

[0085]

[0086]

[0087] Modulated , , , and original features and Output via residual connection and :

[0088]

[0089]

[0090] Step 4.4: Higher-order channel statistical enhancement. This step employs second-order and higher-order statistical layers, capturing complex channel interactions through an iterative attention mechanism. Each order includes a cascaded structure of adaptive average pooling and two fully connected layers (containing ReLU activation and Sigmoid output). Let the order be... ,right and conduct The weighted summation for each iteration is as follows:

[0091]

[0092]

[0093] in For the first The channel weights generated by the order, The input features for each iteration, This structure preserves the original features through residual connections, achieving cross-modal channel-level feature alignment and enhancement.

[0094] Step 4.5: Residual Connection and Adaptive Weight Fusion. The results obtained in Step 4.3... and step 4.4 Weighted after the next iteration Obtained through residual connection Event modality and Obtained through residual connection .Will and The parts are then pieced together to form the final fused feature. :

[0095]

[0096] This fusion module captures global structural dependencies through frequency domain interaction, enhances local details through multi-scale context, and achieves semantic association through high-order channel statistical modeling, realizing deep complementarity and synergy between RGB and Event modalities in the three dimensions of time, space, and channel.

[0097] Step 5: The target is fed into the tracking head for target classification and bounding box regression to obtain the final target tracking result.

[0098] Step 5.1: Classification Feature Extraction and Score Prediction. Classification features are extracted from the fused multi-scale features and fed into a classification feature extractor based on the Discriminant Model Prediction (DiMPNet) framework. A linear classification filter is trained using the features of the template frame, and this filter is cross-correlated with the fused features of the search region to generate a classification score map, where the score at each location represents the confidence level that a tracked target exists at that location.

[0099] Step 5.2: Bounding Box Regression and Location Refinement. Multi-scale features for bounding box regression are extracted from the fused features and input into a regression network based on IoU prediction. A series of candidate bounding boxes are generated in the search region. The regression network predicts the Intersection over Union (IoU) between each candidate box and the ground truth bounding box, obtaining an IoU score map. Combining the classification score and IoU score from Step 5.1, all candidate boxes are weighted and ranked, and high-scoring candidate boxes undergo refined location offset regression to predict center point shift. With scale adjustment This enables precise positioning of the bounding box.

[0100] Step 5.3: Tracking Result Integration and Output. The classification score and IoU score are weighted and fused to obtain the final confidence score for each candidate bounding box. Non-Maximum Suppression (NMS) is used to select the most representative bounding boxes. The bounding box with the highest confidence is selected as the tracking result for the current frame, and its center coordinates are output. With size Finally, based on the tracking confidence and the degree of change in the target's appearance, the target template features are adaptively updated to take into account the target's deformation, rotation, and illumination changes during the tracking process.

[0101] This tracking head module fully leverages the complementary advantages of dual-modal fusion features and achieves a balance between accuracy and efficiency through multi-task learning of classification and regression tasks, providing a complete solution for reliable tracking in complex dynamic scenarios.

[0102] The apparatus for implementing the method includes Figure 1 The system consists of three parts: a template branch, a search branch, and a tracking head. The search branch has the same structure as the template branch and shares network weights. The RGB modality is directly fed into the ResNet-18 network for feature extraction, while the Event modality is first fed into an adaptive neuron module for temporal filtering and information enhancement, and then into the ResNet-18 network for feature extraction. The extracted features from the two modalities are then fused into a higher-order spatiotemporal feature fusion module. The fused features from the template and search branches are fed into the tracking head, where a classifier generates confidence scores and a regressor generates the intersection-over-union (IoU) ratio. Finally, based on the confidence scores and IoU ratio, a prediction result for target tracking is generated.

[0103] like Figure 2 As shown, the adaptive neuron module comprises three parts: adaptive threshold generation, dynamic membrane time constant adaptation, and neighborhood weighting. The adaptive threshold is dynamically adjusted through three learnable parameters. The membrane time constant is dynamically adapted to the network input based on max pooling and average pooling. Neighborhood weighting uses neurons based on the adaptive threshold and dynamic membrane time constant to perform temporal filtering and information enhancement on the input.

[0104] High-order spatiotemporal feature fusion module: such as Figure 3 As shown, the system comprises two parts: the frequency domain and the spatial-channel domain. In the frequency domain, Fourier transform and inverse Fourier transform are used to enhance the feature interaction between the RGB and Event modalities. In the spatial-channel domain, multi-scale features of the two modalities are extracted using convolutional kernels of different sizes, and complex channel interactions are captured through an iterative attention mechanism using a high-order statistical layer. Finally, the dual-modal features, after deep interaction in the frequency, spatial, and channel domains, are concatenated to obtain the final fused features.

[0105] Figure 4 This paper compares the quantitative tracking results of the method of this invention with those of seven representative conventional target tracking methods on the VisEvent dataset. Analysis of the PR curve in Figure (a) shows that the tracking results of this invention exceed CLNet by 3.1% in accuracy. As can be seen from the SR curve in Figure (b), the tracking results of this invention exceed CLNet by 1% in success rate.

[0106] Table 1 compares the quantitative tracking results of the method of this invention with those of seven representative conventional target tracking methods on the FE108 dataset. RPR represents the accuracy of the PR curve when the threshold is set to 20 pixels. PSR represents the success rate of the SR curve when the threshold is set to 0.5. As can be seen from the RPR and RSR metrics in the table, the tracking results of this invention outperform other algorithms in both accuracy and success rate.

[0107] Table 1 Experimental results on the FE108 dataset

[0108]

[0109] Figure 5 This figure compares the qualitative tracking results of the method of this invention with conventional target tracking methods on the VisEvent dataset. The red boxes indicate the target position and size tracked by the method of this invention, while the green boxes represent the actual target position and size. The qualitative analysis includes complex situations such as high-speed target movement and low illumination. As can be seen from the figure, the Event frame has a clearer target edge representation compared to the RGB frame and does not contain static background information. The examples in the figure show that the target tracking results obtained using the method of this invention are more ideal compared to other methods.

[0110] The above analysis and verification results show that the spatiotemporal feature fusion target tracking method based on RGB / Event disclosed in this embodiment can improve the target tracking accuracy and efficiency in complex scenarios such as high dynamic range, high-speed target movement, and low illumination.

[0111] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is merely a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A target tracking method based on spatiotemporal feature fusion of RGB / Event, characterized in that: Includes the following steps: Step 1: Synchronous acquisition, timing alignment, and preprocessing of RGB-E dual-modal data to obtain RGB template frames. and RGB search frame Event template frame and Event search frame ; Step 2: Based on the three learning weight parameters , , ;right and Global average pooling and global max pooling are performed separately, and the mixture is fed into a fully connected network and mapped to the interval [0.5, 0.95] to dynamically generate the membrane time constant. Local neighborhood weighting is performed using 3×3 grouped convolution, based on... Update the membrane potential, and when the membrane potential exceeds the dynamic threshold Timely pulse delivery to achieve control and Temporal filtering and information enhancement yielded and ; Step 3: Based on the results obtained in Step 1 and and the result obtained in step two and They share the same ResNet-18 backbone network for feature extraction to obtain spatially aligned and semantically consistent bimodal feature representations. and ; Step 4: Fuse the dual-modal features to obtain ; Step 5: The data is fed into the tracking head to obtain the final target tracking result, achieving spatiotemporal feature fusion target tracking.

2. The method as described in claim 1, characterized in that: The implementation method for step 1 is as follows: Step 1.1: Synchronously acquire aligned RGB image and event stream data; the event stream data consists of a series of quadruple events. Composition, among which, and Represents the spatial coordinates of the event. The timestamp indicates when the event occurred; p indicates the polarity of the event; Step 1.2: To preserve the high-resolution temporal information of the event stream data and maintain time alignment with the RGB images, the exposure time T of the RGB images is evenly divided into n time windows. Within each time window, the event stream data is aggregated into event frames. The value of n typically ranges from 2 to 8. For the i-th event stream data... It does not distinguish between positive and negative polarities, and is located at the corresponding position where the event occurs. Pixel values ​​are accumulated; finally, each RGB image generates n corresponding event frames, each event frame image has 1 channel, and the resolution is the same as the RGB frame, and the tracking target position corresponds to the RGB frame; Step 1.3: Perform preprocessing on the RGB image obtained in Step 1.1 to obtain the RGB template frame. and RGB search frame Preprocess the n event frames corresponding to the RGB images obtained in step 1.2 to obtain Event template frames. and Event search frame .

3. The method as described in claim 1, characterized in that: The three learning weight parameters mentioned in step 2 , , These are, respectively, event density weights Calculated using the global pixel average of the current event frame, it reflects the sparsity of the event; Motion clustering weights By calculating the variance of events within a local 3×3 neighborhood, moving targets typically generate event clusters with high variance, thus this can be used to assess the spatial clustering of events; temporal variation weights. The dynamic threshold is calculated by averaging the absolute differences between the current frame and the previous frame to measure the intensity of dynamic changes in the scene; the final dynamic threshold is... pass , , It is obtained by weighted summation with the corresponding density, motion pattern, and temporal changes.

4. The method as described in claim 1, characterized in that: Step 2 describes the dynamic generation of the membrane time constant. The method is as follows: in, and These are average pooling and max pooling, respectively. and For activation function, and These are learnable weights; This design allows neurons to adjust their membrane time constant according to their response characteristics. This allows for adaptive adjustment of the information decay rate, enhancing the model's adaptability to various motion modes.

5. The method as described in claim 1, characterized in that: Step 2 describes the completion of the task. and The temporal filtering and information enhancement method uses a 3×3 grouped convolution kernel to... and Local neighborhood information fusion is performed, with convolutional kernel weights initialized to 0.2 for the center pixel and 0.1 for the surrounding pixels to enhance the spatial correlation of events. The weighted feature input is based on the leaky integral-fire LIF model's spiking neurons, whose membrane potential... It will increase as input events accumulate: in The time constant of the dynamic membrane. The timestamp of the current moment. The input at the current moment is used when the membrane potential exceeds the dynamic threshold. At that time, neurons fire impulses Subsequently, the membrane potential recovers according to the reset rule, and the neuron eventually outputs a pulse weight matrix with the same shape as the input. Multiplying the weight matrix element-wise with the original input can enhance the motion event and filter out noise, resulting in the enhanced version. and .

6. The method as described in claim 1, characterized in that: The implementation method for step 3 is as follows: Step 1 and The input is passed through a ResNet-18 network, sequentially through convolutional layers, batch normalization layers, and activation functions, to extract multi-scale spatial feature maps. ; will be from step 2 and Input the same ResNet-18 network to extract spatiotemporal feature maps containing temporal motion information. ResNet-18 contains outputs from multiple layers. The output of Layer 2 is selected for precise target localization, and the output of Layer 3 is selected for target recognition and contextual understanding.

7. The method as described in claim 1, characterized in that: The implementation method for step 4 is as follows: Step 4.1: Process the RGB feature map obtained in Step 3. Event Feature Map Perform two-dimensional fast Fourier transforms to obtain the frequency domain representation. and The element-wise product of the two modal features is calculated in the frequency domain to generate the joint frequency response. ; after that Perform an inverse Fourier transform to restore the spatial domain and obtain a preliminary attention map. Then, normalize along the channel dimension to generate the final spatial attention weights. The obtained attention weights are combined with and Perform weighted modulation and output the modulated features. and ; Step 4.2: Feature Analysis and , respectively , , Three different convolutional kernels are used to extract features, capturing multi-scale information from local details to global context; the three outputs are concatenated along the channel dimension and then processed through a single... Convolutional layers are fused to generate cross-modal attention weights. and ; Step 4.3: Using weights and Modulation characteristics and ,get , ; Modulated using a self-attention mechanism , Modulated , , , and original features and Output via residual connection and ; Step 4.4: Use second-order or higher statistical layers to... and conduct Weighted sub-iteration: in For the first The channel weights generated by the order, The input features for each iteration, , Step 4.5: Take the result obtained in Step 4.3 and step 4.4 Weighted after the next iteration Obtained through residual connection Event modality and Obtained through residual connection ;Will and The parts are then pieced together to form the final fused feature. ; This fusion module captures global structural dependencies through frequency domain interaction, enhances local details through multi-scale context, and achieves semantic association through high-order channel statistical modeling, realizing deep complementarity and synergy between RGB and Event modalities in the three dimensions of time, space, and channel.

8. The method as described in claim 1, characterized in that: The implementation method for step 5 is as follows: Will The data is fed into a classifier, which performs cross-correlation between the target template features and the search region features to generate a classification score map. Then, the probability distribution of the target location is obtained by using a Softmax or Sigmoid activation function. Will The data is fed into a regressor, which modulates the initial target box using the predicted position offset. The bounding box is then filtered using the IoU score, and redundant detections are removed using non-maximum suppression. By selecting the candidate box with the highest classification score as the final tracking result, the center coordinates, height, and width of the bounding box corresponding to the regressor are output, and the target template features are updated to adapt to changes in template appearance.

9. An apparatus for implementing the method according to any one of claims 1 to 8, characterized in that: It consists of three parts: a template branch, a search branch, and a tracking head. The search branch and the template branch have the same structure and share network weights. The RGB modality is directly fed into the ResNet-18 network for feature extraction, while the Event modality is first fed into an adaptive neuron module for temporal filtering and information enhancement, and then into the ResNet-18 network for feature extraction. The features extracted from the two modalities are then fused into a higher-order spatiotemporal feature fusion module. The features fused from the template branch and the search branch are fed into the tracking head, where a classifier generates confidence scores and a regressor generates the intersection-over-union ratio (IoU). Based on the confidence scores and IoU, a prediction result for target tracking is generated. The adaptive neuron module consists of three parts: adaptive threshold generation, dynamic membrane time constant adaptation, and neighborhood weighting. The adaptive threshold is dynamically adjusted through three learning parameters; the membrane time constant is dynamically adapted to the network input based on max pooling and average pooling; and the neighborhood weighting uses neurons based on the adaptive threshold and dynamic membrane time constant to perform temporal filtering and information enhancement on the input. The high-order spatiotemporal feature fusion module consists of two parts: the frequency domain and the spatial-channel domain. In the frequency domain, Fourier transform and inverse Fourier transform are used to enhance the feature interaction between the RGB and Event modalities. In the spatial-channel domain, multi-scale features of the two modalities are extracted by convolutional kernels of different sizes, and complex channel interactions are captured by an iterative attention mechanism of a high-order statistical layer. Finally, the dual-modal features after deep interaction in the frequency domain, spatial domain, and channel domain are concatenated to obtain the final fused features.

Images