A spiking neural network (SNN) object tracking method and system fusing events and RGB images

Abstract

The application discloses a kind of SNN target tracking method and system of fusion event and RGB image, the method of the application includes by time stack-based representation method, event stream from event camera is converted into event frame image, and with the RGB frame image from RGB camera is respectively aligned in time and space;Event frame image and RGB frame image after alignment are detected using pulse neural network SNN, the pulse neural network SNN includes feature extraction module, feature fusion module and detection head;The similarity of the anchor frame extracted using pulse neural network SNN and fusion feature map is used to learn the target to track multiple targets.The application aims to realize the fusion of fusion event and RGB image to ensure the high accuracy and high efficiency of multi-target tracking, and maintain robust detection performance and tracking performance in static and high motion scene.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology in the field of computer vision, and in particular to a SNN target tracking method and system for fusing events and RGB images. Background Art

[0002] Multi-target tracking is a computer vision technique that aims to simultaneously detect and track multiple objects in a series of consecutive video frames. Its core task is to identify different targets in a scene and continuously track the position and motion trajectory of each target throughout the entire video sequence. Currently, mainstream multi-target tracking methods are primarily based on artificial neural networks (ANNs). While ANNs perform well in many computer vision tasks, they differ significantly from biological neural systems and struggle to simulate the complex functions of the brain. When processing information, all neurons activate simultaneously, resulting in high computational and energy consumption, and potentially limited by computational latency in real-time applications. Spiking neural networks (SNNs), on the other hand, more closely resemble the workings of biological neurons, offering greater bioinspiration. They activate only when spikes are received, resulting in higher energy efficiency. Furthermore, SNNs, through their event-driven mechanism, can effectively process sparse and irregular data, making them naturally suited to processing dynamic, time-dependent data. Deployment on hardware platforms such as brain-inspired chips also allows for low latency.

[0003] Compared to traditional RGB cameras, event cameras' superior temporal resolution enables them to capture rich information between frames, avoiding the effects of motion blur, making them prime candidates for target tracking. They also offer advantages such as low energy consumption and high dynamic range, and can operate even under overexposure conditions. However, in real-world environments, using either camera individually has its advantages and disadvantages. Event cameras provide precise temporal and motion information, while RGB cameras provide rich spatial and color information. Fusion can achieve better perception in dynamic scenes, improving the accuracy and robustness of visual tasks such as detection and tracking. Meanwhile, current mainstream multi-target tracking methods are primarily based on artificial neural networks (ANNs). While ANNs perform well in many computer vision tasks, they differ significantly from biological neural systems and struggle to simulate the complex functions of the brain. When processing information, all neurons activate simultaneously, resulting in high computational and energy consumption, and potentially limited by computational latency in real-time applications. Spiking neural networks (SNNs), on the other hand, more closely resemble the workings of biological neurons and offer greater bioinspiration. They activate only when spikes are received, resulting in higher energy efficiency. Furthermore, SNNs, through their event-driven mechanism, can effectively process sparse and irregular data, making them naturally suited to handling dynamic, time-dependent data. Deployment on hardware platforms such as brain-inspired chips also allows for low latency. Therefore, combining event data with SNNs can ensure high-precision and high-efficiency multi-target tracking. However, how to achieve the integration of these two approaches to ensure high-precision and high-efficiency multi-target tracking remains a key technical challenge that needs to be addressed. Summary of the Invention

[0004] Technical problem to be solved by the present invention: In response to the above-mentioned problems of the prior art, a SNN target tracking method and system for fusing events and RGB images are provided. The present invention aims to achieve the fusion of fused events and RGB images to ensure high precision and high efficiency of multi-target tracking, and maintain robust detection performance and tracking performance in static and high-motion scenes.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A method for tracking an object by using a SNN that fuses events and RGB images comprises the following steps:

[0007] S1, converts the event stream from the event camera into event frame images through a time stacking-based representation method, and aligns them with the RGB frame images from the RGB camera in time and space respectively;

[0008] S2, the aligned event frame image and RGB frame images Target detection is performed using a pulse neural network (SNN), which includes a feature extraction module, a feature fusion module, and a detection head. The pulse neural network performs target detection by respectively extracting the aligned event frame images through the feature extraction network. and RGB frame images Perform feature extraction and use the feature fusion module to extract the event frame feature map and RGB frame feature map Fusion obtains fusion feature map ; Through the detection head to fuse feature maps Perform target detection to generate anchor boxes;

[0009] S3, using anchor boxes and fusion feature maps To learn the similarity of targets for multi-target tracking.

[0010] Optionally, in step S1, converting the event stream from the event camera into an event frame image by a time stacking-based representation method includes: for a specified start time and end time Selected event streams between , among which events Contains pixel coordinates , in microseconds The timestamp in units of 1 and the polarity indicating the brightness change , is the number of events; the event stream will be selected Divided into time-based For each bin, the normalized timestamp of the most recent event is calculated according to the following formula: and the normalized timestamp of the most recent event The pixel values ​​assigned to each bin are:

[0011] ,

[0012] ,

[0013] In the above formula, An event frame image in the form of a three-dimensional tensor Center coordinates The pixel value of Represents the three dimensions of pixel x coordinate, pixel y coordinate and timestamp, event frame image The sizes of the three dimensions of pixel x coordinate, pixel y coordinate and timestamp are ,function The function expression is:

[0014] ,

[0015] in, For function The independent variable, Indicates taking the maximum value.

[0016] Optionally, the step S1 of aligning the RGB frame images from the RGB camera in time and space respectively includes: using the trigger timestamp and acquisition frequency of the RGB frame images to make the start accumulation time of the event frame image equal to the trigger time of the corresponding RGB frame image, and taking the time step of the acquisition frequency of the RGB frame as The event camera and the RGB frame camera are aligned in time, where is the starting time, The end time is 0. The intrinsic and extrinsic parameters of the event camera and RGB frame camera are obtained by the specified camera calibration method. The intrinsic and extrinsic parameters are used to dedistort the event frame image and the RGB frame image and perform matrix transformation to synchronize the event camera and the RGB frame camera in space.

[0017] Optionally, the feature extraction network in step S2 includes a backbone network and a multi-scale module, wherein the backbone network is used to transform the aligned event frame images into and RGB frame images Extract feature maps of three different scales respectively; the multi-scale network is a batch normalization module with pulse convolution Feature pyramid network based on path aggregation , the feature pyramid network The processing of the three different scale feature maps output by the backbone network includes: passing the first scale feature map through the first pulse convolution batch normalization module After processing and upsampling, it is connected with the second scale feature map and then passes through the combination module CSP2_1 and the second pulse convolution batch normalization module After processing and upsampling, it is connected with the third scale feature map and then processed by a combination module CSP2_1 to obtain the first scale feature; the first scale feature is processed by the third pulse convolution batch normalization module Processing, and then with the second pulse convolution batch normalization module After the features are processed, they are connected and then processed by a combination module CSP2_1 to obtain the features of the second scale; the features of the second scale are processed by the fourth pulse convolution batch normalization module Processing, and then with the first pulse convolution batch normalization module After the processed features are connected, they are processed by a combination module CSP2_1 to obtain the features of the third scale; the features of the three scales are then passed through a multi-head attention head to obtain the event frame feature map extracted by the feature extraction network Or RGB frame feature map The combined module CSP2_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module Then connected to a 2-cascaded pulse convolution batch normalization module , and the two branches are connected through a connection module and then pass through a pulse convolution batch normalization module Get the corresponding output features; the pulse convolution batch normalization module The function expression is:

[0018] ,

[0019] In the above formula, For the spike convolution batch normalization module The output features of For the spike convolution batch normalization module The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The features obtained with The function expression of the convolutional layer inference of quantized neurons is:

[0020] ,

[0021] ,

[0022] In the above formula, is the time step integer-valued impulse tensor of , is the clipping function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into a binary impulse tensor.

[0023] Optionally, the backbone network includes a focusing module, a first pulse convolution batch normalization module connected in sequence , combined module CSP1_1, the second pulse convolution batch normalization module , the first combination module CSP1_3, the third pulse convolution batch normalization module , the second combination module CSP1_3, the fourth pulse convolution batch normalization module , combination module SPP and combination module CSP2_1, where the output of the first combination module CSP1_3 is the first feature map output by the backbone network, the output of the second combination module CSP1_3 is the second feature map output by the backbone network, and the output of CSP2_1 is the third feature map output by the backbone network; the focusing module has multiple parallel slicing modules, a connection module and a pulse convolution batch normalization module The combined module CSP1_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module There is a cascaded residual module Res after the connection, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module Get the corresponding output features; the combined module CSP1_3 includes two parallel branches, one branch is a pulse convolution batch normalization module , the other branch is a pulse convolution batch normalization module There are three cascaded residual modules Res at the end, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module. Get the corresponding output features; the combination module SPP includes a pulse convolution batch normalization module The three largest pooling branches connected later, and the outputs of the three largest pooling branches and the pulse convolution batch normalization module The outputs are connected together through a connection module and then passed through a pulse convolution batch normalization module. Get the corresponding output features; the residual module Res includes two pulse convolution batch normalization modules cascaded in sequence And a residual connection, one input of which comes from the second pulse convolution batch normalization module The output of , and the other input comes from the input of the residual module Res.

[0024] Optionally, the function expression of the fusion module is:

[0025] ,

[0026] In the above formula, represents the fused feature map, Indicates with Convolutional networks of quantized neurons, represents the feature map after concatenation and merging, Represents the event frame feature map, and has:

[0027] ,

[0028] In the above formula, Indicates concatenation merging, Represents the RGB frame feature map; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The pulse convolution batch normalization module is constructed in series. The function expression is:

[0029] ,

[0030] In the above formula, For the spike convolution batch normalization module The output features of For the spike convolution batch normalization module The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The features obtained with The function expression of the convolutional layer inference of quantized neurons is:

[0031] ,

[0032] ,

[0033] In the above formula, is the time step integer-valued impulse tensor of , is the clipping function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into a binary impulse tensor.

[0034] Optionally, in step S3, the anchor frame and the fusion feature map are used The method of learning target similarity to track multiple targets includes: taking the current frame as the key frame to be processed, randomly selecting a reference frame from the temporal neighborhood of the key frame, and using the pre-trained ReID network to calculate the embedded feature vectors of the key frame and the reference frame according to the following formula:

[0035] ,

[0036] ,

[0037] In the above formula, is the embedded feature vector of the key frame, is the embedded feature vector of the reference frame, is a fully connected neural network, Indicates with Convolutional networks of quantized neurons, For the region of interest alignment operation, is the fusion feature map of the key frame , is the fusion feature map of the reference frame , is the anchor box of the key frame, is the anchor frame of the reference frame; the functional expression of the loss function used in the ReID network training is:

[0038] ,

[0039] In the above formula, is the loss function, 、 、 are respectively the embedded feature vectors of the key frames of the training samples and the positive and negative samples of the training samples in the reference frames; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The pulse convolution batch normalization module is constructed in series. The function expression is:

[0040] ,

[0041] In the above formula, For the spike convolution batch normalization module The output features of For the spike convolution batch normalization module The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The features obtained with The function expression of the convolutional layer inference of quantized neurons is:

[0042] ,

[0043] ,

[0044] In the above formula, is the time step integer-valued impulse tensor of , is the clipping function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into a binary impulse tensor.

[0045] In addition, the present invention also provides an SNN target tracking system for fusing events and RGB images, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the SNN target tracking method for fusing events and RGB images.

[0046] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the SNN target tracking method of the fusion event and RGB image through a processor.

[0047] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the SNN target tracking method of fusion events and RGB images through a processor.

[0048] Compared with the prior art, the present invention mainly has the following advantages:

[0049] 1. The present invention uses a time stacking (SBT)-based event stream representation method to generate high-frame-rate event images that are less prone to motion blur while retaining more useful information.

[0050] 2. The present invention effectively fuses event and RGB data through a multimodal fusion method, avoiding the insufficient extraction of temporal and spatial information of samples by a single modality. This fusion method retains the clear image of the RGB camera in space, while obtaining high-precision motion information of the event camera in time. The two complement each other in tracking, detection and other tasks, maintaining robust detection and tracking performance in both static and high-motion scenes, and improving detection accuracy and tracking continuity.

[0051] 3. The present invention constructs a network model of a pulse neural network combined with quantized neurons, which reduces power consumption while maintaining performance comparable to that of ANN. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0053] Figure 2 Schematic diagram of the structure and working principle of the spiking neural network SNN in an embodiment of the present invention.

[0054] Figure 3 Schematic diagram of the network structure of the feature extraction network and its multi-scale network in an embodiment of the present invention.

[0055] Figure 4 Schematic diagram of the network structure of the combination module CSP2_1 in an embodiment of the present invention.

[0056] Figure 5 This is the pulse convolution batch normalization module in the embodiment of the present invention Schematic diagram of the network structure.

[0057] Figure 6 Schematic diagram of the network structure of the backbone network in an embodiment of the present invention.

[0058] Figure 7 Schematic diagram of the network structure of the focusing module in an embodiment of the present invention.

[0059] Figure 8Schematic diagram of a general network structure of the combination module CSP1_1 and the combination module CSP1_3 in an embodiment of the present invention.

[0060] Figure 9 Schematic diagram of the network structure of the combined module SPP in an embodiment of the present invention.

[0061] Figure 10 Schematic diagram of the network structure of the residual module Res in an embodiment of the present invention.

[0062] Figure 11 Schematic diagram of the network structure of the fusion module in an embodiment of the present invention.

[0063] Figure 12 Schematic diagram of the principle of multi-target tracking in an embodiment of the present invention.

[0064] Figure 13 Schematic diagram of the working principle of the ReID network in an embodiment of the present invention.

[0065] Figure 14 2 is a comparison of the tracking results of the spiking neural network (SNN) in an embodiment of the present invention, where (a) to (c) are the traditional single-image modality tracking methods, and (d) to (f) are the multi-modal tracking results of the spiking neural network (SNN). DETAILED DESCRIPTION

[0066] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0067] like Figure 1 As shown, the SNN target tracking method for fusing events and RGB images in this embodiment includes the following steps:

[0068] S1, converts the event stream from the event camera into event frame images through a time stacking-based representation method, and aligns them with the RGB frame images from the RGB camera in time and space respectively;

[0069] S2, the aligned event frame image and RGB frame images Use pulse neural network SNN for target detection, such as Figure 2 As shown, the pulse neural network SNN in this embodiment includes a feature extraction module, a feature fusion module and a detection head. The pulse neural network performs target detection by respectively extracting the aligned event frame images through the feature extraction network. and RGB frame images Perform feature extraction and use the feature fusion module to extract the event frame feature map and RGB frame feature map Fusion obtains fusion feature map ; Through the detection head to fuse feature maps Perform target detection to generate anchor boxes;

[0070] S3, using anchor boxes and fusion feature maps To learn the similarity of targets for multi-target tracking.

[0071] In this embodiment, since an asynchronous event stream is used as the input of the neural network, this event representation method is used to convert the event stream into a matrix similar to a tensor while retaining as much useful information as possible. Specifically, in step S1 of this embodiment, the event stream from the event camera is converted into an event frame image using a time stacking-based representation method, including: for the specified start time and end time Selected event streams between , among which events Contains pixel coordinates , in microseconds The timestamp in units of 1 and the polarity indicating the brightness change , is the number of events; the event stream will be selected Divided into time-based For each bin, the normalized timestamp of the most recent event is calculated according to the following formula: and the normalized timestamp of the most recent event The pixel values ​​assigned to each bin are:

[0072] ,

[0073] ,

[0074] In the above formula, An event frame image in the form of a three-dimensional tensor Center coordinates The pixel value of Represents the three dimensions of pixel x coordinate, pixel y coordinate and timestamp, event frame image The sizes of the three dimensions of pixel x coordinate, pixel y coordinate and timestamp are ,function The function expression is:

[0075] ,

[0076] in, For function The independent variable, Indicates taking the maximum value. In some embodiments, the event frame image and the RGB frame camera image are aligned in time and space respectively, including: using a trigger signal to perform time alignment, and using the internal and external parameters of the event camera and the RGB frame camera to perform spatial alignment. In this embodiment, the event camera and the RGB frame camera are synchronized in time by using the trigger timestamp and acquisition frequency of the RGB frame image; the internal and external parameters of the event camera and the RGB frame camera are obtained by some calibration methods to perform dedistortion and matrix transformation on the image, so that the event camera and the RGB frame camera are synchronized in space, and then the label on the RGB frame image is also the event image label. In step S1 of this embodiment, the alignment in time and space with the RGB frame image from the RGB camera includes: using the trigger timestamp and acquisition frequency of the RGB frame image to make the start accumulation time of the event frame image equal to the trigger time of the corresponding RGB frame image, and taking the time step of the acquisition frequency of the RGB frame as (Specifically in this embodiment ) so that the event camera and the RGB frame camera are temporally aligned, where is the starting time, The end time is obtained by the specified camera calibration method, and the internal and external parameters of the event camera and the RGB frame camera are obtained. The internal and external parameters are used to dedistort the event frame image and the RGB frame image and perform matrix transformation, so that the event camera and the RGB frame camera are synchronized in space. Specifically, in this embodiment, the event frame image and the RGB frame image are aligned in time and space respectively, by using the trigger timestamp and acquisition frequency of the RGB frame image. , so that the start accumulation time of the event frame image is equal to the trigger time of the corresponding RGB frame image, and the acquisition frequency of the RGB frame To get the time step , so that the event camera and RGB frame camera are synchronized in time, and the internal and external parameters of the event camera and RGB frame camera are obtained through the checkerboard and Zhang calibration methods to perform image dedistortion and matrix transformation, so that the event camera and RGB frame camera are synchronized in space, that is, the label on the RGB frame image is also the label of the event frame image.

[0077] The feature extraction of event frame images is similar to that of RGB frame images. Figure 3 As shown, the feature extraction network in step S2 of this embodiment includes a backbone network and a multi-scale module. The backbone network is used to transform the aligned event frame image into and RGB frame images Extract feature maps of three different scales respectively; the multi-scale network is a batch normalization module with pulse convolution Feature pyramid network based on path aggregation ,like Figure 3As shown, the feature pyramid network The processing of the three different scale feature maps output by the backbone network includes: passing the first scale feature map through the first pulse convolution batch normalization module After processing and upsampling, it is concatenated with the second scale feature map and then passes through the combination module CSP2_1 and the second pulse convolution batch normalization module. After processing and upsampling, it is connected with the third scale feature map and then processed by a combination module CSP2_1 to obtain the first scale feature; the first scale feature is processed by the third pulse convolution batch normalization module Processing, and then with the second pulse convolution batch normalization module After processing the features obtained, they are concatenated and then processed by a combination module CSP2_1 to obtain the features of the second scale; the features of the second scale are processed by the fourth pulse convolution batch normalization module Processing, and then with the first pulse convolution batch normalization module After the processed features are connected, they are processed by a combination module CSP2_1 to obtain the features of the third scale; the features of the three scales are then passed through a multi-head attention head to obtain the event frame feature map extracted by the feature extraction network Or RGB frame feature map .

[0078] like Figure 4 As shown, the combined module CSP2_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module Then connected to a 2-cascaded pulse convolution batch normalization module , and the two branches are connected through a connection module and then pass through a pulse convolution batch normalization module Get the corresponding output features; such as Figure 5 As shown, the pulse convolution batch normalization module The function expression is:

[0079] ,

[0080] In the above formula, For the spike convolution batch normalization module The output features of For the spike convolution batch normalization module The input features of ,in Indicates the dimension, T represents the time step of the spiking neural network, C is the number of channels of the tensor, W and H are the width and height of the tensor, is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The features obtained with The function expression of the convolutional layer inference of quantized neurons is:

[0081] ,

[0082] ,

[0083] In the above formula, is the time step integer-valued impulse tensor of , is the clipping function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step integer-valued impulse tensor of , To split the integer-valued impulse tensor into the indices of the binary impulse tensor, The time step is extended to , and the integer value Convert to pulse train For example, if ,but Pulse sequence Contains only , so all multi-accumulation operations can be converted to sparse accumulation operations, which can ensure the pulse driving during inference. As neurons of the spiking neural network, using integer values ​​during training helps reduce quantization errors and makes training more stable. At the same time, converting them into binary pulses during inference retains the low energy consumption of the spiking neural network.

[0084] like Figure 6 As shown, the backbone network consists of a sequentially connected focusing module, a first pulse convolution batch normalization module , combined module CSP1_1, the second pulse convolution batch normalization module , the first combination module CSP1_3, the third pulse convolution batch normalization module , the second combination module CSP1_3, the fourth pulse convolution batch normalization module , combination module SPP and combination module CSP2_1, where the output of the first combination module CSP1_3 is the first feature map output by the backbone network, the output of the second combination module CSP1_3 is the second feature map output by the backbone network, and the output of CSP2_1 is the third feature map output by the backbone network.

[0085] like Figure 7 As shown, the focusing module has multiple parallel slicing modules, a connection module and a pulse convolution batch normalization module. Connected in sequence.

[0086] Figure 8 Schematic diagram of the general network structure of the combination module CSP1_1 and the combination module CSP1_3 in the embodiment of the present invention. The combination modules CSP1_1 to CSP1_3 are represented as combination modules CSP1_a. The difference is that the number a of residual modules Res is different, that is, the number a of residual modules Res in the combination module CSP1_1 is 1, the number a of residual modules Res in the combination module CSP1_2 is 2, and the number a of residual modules Res in the combination module CSP1_3 is 3. Figure 8 As shown, the combined module CSP1_1 includes two parallel branches, one branch is a pulse convolution batch normalization module , the other branch is a pulse convolution batch normalization module There is a cascaded residual module Res after the connection, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module Get the corresponding output features; the combined module CSP1_3 includes two parallel branches, one branch is a pulse convolution batch normalization module , the other branch is a pulse convolution batch normalization module There are three cascaded residual modules Res at the end, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module. Get the corresponding output features;

[0087] like Figure 9 As shown, the combined module SPP includes a pulse convolution batch normalization module The three largest pooling branches connected later, and the outputs of the three largest pooling branches and the pulse convolution batch normalization module The outputs are connected together through a connection module and then passed through a pulse convolution batch normalization module. Get the corresponding output features; among them, the output of the three maximum pooling branches and the pulse convolution batch normalization module The output is equivalent to forming a maximum pooling method of 1×1, 5×5, 9×9 and 13×13 for multi-scale feature fusion.

[0088] like Figure 10 As shown, the residual module Res consists of two pulse convolution batch normalization modules cascaded in sequence. And a residual connection, one input of which comes from the second pulse convolution batch normalization module The output of , and the other input comes from the input of the residual module Res.

[0089] After feature extraction, the event frame feature map and the RGB frame feature map are obtained, and the event frame feature map and the RGB frame feature map are transmitted to the constructed feature fusion module for feature fusion to obtain a fused feature map. In this embodiment, by effectively utilizing the advantages of the pulse neural network and feature extraction, high-quality feature extraction capabilities are maintained in static and high-motion scenes. In some embodiments, such as Figure 11 As shown, the function expression of the fusion module is:

[0090] ,

[0091] In the above formula, represents the fused feature map, Indicates with Convolutional networks of quantized neurons, represents the feature map after concatenation and merging, Represents the event frame feature map, and has:

[0092] ,

[0093] In the above formula, Indicates concatenation merging, Figure 11 The circled c represents Represents the RGB frame feature map; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The series structure will not be described in detail here.

[0094] In this embodiment, feature fusion aims to integrate RGB image frames and event representations of different frequencies, extract complementary information to generate high-frequency fusion features. By effectively utilizing the advantages of both modes, robust tracking performance is maintained in static and high-motion scenes. In some embodiments, the fused feature map is used to The pulse neural network detection head under the framework of Detection performs target detection and generates a series of anchor boxes, including: The pulse neural network detection head under the framework performs gradient feedback training of the target detection loss function, and obtains a series of fused feature maps with the same number of channels and relatively close to the real target. and a series of candidate anchor boxes of categories. In this embodiment, by using The framework pulse neural network detection head provides high-quality candidate tracking targets for the subsequent multi-target tracking process with high detection accuracy and low power consumption, and uses a series of fused feature maps with the same number of channels and a series of candidate anchor boxes to learn the similarity of the targets. In some embodiments, such as Figure 12 and Figure 13 As shown, in step S3, the anchor frame and the fusion feature map are used The method of learning target similarity to track multiple targets includes: taking the current frame as the key frame to be processed, randomly selecting a reference frame from the temporal neighborhood of the key frame, and using the pre-trained ReID network to calculate the embedded feature vectors of the key frame and the reference frame according to the following formula:

[0095] ,

[0096] ,

[0097] In the above formula, is the embedded feature vector of the key frame, is the embedded feature vector of the reference frame, is a fully connected neural network, Indicates with Convolutional networks of quantized neurons, For the region of interest alignment operation, is the fusion feature map of the key frame , is the fusion feature map of the reference frame , is the anchor box of the key frame, is the anchor frame of the reference frame; in this embodiment, the corresponding feature embedding is obtained by dividing the positive and negative samples and projecting them into the feature map, and then a contrastive learning loss function is constructed to learn high-quality similarity for tracking association. Specifically, the function expression of the loss function used in the ReID network training is:

[0098] ,

[0099] In the above formula, is the loss function, 、 、 are respectively the embedded feature vectors of the key frames of the training samples and the positive and negative samples of the training samples in the reference frames; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules As an optional implementation, in this embodiment, the key frames used for training are given , and then from its temporal neighborhood Randomly select a reference frame The neighborhood distance is constrained by the interval k=3. A ReID network based on contrastive learning is constructed using the anchor boxes of the key frame and the reference frame, the fusion feature map of the frame, and the real target bbox and category. The network can effectively associate the same target in the previous and next frames. Region of Interest Align (ROI) is an improvement to Roi pooling. It can more accurately align features within a Roi, avoiding information loss during the Roi pooling process. The network of keyframes and reference frames is shared, outputting the embedded feature vectors of the keyframe and reference frame, respectively. During the inference phase, after object detection is performed using Spike-Yolox to obtain the detection output boxes, the dot product similarity is calculated using each object's "embedded" feature with all tracked objects in the previous frame. After normalization, the tracked object with the highest similarity is considered the associated target, and so on, completing the tracking process. In this embodiment, a Spike-Yolox framework-based spiking neural network detection head is used to provide high-quality candidate tracking targets with high detection accuracy and low power consumption for the subsequent multi-target tracking process. Furthermore, a series of fused feature maps with the same number of channels and a series of candidate anchor boxes are used to obtain corresponding feature embeddings. Comparative learning is then performed on the targets to develop a high-quality association network for tracking association.

[0100] In order to verify the SNN target tracking method of fusion event and RGB image in this embodiment, the Datasets to train and test spiking neural networks (SNNs), Figure 14 The following is a comparison of the tracking results of the spiking neural network (SNN) in this embodiment, where (a) to (c) are the traditional single-image tracking methods, and (d) to (f) are the multi-modal tracking results of the spiking neural network (SNN) in this embodiment. Figure 14 It can be seen that the multi-modal tracking result of the spiking neural network SNN in this embodiment has a better multi-target tracking effect than the traditional single image modality tracking method.

[0101] In this embodiment, The dataset uses a multi-object tracking training set for training and a validation set for evaluation. Two well-known metrics, mIDF1 and mMOTA, are used for evaluation. mIDF1 (Identity F1 Score) is an important indicator for measuring identity consistency in multi-object tracking. It combines true positives (IDTP), false positives (IDFP), and false negatives (IDFN). mMOTA (Multiple Object Tracking Accuracy) is a metric that measures the overall performance of a multi-object tracking algorithm. It accounts for false positives, false negatives, and identity switches. The evaluation metrics for single-image RGB frame tracking are mIDF1 = 51.5% and mMOTA = 35.8%, respectively. For multi-modal event frame and RGB frame tracking using the method of this embodiment, the evaluation metrics are mIDF1 = 48.9% and mMOTA = 37.5%. By comparison, the multimodal event frame and RGB frame tracking evaluation using the method of this embodiment is 1.7% higher than the mMOTA of the single image modality RGB frame tracking, but this may be due to the influence of the event frame features, which will cause the mIDF1 to decrease. However, since mMOTA calculates the matching of all frames and evaluates detection and tracking more comprehensively, while mIDF1 only evaluates tracking, the mMOTA evaluation result has a greater weight. Therefore, in summary, the SNN target tracking method that integrates events and RGB images in this embodiment can maintain robust tracking performance by utilizing the advantages of both event camera and RGB frame camera modes through the feature fusion network.

[0102] To sum up, in order to achieve multimodal fusion of event streams and RGB frame images in this embodiment, this embodiment requires that the event streams and RGB frame images can be synchronously aligned in time and space; in order to reduce power consumption loss, this embodiment needs to use a pulse neural network to extract features from event frame images and RGB frame images; in order to improve the ability of scene understanding and image processing, this embodiment performs feature fusion based on event frame feature maps and RGB frame camera feature maps; in order to improve the tracking ability of neural networks, this embodiment performs target detection based on a pulse neural network detection head under the Spike-yolox framework, and generates a series of anchor frames, and uses the anchor frames to learn the similarity of targets for multi-target tracking, thereby achieving the fusion of fused events and RGB images to ensure high precision and high efficiency of multi-target tracking, and maintain robust detection performance and tracking performance in static and high-motion scenes.

[0103] In addition, this embodiment also provides an SNN target tracking system for fusing events and RGB images, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the SNN target tracking method for fusing events and RGB images.

[0104] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the SNN target tracking method of the fusion event and RGB image through a processor.

[0105] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the SNN target tracking method of fusion events and RGB images through a processor.

[0106] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0107] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A SNN target tracking method that integrates events and RGB images, characterized in that: The steps include: S1, converts the event stream from the event camera into event frame images through a time stacking-based representation method, and aligns them with the RGB frame images from the RGB camera in time and space respectively; S2, the aligned event frame image and RGB frame images The target detection is performed using a pulse neural network SNN, which includes a feature extraction module, a feature fusion module and a detection head. The target detection of the pulse neural network includes: respectively merging the aligned event frame images through the feature extraction module and RGB frame images Perform feature extraction and use the feature fusion module to extract the event frame feature map and RGB frame feature map Fusion obtains fusion feature map ; Through the detection head to fuse feature maps Perform target detection to generate anchor boxes; S3, using anchor boxes and fusion feature maps To learn the similarity of targets for multi-target tracking; The feature extraction module in step S2 includes a backbone network and a multi-scale module. The backbone network is used to transform the aligned event frame images into and RGB frame images Extract feature maps of three different scales respectively; the multi-scale network is a batch normalization module with pulse convolution Feature pyramid network based on path aggregation , the feature pyramid network The processing of the three different scale feature maps output by the backbone network includes: passing the first scale feature map through the first pulse convolution batch normalization module After processing and upsampling, it is connected with the second scale feature map and then passes through the combination module CSP2_1 and the second pulse convolution batch normalization module After processing and upsampling, it is connected with the third scale feature map and then processed by a combination module CSP2_1 to obtain the first scale feature; the first scale feature is processed by the third pulse convolution batch normalization module Processing, and then with the second pulse convolution batch normalization module After the features are processed, they are connected and then processed by a combination module CSP2_1 to obtain the features of the second scale; the features of the second scale are processed by the fourth pulse convolution batch normalization module Processing, and then with the first pulse convolution batch normalization module After the processed features are connected, they are processed by a combination module CSP2_1 to obtain the features of the third scale; the features of the three scales are then processed by a multi-head attention head to obtain the event frame feature map extracted by the feature extraction module Or RGB frame feature map The combined module CSP2_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module Then connected to a 2-cascaded pulse convolution batch normalization module , and the two branches are connected through a connection module and then pass through a pulse convolution batch normalization module Get the corresponding output features; the pulse convolution batch normalization module The function expression is: ， In the above formula, For the spike convolution batch normalization module The output features of For the spike convolution batch normalization module The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The features obtained with The function expression of the convolutional layer inference of quantized neurons is: ， ， In the above formula, is the time step integer-valued impulse tensor of , is the clipping function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step integer-valued impulse tensor of , is the index for splitting the integer-valued impulse tensor into a binary impulse tensor; The function expression of the feature fusion module is: ， In the above formula, represents the fused feature map, Indicates with Convolutional networks of quantized neurons, represents the feature map after concatenation and merging, Represents the event frame feature map, and has: ， In the above formula, Indicates concatenation merging, Represents the RGB frame feature map; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The pulse convolution batch normalization module is constructed in series. The function expression is: ， In the above formula, For the spike convolution batch normalization module The output features of For the spike convolution batch normalization module The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The features obtained with The function expression of the convolutional layer inference of quantized neurons is: ， ， In the above formula, is the time step integer-valued impulse tensor of , is the clipping function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into a binary impulse tensor.

2. The SNN target tracking method for fusion of events and RGB images according to claim 1, characterized in that In step S1, the event stream from the event camera is converted into an event frame image by a time stacking-based representation method, including: for a specified start time and end time Selected event streams between , among which events Contains pixel coordinates , in microseconds The timestamp in units of 1 and the polarity indicating the brightness change , is the number of events; the event stream will be selected Divided into time-based For each bin, the normalized timestamp of the most recent event is calculated according to the following formula: and the normalized timestamp of the most recent event The pixel values ​​assigned to each bin are: ， ， In the above formula, An event frame image in the form of a three-dimensional tensor Center coordinates The pixel value of Represents the three dimensions of pixel x coordinate, pixel y coordinate and timestamp, event frame image The sizes of the three dimensions of pixel x coordinate, pixel y coordinate and timestamp are ,function The function expression is: ， in, For function The independent variable, Indicates taking the maximum value.

3. The SNN target tracking method for fusion of events and RGB images according to claim 1, characterized in that In step S1, the RGB frame images from the RGB camera are aligned in time and space respectively, including: using the trigger timestamp and acquisition frequency of the RGB frame images to make the start accumulation time of the event frame image equal to the trigger time of the corresponding RGB frame image, and taking the time step of the acquisition frequency of the RGB frame as The event camera and the RGB frame camera are aligned in time, where is the starting time, The end time is 0. The intrinsic and extrinsic parameters of the event camera and RGB frame camera are obtained by the specified camera calibration method. The intrinsic and extrinsic parameters are used to dedistort the event frame image and the RGB frame image and perform matrix transformation to synchronize the event camera and the RGB frame camera in space.

4. The SNN target tracking method for fusion of events and RGB images according to claim 1, characterized in that The backbone network includes a focus module, a first pulse convolution batch normalization module, and a sequentially connected , combined module CSP1_1, the second pulse convolution batch normalization module , the first combination module CSP1_3, the third pulse convolution batch normalization module , the second combination module CSP1_3, the fourth pulse convolution batch normalization module , combination module SPP and combination module CSP2_1, where the output of the first combination module CSP1_3 is the first feature map output by the backbone network, the output of the second combination module CSP1_3 is the second feature map output by the backbone network, and the output of CSP2_1 is the third feature map output by the backbone network; the focusing module has multiple parallel slicing modules, a connection module and a pulse convolution batch normalization module The combined module CSP1_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module There is a cascaded residual module Res after the connection, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module Get the corresponding output features; the combined module CSP1_3 includes two parallel branches, one branch is a pulse convolution batch normalization module , the other branch is a pulse convolution batch normalization module There are three cascaded residual modules Res at the end, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module. Get the corresponding output features; the combination module SPP includes a pulse convolution batch normalization module The three largest pooling branches connected later, and the outputs of the three largest pooling branches and the pulse convolution batch normalization module The outputs are connected together through a connection module and then passed through a pulse convolution batch normalization module. Get the corresponding output features; the residual module Res includes two pulse convolution batch normalization modules cascaded in sequence And a residual connection, one input of which comes from the second pulse convolution batch normalization module The output of , and the other input comes from the input of the residual module Res.

5. The SNN target tracking method for fusion of events and RGB images according to claim 1, characterized in that: In step S3, the anchor frame and fusion feature map are used The method of learning target similarity to track multiple targets includes: taking the current frame as the key frame to be processed, randomly selecting a reference frame from the temporal neighborhood of the key frame, and using the pre-trained ReID network to calculate the embedded feature vectors of the key frame and the reference frame according to the following formula: ， ， In the above formula, is the embedded feature vector of the key frame, is the embedded feature vector of the reference frame, is a fully connected neural network, Indicates with Convolutional networks of quantized neurons, For the region of interest alignment operation, is the fusion feature map of the key frame , is the fusion feature map of the reference frame , is the anchor box of the key frame, is the anchor frame of the reference frame; the functional expression of the loss function used in the ReID network training is: ， In the above formula, is the loss function, 、 、 are respectively the embedded feature vectors of the key frames of the training samples and the positive and negative samples of the training samples in the reference frames; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The pulse convolution batch normalization module is constructed in series. The function expression is: ， In the above formula, For the spike convolution batch normalization module The output features of For the spike convolution batch normalization module The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The features obtained with The function expression of the convolutional layer inference of quantized neurons is: ， ， In the above formula, is the time step integer-valued impulse tensor of , is the clipping function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into a binary impulse tensor.

6. A SNN target tracking system that fuses events and RGB images, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the SNN target tracking method for fusing events and RGB images as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instructions are programmed or configured to execute the SNN target tracking method for fusing events and RGB images as described in any one of claims 1 to 5 through a processor.

8. A computer program product comprising a computer program or instructions, characterized in that The computer program or instructions are programmed or configured to execute the SNN target tracking method for fusing events and RGB images as described in any one of claims 1 to 5 through a processor.

Images