SNN target tracking method and system fusing event and RGB image

Abstract

The invention discloses an event and RGB image fused SNN target tracking method and system, and the method comprises the steps: converting an event flow from an event camera into an event frame image through a representation method based on time stacking, and enabling the event frame image to be aligned with an RGB frame image from an RGB camera in time and space; performing target detection on the event frame image # imgabs0 # and the RGB frame image # imgabs1 # after alignment by using a spiking neural network SNN, wherein the spiking neural network SNN comprises a feature extraction module, a feature fusion module and a detection head; and learning the similarity of the targets by using the anchor frame extracted by the spiking neural network SNN and the fusion feature map # imgabs2 # to perform multi-target tracking. The invention aims to realize fusion of the fusion event and the RGB image so as to ensure high precision and high efficiency of multi-target tracking, and keep stable detection performance and tracking performance in static and high-motion scenes.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology in the field of computer vision, and in particular to an SNN target tracking method and system for fusing events and RGB images. Background Art

[0002] Multi-target tracking is a computer vision technology that aims to detect and track multiple target objects simultaneously in a series of continuous video frames. Its core task is to identify different targets in the scene and continuously track the position and motion trajectory of each target throughout the video sequence. At present, the mainstream multi-target tracking methods are mainly based on artificial neural networks (ANN). Although ANN performs well in many computer vision tasks, ANN is quite different from the biological nervous system. It is difficult to simulate the complex functions of the brain. When processing information, all neurons are activated at the same time, resulting in large computational and energy consumption. It may be limited by computational delays in real-time applications. The spiking neural network (SNN) is closer to the working mode of biological neurons, has better bio-inspiration, and is only activated when a pulse is received, which is more energy-efficient. At the same time, SNN can effectively process sparse and irregular data through an event-driven mechanism, so it is naturally suitable for processing dynamic and time-related data. It can also achieve low latency when deployed on hardware platforms such as brain-like chips.

[0003] Compared with traditional RGB cameras, the excellent temporal resolution of event cameras enables them to capture rich information between frames and avoid the influence of motion blur, making them a prime candidate for target tracking. They also have the advantages of low energy consumption, high dynamic range, and can work properly even under overexposure conditions. However, in real-world environments, using a single camera has its own advantages and disadvantages. Event cameras provide precise time and motion information, while RGB cameras provide rich spatial and color information. Through fusion, better perception effects can be obtained in dynamic scenes, improving the accuracy and robustness of visual tasks such as detection and tracking. At the same time, the current mainstream multi-target tracking methods are mainly based on artificial neural networks (ANN). Although ANN performs well in many computer vision tasks, it is quite different from the biological nervous system and is difficult to simulate the complex functions of the brain. When processing information, all neurons are activated at the same time, resulting in large computational and energy consumption, and may be limited by computational delays in real-time applications. The spiking neural network (SNN) is closer to the working mode of biological neurons, has better biological inspiration, and is only activated when a pulse is received, which is more energy efficient. At the same time, SNN can effectively process sparse and irregular data through an event-driven mechanism, so it is naturally suitable for processing dynamic and time-related data, and can also achieve low latency when deployed on hardware platforms such as brain-like chips. Therefore, combining event data and SNN can ensure high accuracy and efficiency of multi-target tracking. However, how to achieve the fusion of the two to ensure high accuracy and efficiency of multi-target tracking is still a key technical problem that needs to be solved. Summary of the invention

[0004] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a SNN target tracking method and system for fusing events and RGB images are provided. The present invention aims to achieve the fusion of fused events and RGB images to ensure high precision and high efficiency of multi-target tracking, and maintain robust detection performance and tracking performance in static and high-motion scenes.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A SNN target tracking method for fusing events and RGB images comprises the following steps: S1, the event stream from the event camera is converted into an event frame image through a time stacking-based representation method, and is aligned with the RGB frame image from the RGB camera in time and space respectively; S2, the aligned event frame image and RGB frame images The target detection is performed using a pulse neural network SNN, wherein the pulse neural network SNN includes a feature extraction module, a feature fusion module and a detection head. The target detection of the pulse neural network includes: respectively merging the aligned event frame images through the feature extraction network and RGB frame images Perform feature extraction and use the feature fusion module to extract the event frame feature map And RGB frame feature map Fusion obtains fusion feature map ; Through the detection head to fuse feature maps Perform target detection to generate anchor boxes; S3, using anchor frame and fusion feature map To learn the similarity of targets for multi-target tracking.

[0006] Optionally, in step S1, converting the event stream from the event camera into an event frame image by a time stacking-based representation method includes: for a specified start time and end time Selected event streams between , among which Events Contains pixel coordinates , in microseconds A timestamp in units and the polarity of the brightness change , is the number of events; the event flow will be selected Divided into Bins, calculate the normalized timestamp of the most recent event according to the following formula and normalized timestamp of the most recent event The pixel values ​​assigned to each bin are: , , In the above formula, The event frame image is a three-dimensional tensor Central coordinates The pixel value of Represents the three dimensions of pixel x coordinate, pixel y coordinate and timestamp, event frame image The sizes of the three dimensions of pixel x coordinate, pixel y coordinate and timestamp are ,function The function expression is: , in, For function The independent variable, Indicates taking the maximum value.

[0007] Optionally, the aligning of the RGB frame images from the RGB camera in time and space in step S1 includes: by using the trigger timestamp and acquisition frequency of the RGB frame images, making the start accumulation time of the event frame image equal to the trigger time of the corresponding RGB frame image, and taking the time step of the acquisition frequency of the RGB frame as The event camera and the RGB frame camera are synchronized in time, where is the starting time, is the end time; the internal and external parameters of the event camera and the RGB frame camera are obtained by the specified camera calibration method, and the event frame image and the RGB frame image are dedistorted and matrix transformed using the internal and external parameters, so that the event camera and the RGB frame camera are synchronized in space.

[0008] Optionally, the feature extraction network in step S2 includes a backbone network and a multi-scale module, wherein the backbone network is used to transform the aligned event frame images and RGB frame images Three feature maps of different scales are extracted respectively; the multi-scale network is a batch normalization module with pulse convolution Feature pyramid network based on path aggregation , the feature pyramid network The processing of the three different scale feature maps output by the backbone network includes: passing the first scale feature map through the first pulse convolution batch normalization module After processing and upsampling, it is connected with the second scale feature map and then passes through the combination module CSP2_1 and the second pulse convolution batch normalization module. After processing and upsampling, it is connected with the third scale feature map and then processed by a combination module CSP2_1 to obtain the first scale feature; the first scale feature is processed by the third pulse convolution batch normalization module Processing, and then with the second pulse convolution batch normalization module After the features are processed, they are connected and then processed by a combination module CSP2_1 to obtain the features of the second scale; the features of the second scale are processed by the fourth pulse convolution batch normalization module Processing, and then with the first pulse convolution batch normalization module After the processed features are connected, they are processed by a combination module CSP2_1 to obtain the features of the third scale; the features of the three scales are then processed by a multi-head attention head to obtain the event frame feature map extracted by the feature extraction network Or RGB frame feature map The combined module CSP2_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module Then connected with 2 cascaded spike convolution batch normalization modules , and the two branches are connected through a connection module and then through a pulse convolution batch normalization module Get the corresponding output features; the pulse convolution batch normalization module The function expression is: , In the above formula, Batch normalization module for spike convolution The output features of Batch Normalization Module for Spike Convolution The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The obtained features, with The function expression of the convolutional layer inference of quantized neurons is: , , In the above formula, is the time step An integer-valued impulse tensor of is the limiting function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step An integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into binary impulse tensors.

[0009] Optionally, the backbone network includes a focusing module, a first pulse convolution batch normalization module connected in sequence , combined module CSP1_1, the second pulse convolution batch normalization module , the first combination module CSP1_3, the third pulse convolution batch normalization module , the second combination module CSP1_3, the fourth pulse convolution batch normalization module , a combination module SPP and a combination module CSP2_1, wherein the output of the first combination module CSP1_3 is the first feature map output by the backbone network, the output of the second combination module CSP1_3 is the second feature map output by the backbone network, and the output of CSP2_1 is the third feature map output by the backbone network; the focusing module has multiple parallel slicing modules, a connection module and a pulse convolution batch normalization module The combined module CSP1_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module There is a cascaded residual module Res after the connection, and the two branches are connected by a connection module and then pass through a pulse convolution batch normalization module The corresponding output features are obtained; the combined module CSP1_3 includes two parallel branches, one branch is a pulse convolution batch normalization module , the other branch is a pulse convolution batch normalization module After that, there are three cascaded residual modules Res, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module. Get the corresponding output features; the combined module SPP includes a pulse convolution batch normalization module The three largest pooling branches connected later, and the outputs of the three largest pooling branches and the pulse convolution batch normalization module The outputs are connected together through a connection module and then passed through a pulse convolution batch normalization module. The corresponding output features are obtained; the residual module Res includes two pulse convolution batch normalization modules cascaded in sequence And a residual connection, one of the inputs of this residual connection comes from the second spike convolution batch normalization module The output of , and the other input comes from the input of the residual module Res.

[0010] Optionally, the function expression of the fusion module is: , In the above formula, represents the fused feature map, Indicates with Convolutional networks of quantized neurons, represents the feature map after concatenation and merging, Represents the event frame feature map, and has: , In the above formula, Indicates serial merging, Represents an RGB frame feature map; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The pulse convolution batch normalization module is constructed in series. The function expression is: , In the above formula, Batch normalization module for spike convolution The output features of Batch normalization module for spike convolution The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The obtained features, with The function expression of the convolutional layer inference of quantized neurons is: , , In the above formula, is the time step An integer-valued impulse tensor of is the limiting function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step An integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into binary impulse tensors.

[0011] Optionally, in step S3, the anchor frame and the fusion feature map are used The method of learning the similarity of targets for multi-target tracking includes: taking the current frame as the key frame to be processed, randomly selecting a reference frame from the time neighborhood of the key frame, and using the pre-trained ReID network to calculate the embedded feature vectors of the key frame and the reference frame according to the following formula: , , In the above formula, is the embedded feature vector of the key frame, is the embedded feature vector of the reference frame, is a fully connected neural network, Indicates with Convolutional networks of quantized neurons, For the region of interest alignment operation, is the fusion feature map of the key frame , is the fusion feature map of the reference frame , is the anchor box of the key frame, is the anchor frame of the reference frame; the function expression of the loss function used in the ReID network training is: , In the above formula, is the loss function, , , are respectively the embedded feature vectors of the key frames of the training sample and the positive and negative samples of the training sample in the reference frame; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The pulse convolution batch normalization module is constructed in series. The function expression is: , In the above formula, Batch normalization module for spike convolution The output features of Batch normalization module for spike convolution The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The obtained features, with The function expression of the convolutional layer inference of quantized neurons is: , , In the above formula, is the time step An integer-valued impulse tensor of is the limiting function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step An integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into binary impulse tensors.

[0012] In addition, the present invention also provides an SNN target tracking system for fusing events and RGB images, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the SNN target tracking method for fusing events and RGB images.

[0013] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the SNN target tracking method of fusion events and RGB images through a processor.

[0014] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the SNN target tracking method of fusion events and RGB images through a processor.

[0015] Compared with the prior art, the present invention mainly has the following advantages: 1. The present invention uses an event stream representation method based on time stacking (SBT) to generate high-frame rate event images that are less prone to motion blur while retaining more useful information.

[0016] 2. The present invention effectively fuses event and RGB data through a multimodal fusion method, avoiding insufficient extraction of temporal and spatial information of samples by a single modality. This fusion method is used to retain the clear image of the RGB camera in space, while obtaining high-precision motion information of the event camera in time, complementing each other's deficiencies in tracking, detection and other tasks, and maintaining robust detection and tracking performance in static and high-motion scenes, improving detection accuracy and tracking continuity.

[0017] 3. The present invention constructs a network model of a pulse neural network combined with quantized neurons, which reduces power consumption while maintaining performance comparable to that of ANN. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0019] Figure 2 Schematic diagram of the structure and working principle of the pulse neural network SNN in an embodiment of the present invention.

[0020] Figure 3 Schematic diagram of the network structure of the feature extraction network and its multi-scale network in an embodiment of the present invention.

[0021] Figure 4 Schematic diagram of the network structure of the combination module CSP2_1 in an embodiment of the present invention.

[0022] Figure 5 It is a pulse convolution batch normalization module in the embodiment of the present invention. Schematic diagram of the network structure.

[0023] Figure 6 Schematic diagram of the network structure of the backbone network in an embodiment of the present invention.

[0024] Figure 7 Schematic diagram of the network structure of the focusing module in an embodiment of the present invention.

[0025] Figure 8 Schematic diagram of a general network structure of the combination module CSP1_1 and the combination module CSP1_3 in an embodiment of the present invention.

[0026] Fig. 9 Schematic diagram of the network structure of the combined module SPP in an embodiment of the present invention.

[0027] Fig.10 Schematic diagram of the network structure of the residual module Res in an embodiment of the present invention.

[0028] Fig.11 Schematic diagram of the network structure of the fusion module in an embodiment of the present invention.

[0029] Fig.12 The figure is a schematic diagram of the principle of multi-target tracking in an embodiment of the present invention.

[0030] Fig.13 Schematic diagram of the working principle of the ReID network in an embodiment of the present invention.

[0031] Fig.14 1 is a comparison of the tracking results of the spiking neural network SNN in the embodiment of the present invention, where (a) to (c) are the traditional single image modality tracking methods, and (d) to (f) are the multi-modal tracking results of the spiking neural network SNN. DETAILED DESCRIPTION

[0032] The specific implementation modes of the present invention will be further described in detail below with reference to the accompanying drawings.

[0033] like Figure 1 As shown, the SNN target tracking method for fusing events and RGB images in this embodiment includes the following steps: S1, the event stream from the event camera is converted into an event frame image through a time stacking-based representation method, and is aligned with the RGB frame image from the RGB camera in time and space respectively; S2, the aligned event frame image and RGB frame images Use pulse neural network SNN for target detection, such as Figure 2 As shown, the pulse neural network SNN in this embodiment includes a feature extraction module, a feature fusion module and a detection head. The pulse neural network performs target detection by: respectively merging the aligned event frame images through the feature extraction network and RGB frame images Perform feature extraction and use the feature fusion module to extract the event frame feature map And RGB frame feature map Fusion obtains fusion feature map ; Through the detection head to fuse feature maps Perform target detection to generate anchor boxes; S3, using anchor frame and fusion feature map To learn the similarity of targets for multi-target tracking.

[0034] In this embodiment, since an asynchronous event stream is used as the input of the neural network, this event representation method is used to convert the event stream into a matrix similar to a tensor while retaining as much useful information as possible. Specifically, in step S1 of this embodiment, the event stream from the event camera is converted into an event frame image by using a representation method based on time stacking, including: for the specified start time and end time Selected event streams between , among which Events Contains pixel coordinates , in microseconds A timestamp in units and the polarity of the brightness change , is the number of events; the event flow will be selected Divided into Bins, calculate the normalized timestamp of the most recent event according to the following formula and normalized timestamp of the most recent event The pixel values ​​assigned to each bin are: , , In the above formula, The event frame image is a three-dimensional tensor Central coordinates The pixel value of Represents the three dimensions of pixel x coordinate, pixel y coordinate and timestamp, event frame image The sizes of the three dimensions of pixel x coordinate, pixel y coordinate and timestamp are ,function The function expression is: , in, For function The independent variable, Indicates taking the maximum value. In some embodiments, the event frame image is aligned with the RGB frame camera image in time and space respectively, including: using a trigger signal to perform time alignment, and using the internal and external parameters of the event camera and the RGB frame camera to perform spatial alignment. In this embodiment, the event camera and the RGB frame camera are synchronized in time by using the trigger timestamp and acquisition frequency of the RGB frame image; the internal and external parameters of the event camera and the RGB frame camera are obtained by some calibration methods to dedistort the image and perform matrix transformation, so that the event camera and the RGB frame camera are synchronized in space, and then the label on the RGB frame image is also the event image label. In step S1 of this embodiment, the alignment in time and space with the RGB frame image from the RGB camera includes: by using the trigger timestamp and acquisition frequency of the RGB frame image, the start accumulation time of the event frame image is equal to the trigger time of the corresponding RGB frame image, and the time step of the acquisition frequency of the RGB frame is (Specifically in this embodiment ) makes the event camera and RGB frame camera synchronously aligned in time, where is the starting time, The end time is obtained by the specified camera calibration method, and the internal and external parameters of the event camera and the RGB frame camera are used to dedistort and transform the event frame image and the RGB frame image, so that the event camera and the RGB frame camera are synchronized in space. Specifically, in this embodiment, the event frame image and the RGB frame image are then aligned in time and space respectively, and the trigger timestamp and acquisition frequency of the RGB frame image are used. , so that the start accumulation time of the event frame image is equal to the trigger time of the corresponding RGB frame image, and the acquisition frequency of the RGB frame To get the time step , so that the event camera and the RGB frame camera are synchronized in time, and the internal and external parameters of the event camera and the RGB frame camera are obtained through the checkerboard and Zhang calibration methods to dedistort the image and perform matrix transformation, so that the event camera and the RGB frame camera are synchronized in space, that is, the label on the RGB frame image is also the label of the event frame image.

[0035] The feature extraction of event frame images is similar to that of RGB frame images. Figure 3 As shown, the feature extraction network in step S2 of this embodiment includes a backbone network and a multi-scale module. The backbone network is used to transform the aligned event frame image and RGB frame images Three feature maps of different scales are extracted respectively; the multi-scale network is a batch normalization module with pulse convolution Feature pyramid network based on path aggregation ,like Figure 3 As shown, the feature pyramid network The processing of the three different scale feature maps output by the backbone network includes: passing the first scale feature map through the first pulse convolution batch normalization module After processing and upsampling, it is concatenated with the second scale feature map and then passes through the combination module CSP2_1 and the second pulse convolution batch normalization module. After processing and upsampling, it is connected with the third scale feature map and then processed by a combination module CSP2_1 to obtain the first scale feature; the first scale feature is processed by the third pulse convolution batch normalization module Processing, and then with the second pulse convolution batch normalization module After processing the features obtained, they are concatenated and then processed by a combination module CSP2_1 to obtain the features of the second scale; the features of the second scale are processed by the fourth pulse convolution batch normalization module Processing, and then with the first pulse convolution batch normalization module After the processed features are connected, they are processed by a combination module CSP2_1 to obtain the features of the third scale; the features of the three scales are then processed by a multi-head attention head to obtain the event frame feature map extracted by the feature extraction network Or RGB frame feature map .

[0036] like Figure 4 As shown, the combined module CSP2_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module Then connected with 2 cascaded spike convolution batch normalization modules , and the two branches are connected through a connection module and then through a pulse convolution batch normalization module Get the corresponding output features; such as Figure 5 As shown, the pulse convolution batch normalization module The function expression is: , In the above formula, Batch Normalization Module for Spike Convolution The output features of Batch Normalization Module for Spike Convolution The input features of ,in represents the dimension, T represents the time step of the spiking neural network, C is the number of channels of the tensor, W and H are the width and height of the tensor, is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The obtained features, with The function expression of the convolutional layer inference of quantized neurons is: , , In the above formula, is the time step An integer-valued impulse tensor of is the limiting function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step An integer-valued impulse tensor of , To split the integer-valued impulse tensor into indices for the binary impulse tensor, The time step is extended to , and the integer value Convert to pulse train For example, if ,but . Pulse sequence Contains only , so all multi-accumulation operations can be converted to sparse accumulation operations, which can ensure the pulse driving during inference. As neurons of the spiking neural network, using integer values ​​during training helps reduce quantization errors and makes the training more stable. At the same time, converting them into binary pulses during inference retains the low energy consumption of the spiking neural network.

[0037] like Figure 6 As shown, the backbone network consists of a sequentially connected focusing module, a first pulse convolution batch normalization module , combined module CSP1_1, the second pulse convolution batch normalization module , the first combination module CSP1_3, the third pulse convolution batch normalization module , the second combination module CSP1_3, the fourth pulse convolution batch normalization module , combination module SPP and combination module CSP2_1, where the output of the first combination module CSP1_3 is the first feature map output by the backbone network, the output of the second combination module CSP1_3 is the second feature map output by the backbone network, and the output of CSP2_1 is the third feature map output by the backbone network.

[0038] like Figure 7 As shown, the focusing module has multiple parallel slicing modules, a connection module and a pulse convolution batch normalization module. Connected in sequence.

[0039] Figure 8Schematic diagram of the general network structure of the combined module CSP1_1 and the combined module CSP1_3 in the embodiment of the present invention, the combined modules CSP1_1 to CSP1_3 are represented as combined modules CSP1_a, the difference is that the number a of residual modules Res is different, that is, the number a of residual modules Res in the combined module CSP1_1 is 1, the number a of residual modules Res in the combined module CSP1_2 is 2, and the number a of residual modules Res in the combined module CSP1_3 is 3. Figure 8 As shown, the combined module CSP1_1 includes two parallel branches, one branch is a pulse convolution batch normalization module , the other branch is a pulse convolution batch normalization module There is a cascaded residual module Res after the connection, and the two branches are connected by a connection module and then pass through a pulse convolution batch normalization module Get the corresponding output features; the combined module CSP1_3 includes two parallel branches, one branch is a pulse convolution batch normalization module , the other branch is a pulse convolution batch normalization module After that, there are three cascaded residual modules Res, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module. Get the corresponding output features; like Fig. 9 As shown, the combined module SPP includes a pulse convolution batch normalization module The three largest pooling branches connected later, and the outputs of the three largest pooling branches and the pulse convolution batch normalization module The outputs of are connected together through a connection module and then through a pulse convolution batch normalization module. Get the corresponding output features; among them, the output of the three maximum pooling branches and the pulse convolution batch normalization module The output is equivalent to forming a maximum pooling method of 1×1, 5×5, 9×9 and 13×13 for multi-scale feature fusion.

[0040] like Fig.10 As shown, the residual module Res includes two pulse convolution batch normalization modules cascaded in sequence. And a residual connection, one input of which comes from the second spike convolution batch normalization module The output of , and the other input comes from the input of the residual module Res.

[0041] After feature extraction, the event frame feature map and the RGB frame feature map are obtained, and the event frame feature map and the RGB frame feature map are transmitted to the constructed feature fusion module for feature fusion to obtain a fused feature map. In this embodiment, by effectively utilizing the advantages of pulse neural networks and feature extraction, high-quality feature extraction capabilities are maintained in static and high-motion scenes. In some embodiments, such as Fig.11 As shown, the function expression of the fusion module is: , In the above formula, represents the fused feature map, Indicates with Convolutional networks of quantized neurons, represents the feature map after concatenation and merging, Represents the event frame feature map, and has: , In the above formula, Indicates serial merging, Fig.11 The circled c represents Represents an RGB frame feature map; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The series structure will not be described in detail here.

[0042] In this embodiment, feature fusion aims to integrate RGB image frames and event representations of different frequencies, extract complementary information to generate high-frequency fused features. By effectively utilizing the advantages of both modes, robust tracking performance is maintained in static and high-motion scenes. In some embodiments, the fused feature map is used to The spiking neural network detection head performs target detection under the framework of and generates a series of anchor boxes, including: The spiking neural network detection head under the framework performs gradient feedback training on the target detection loss function, and obtains a series of fused feature maps with the same number of channels and a map that is closer to the real target. and a series of candidate anchor boxes of categories. In this embodiment, by using The framework pulse neural network detection head provides high-quality candidate tracking targets for the subsequent multi-target tracking process with high detection accuracy and low power consumption, and uses a series of fused feature maps with the same number of channels and a series of candidate anchor boxes to learn the similarity of the targets. In some embodiments, Fig.12 and Fig.13 As shown, in step S3, the anchor frame and the fusion feature map are used The method of learning the similarity of targets for multi-target tracking includes: taking the current frame as the key frame to be processed, randomly selecting a reference frame from the time neighborhood of the key frame, and using the pre-trained ReID network to calculate the embedded feature vectors of the key frame and the reference frame according to the following formula: , , In the above formula, is the embedded feature vector of the key frame, is the embedded feature vector of the reference frame, is a fully connected neural network, Indicates with Convolutional networks of quantized neurons, For the region of interest alignment operation, is the fusion feature map of the key frame , is the fusion feature map of the reference frame , is the anchor box of the key frame, is the anchor frame of the reference frame; in this embodiment, the corresponding feature embedding is obtained by dividing the positive and negative samples and projecting them to the feature map, and then the contrastive learning loss function is constructed to learn high-quality similarity for tracking association. Specifically, the function expression of the loss function used in the ReID network training is: , In the above formula, is the loss function, , , are respectively the embedded feature vectors of the key frames of the training sample and the positive and negative samples of the training sample in the reference frame; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules As an optional implementation, in this embodiment, the key frames used for training are given , and then from its temporal neighborhood Randomly select reference frames in The neighborhood distance is constrained by the interval k=3. A ReID network with contrastive learning as the core is constructed by using the anchor boxes of the key frame and the reference frame, the fusion feature map of the frame, and the real target bbox and category. This network can effectively associate the same target in the previous and next frames. Region of interest alignment operation (Region of Interest Align) is an improvement of Roi pooling, which can more accurately align the features in Roi and avoid information loss in the Roi pooling process; the network of key frames and reference frames is shared, and the embedded feature vectors of key frames and reference frames are output respectively. In the inference stage, after using Spike-yolox to detect the target and obtain the detection output frame, the "embed" feature of each target is used to calculate the dot product similarity with all the tracking targets in the previous frame. After normalization, the tracking target with the highest similarity is the associated target of the target, and so on, to complete the tracking process. In this embodiment, the Spike-yolox framework pulse neural network detection head is used to provide high-quality candidate tracking targets for the subsequent multi-target tracking process with high detection accuracy and low power consumption. At the same time, a series of fused feature maps with the same number of channels and a series of candidate anchor frames are used to obtain the corresponding feature embedding, and the target is compared and learned to obtain a high-quality association network for tracking association.

[0043] In order to verify the SNN target tracking method of fusion event and RGB image in this embodiment, the method uses Datasets to train and test spiking neural networks (SNNs), Fig.14 The following is a comparison of the tracking results of the spiking neural network SNN in this embodiment, where (a) to (c) are the traditional single image mode tracking methods, and (d) to (f) are the multi-modal tracking results of the spiking neural network SNN in this embodiment. Fig.14 It can be seen that the multi-modal tracking result of the pulse neural network SNN in this embodiment has a better multi-target tracking effect than the traditional single image modality tracking method.

[0044] In this embodiment, The dataset multi-target tracking training set is used for training and the validation set is used for evaluation. The evaluation indicators are two well-known indicators, mIDF1 and mMOTA. Among them, mIDF1 (Identity F1 Score) is an important indicator for measuring the consistency of identities in multi-target tracking. It combines true positives (IDTP, i.e. the number of correctly matched identities), false positives (IDFP, i.e. the number of incorrectly matched identities) and false negatives (IDFN, i.e. the number of missed identities). mMOTA (Multiple Object Tracking Accuracy) is an indicator for measuring the overall performance of a multi-target tracking algorithm. It takes into account false positives, false negatives and identity switches. The evaluation indicators for tracking using a single image modality RGB frame are mIDF1=51.5% and mMOTA=35.8%, while the evaluation indicators for tracking using a multi-modal event frame and RGB frame using the method of this embodiment are mIDF1=48.9% and mMOTA=37.5%. By comparison, the multimodal event frame and RGB frame tracking evaluation using the method of this embodiment is 1.7% higher than the mMOTA using a single image modality RGB frame tracking, but it may be due to the influence of the event frame features, which will cause the mIDF1 to decrease. However, since mMOTA calculates the matching of all frames, the evaluation of detection and tracking is more comprehensive, while mIDF1 only evaluates tracking, the mMOTA evaluation result has a greater weight. Therefore, on the whole, the SNN target tracking method that fuses events and RGB images in this embodiment can maintain robust tracking performance by utilizing the advantages of both event camera and RGB frame camera modes through a feature fusion network.

[0045] To summarize, in order to achieve multimodal fusion of event streams and RGB frame images in this embodiment, this embodiment requires that the event streams and RGB frame images can be synchronously aligned in time and space; in order to reduce power consumption loss, this embodiment needs to use a pulse neural network to extract features from event frame images and RGB frame images; in order to improve the ability of scene understanding and image processing, this embodiment performs feature fusion based on event frame feature maps and RGB frame camera feature maps; in order to improve the tracking ability of neural networks, this embodiment performs target detection based on a pulse neural network detection head under the Spike-yolox framework, and generates a series of anchor frames, and uses the anchor frames to learn the similarity of targets to track multiple targets, thereby achieving the fusion of fused events and RGB images to ensure high precision and high efficiency of multi-target tracking, and maintain robust detection performance and tracking performance in static and high-motion scenes.

[0046] In addition, this embodiment also provides an SNN target tracking system for fusing events and RGB images, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the SNN target tracking method for fusing events and RGB images.

[0047] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the SNN target tracking method of the fusion event and RGB image through a processor.

[0048] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the SNN target tracking method of fusion events and RGB images through a processor.

[0049] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0050] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A SNN target tracking method integrating events and RGB images, characterized in that: The steps include: S1, the event stream from the event camera is converted into an event frame image through a time stacking-based representation method, and is aligned with the RGB frame image from the RGB camera in time and space respectively; S2, the aligned event frame image and RGB frame images The target detection is performed using a pulse neural network SNN, wherein the pulse neural network SNN includes a feature extraction module, a feature fusion module and a detection head. The target detection of the pulse neural network includes: respectively merging the aligned event frame images through the feature extraction network and RGB frame images Perform feature extraction and use the feature fusion module to extract the event frame feature map And RGB frame feature map Fusion obtains fusion feature map ; Through the detection head to fuse feature maps Perform target detection to generate anchor boxes; S3, using anchor frame and fusion feature map To learn the similarity of targets for multi-target tracking.

2. The SNN target tracking method of fusion event and RGB image according to claim 1, characterized in that: In step S1, the event stream from the event camera is converted into an event frame image by a time stacking-based representation method, including: for a specified start time and end time Selected event streams between , among which Events Contains pixel coordinates , in microseconds The timestamp in units and the polarity of the brightness change , is the number of events; the event flow will be selected Divided into Bins, calculate the normalized timestamp of the most recent event according to the following formula And the normalized timestamp of the most recent event The pixel values ​​assigned to each bin are: ， ， In the above formula, The event frame image is a three-dimensional tensor Central coordinates The pixel value of Represents the three dimensions of pixel x coordinate, pixel y coordinate and timestamp, event frame image The sizes of the three dimensions of pixel x coordinate, pixel y coordinate and timestamp are ,function The function expression is: ， in, For function The independent variable, Indicates taking the maximum value.

3. The SNN target tracking method of fusion event and RGB image according to claim 1, characterized in that: In step S1, the RGB frame images from the RGB camera are aligned in time and space respectively, including: by using the trigger timestamp and acquisition frequency of the RGB frame images, the start accumulation time of the event frame image is equal to the trigger time of the corresponding RGB frame image, and the time step of the acquisition frequency of the RGB frame is The event camera and the RGB frame camera are synchronized in time, where is the starting time, is the end time; the internal and external parameters of the event camera and the RGB frame camera are obtained by the specified camera calibration method, and the event frame image and the RGB frame image are dedistorted and matrix transformed using the internal and external parameters, so that the event camera and the RGB frame camera are synchronized in space.

4. The SNN target tracking method of fusion event and RGB image according to claim 1, characterized in that: The feature extraction network in step S2 includes a backbone network and a multi-scale module. The backbone network is used to transform the aligned event frame images and RGB frame images Three feature maps of different scales are extracted respectively; the multi-scale network is a batch normalization module with pulse convolution Feature pyramid network based on path aggregation , the feature pyramid network The processing of the three different scale feature maps output by the backbone network includes: passing the first scale feature map through the first pulse convolution batch normalization module After processing and upsampling, it is connected with the second scale feature map and then passes through the combination module CSP2_1 and the second pulse convolution batch normalization module. After processing and upsampling, it is connected with the third scale feature map and then processed by a combination module CSP2_1 to obtain the first scale feature; the first scale feature is processed by the third pulse convolution batch normalization module Processing, and then with the second pulse convolution batch normalization module After the features are processed, they are connected and then processed by a combination module CSP2_1 to obtain the features of the second scale; the features of the second scale are processed by the fourth pulse convolution batch normalization module Processing, and then with the first pulse convolution batch normalization module After the processed features are connected, they are processed by a combination module CSP2_1 to obtain the features of the third scale; the features of the three scales are then processed by a multi-head attention head to obtain the event frame feature map extracted by the feature extraction network Or RGB frame feature map The combined module CSP2_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module Then connected with 2 cascaded spike convolution batch normalization modules , and the two branches are connected through a connection module and then through a pulse convolution batch normalization module Get the corresponding output features; the pulse convolution batch normalization module The function expression is: ， In the above formula, Batch Normalization Module for Spike Convolution The output features of Batch normalization module for spike convolution The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The obtained features, with The function expression of the convolutional layer inference of quantized neurons is: ， ， In the above formula, is the time step An integer-valued impulse tensor of is the limiting function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step An integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into binary impulse tensors.

5. The SNN target tracking method of fusion event and RGB image according to claim 4, characterized in that: The backbone network includes a focusing module, a first pulse convolution batch normalization module, and a sequentially connected , combined module CSP1_1, the second pulse convolution batch normalization module , the first combination module CSP1_3, the third pulse convolution batch normalization module , the second combination module CSP1_3, the fourth pulse convolution batch normalization module , a combination module SPP and a combination module CSP2_1, wherein the output of the first combination module CSP1_3 is the first feature map output by the backbone network, the output of the second combination module CSP1_3 is the second feature map output by the backbone network, and the output of CSP2_1 is the third feature map output by the backbone network; the focusing module has multiple parallel slicing modules, a connection module and a pulse convolution batch normalization module The combined module CSP1_1 includes two parallel branches, one of which is a pulse convolution batch normalization module. , the other branch is a pulse convolution batch normalization module There is a cascaded residual module Res after the connection, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module The corresponding output features are obtained; the combined module CSP1_3 includes two parallel branches, one branch is a pulse convolution batch normalization module , the other branch is a pulse convolution batch normalization module After that, there are three cascaded residual modules Res, and the two branches are connected through a connection module and then through a pulse convolution batch normalization module. Get the corresponding output features; the combined module SPP includes a pulse convolution batch normalization module The three largest pooling branches connected later, and the outputs of the three largest pooling branches and the pulse convolution batch normalization module The outputs of are connected together through a connection module and then through a pulse convolution batch normalization module. The corresponding output features are obtained; the residual module Res includes two pulse convolution batch normalization modules cascaded in sequence And a residual connection, one input of which comes from the second spike convolution batch normalization module The output of , and the other input comes from the input of the residual module Res.

6. The SNN target tracking method of fusion event and RGB image according to claim 1, characterized in that: The function expression of the fusion module is: ， In the above formula, represents the fused feature map, Indicates with Convolutional networks of quantized neurons, represents the feature map after concatenation and merging, Represents the event frame feature map, and has: ， In the above formula, Indicates serial merging, Represents an RGB frame feature map; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The pulse convolution batch normalization module is constructed in series. The function expression is: ， In the above formula, Batch normalization module for spike convolution The output features of Batch normalization module for spike convolution The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The obtained features, with The function expression of the convolutional layer inference of quantized neurons is: ， ， In the above formula, is the time step An integer-valued impulse tensor of is the limiting function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step An integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into binary impulse tensors.

7. The SNN target tracking method of fusion event and RGB image according to claim 6, characterized in that: In step S3, the anchor frame and the fusion feature map are used The method of learning the similarity of targets for multi-target tracking includes: taking the current frame as the key frame to be processed, randomly selecting a reference frame from the time neighborhood of the key frame, and using the pre-trained ReID network to calculate the embedded feature vectors of the key frame and the reference frame according to the following formula: ， ， In the above formula, is the embedded feature vector of the key frame, is the embedded feature vector of the reference frame, is a fully connected neural network, Indicates with Convolutional networks of quantized neurons, For the region of interest alignment operation, is the fusion feature map of the key frame , is the fusion feature map of the reference frame , is the anchor box of the key frame, is the anchor frame of the reference frame; the function expression of the loss function used in the ReID network training is: ， In the above formula, is the loss function, , , are respectively the embedded feature vectors of the key frames of the training sample and the positive and negative samples of the training sample in the reference frame; The convolutional network of quantized neurons consists of four spike convolution batch normalization modules The pulse convolution batch normalization module is constructed in series. The function expression is: ， In the above formula, Batch normalization module for spike convolution The output features of Batch normalization module for spike convolution The input features of is batch normalization, is the convolutional layer, For Convolutional layers of quantized neurons for input features The obtained features, with The function expression of the convolutional layer inference of quantized neurons is: ， ， In the above formula, is the time step An integer-valued impulse tensor of is the limiting function, Indicates that the feature Clip to range Inside, is the rounding function, is the time step The membrane potential tensor, To quantify The maximum integer value emitted by the hyperparameter; For the Layer time step No. A binary pulse tensor, For the Layer time step An integer-valued impulse tensor of , Indices for splitting the integer-valued impulse tensor into binary impulse tensors.

8. A SNN target tracking system integrating events and RGB images, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the SNN target tracking method for fusing events and RGB images as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instructions are programmed or configured to execute the SNN target tracking method for fusing events and RGB images as described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instructions are programmed or configured to execute the SNN target tracking method for fusing events and RGB images as described in any one of claims 1 to 7 through a processor.

Images