A scale-invariant target tracking method, device and readable storage medium based on pulse mechanism control

Abstract

This invention discloses a scale-insensitive target tracking method, device, and readable storage medium based on a pulse mechanism control, comprising: (1) constructing an ANN-SNN heterogeneous single-target tracking framework, which obtains target features through feature fusion; (2) training the single-target tracking framework using a contrastive learning method to construct the correlation between the firing threshold of the SNN neurons and the target scale, so as to utilize the information of target scale changes during tracking; (3) in the test application stage, based on the established correlation between the SNN firing threshold and the target scale, adaptive tracking is completed by manually adjusting the neuron threshold in the absence of target scale, so as to improve the tracking accuracy. This invention, by combining the ANN-SNN heterogeneous architecture, contrastive learning, and adaptive neuron threshold adjustment mechanism, can achieve stable tracking of targets of different scales.

Description

Technical Field

[0001] This invention belongs to the field of neuromorphic computing, and in particular relates to a scale-insensitive target tracking method, device and readable storage medium based on impulse mechanism control. Background Technology

[0002] Spiking Neural Networks (SNNs), as the third generation of artificial neural networks, are widely regarded as one of the key technologies for achieving efficient, low-power intelligent computing due to their bio-inspired computational mechanisms. Compared to second-generation artificial neural networks (ANNs), SNNs are closer to the workings of biological nervous systems, using binary pulse signals (0 or 1) to transmit information between neurons, thus constructing an event-driven computational model.

[0003] In SNNs, neuron activation is asynchronously triggered, independent of a global clock signal. This asynchronous event-triggered mechanism makes the computational processes of neurons in the network independent of each other, effectively reducing dependence on computational resources and providing a natural advantage for parallel computing. Furthermore, the sparse activation characteristic of SNNs—that is, computation only occurs when an event happens—is highly consistent with the energy efficiency of biological neural systems, thus being considered an ideal foundation for realizing brain-like intelligent systems.

[0004] Combined with emerging neuromorphic computing hardware, the low-power characteristics of SNNs will demonstrate greater application potential in embedded systems, edge computing devices, and other energy-sensitive scenarios, becoming an important direction for promoting the next generation of artificial intelligence systems.

[0005] In the field of computer vision, Single Object Tracking (SOT) is a fundamental task widely used in scenarios such as autonomous driving, intelligent monitoring, and robotics. Current mainstream SOT methods typically employ a template matching mechanism, which matches similarity between a reference template and the search region, and then integrates multi-scale or multi-channel information using a feature fusion module to determine the target's location. However, these methods often rely on complex post-processing procedures, limiting their performance in terms of real-time performance and computational efficiency.

[0006] In recent years, to simplify the traditional template matching process, researchers have proposed a new tracking paradigm based on sequence generation, which achieves more robust tracking performance by explicitly modeling target motion cues in the time dimension.

[0007] Furthermore, as application scenarios place higher demands on the accuracy of small object tracking, related research has gradually focused on the direction of small object tracking, proposing algorithms such as re-detection mechanisms, SmallTrack based on Siamese Networks, context-aware and multi-scale feature fusion strategies, and algorithms that combine multi-layer knowledge distillation to improve the representation ability of small objects.

[0008] However, existing methods are generally based on traditional ANN architectures, which still face problems such as insufficient adaptability, poor generalization ability and high energy consumption when facing scenarios such as drastic changes in target scale and image domain distribution drift. Summary of the Invention

[0009] This invention provides a scale-insensitive target tracking method, device, and readable storage medium based on pulse mechanism control. By combining ANN-SNN heterogeneous architecture, contrastive learning, and adaptive neuron threshold adjustment mechanism, stable tracking of targets at different scales can be achieved.

[0010] A scale-insensitive target tracking method based on pulse mechanism control includes the following steps:

[0011] (1) Introduce an SNN branch into the ANN target tracking framework to construct an ANN-SNN heterogeneous single target tracking framework. This framework uses feature fusion to obtain target features.

[0012] (2) The single target tracking framework is trained by contrastive learning to construct the correlation between the firing threshold in the SNN neurons and the target scale, so as to utilize the information of target scale change during tracking.

[0013] (3) During the testing and application phase, based on the established correlation between the SNN firing threshold and the target scale, adaptive tracking is achieved by manually adjusting the neuron threshold in the absence of the target scale, so as to improve the tracking accuracy.

[0014] In step (1), a heterogeneous ANN-SNN single-target tracking framework is constructed, specifically including:

[0015] The ANN module is used to extract spatial features and perform preliminary analysis of scale variations.

[0016] The SNN module is used to process target information in the time domain and achieve scale adaptive adjustment through pulse firing.

[0017] The fusion module combines the feature representation capabilities of ANN with the impulse dynamic response of SNN to fuse the features obtained from ANN and SNN, resulting in fused target features that improve the tracking robustness of the target.

[0018] The SNN module contains three pulse convolutional blocks, each of which includes a convolutional layer and a LIF neuron layer. During the information flow, the convolutional layer generates pulse signals, which are then processed by the LIF neuron layer to generate a temporal pulse sequence.

[0019] During feature fusion, the features of the SNN are superimposed onto the features of the ANN through an attention mechanism.

[0020] In step (2), the single-target tracking framework is trained using a contrastive learning method, specifically as follows:

[0021] On the training dataset, a comparative dataset covering different target scales was constructed by cropping different targets and different scenes;

[0022] During training, after calculating the scale of the target on the current training sample, the target is compared with the samples in the constructed comparison dataset. Targets with similar scales are taken as positive samples, and targets with large scale differences are taken as negative samples.

[0023] In step (2), the correlation between the firing threshold and the target scale in the SNN neuron is constructed, specifically as follows:

[0024] During the training phase, the feature extraction module in contrastive learning is used to extract target scale information from the samples; through statistical data analysis of this feature and the transmission of the lightweight FC neural network, the firing threshold of the SNN is obtained.

[0025] In step (3), adaptive tracking is achieved by manually adjusting the neuron threshold, specifically as follows:

[0026] During the testing phase, given the initial position and scale of the tracking target, a suitable firing threshold is derived based on the existing relationship between the SNN firing threshold and scale.

[0027] In the subsequent tracking phase, the model's inference results are used to approximate the scale changes in the actual target tracking process. The changed scale is then used as the basis for derivation to change the SNN firing threshold, thereby further improving the robustness at the target scale.

[0028] A scale-insensitive target tracking device based on pulse mechanism control includes a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement the aforementioned scale-insensitive target tracking method.

[0029] A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the above-described scale-insensitive target tracking method.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. This invention constructs a heterogeneous ANN-SNN single-target tracking framework, employing feature fusion to effectively combine the advantages of both ANN and SNN networks. Based on this tracking framework, and by incorporating contrastive learning to construct scale and neuron threshold mappings, the scale adaptability of the network model is improved, enhancing the tracking stability of the target at different scales.

[0032] 2. This invention enhances generalization ability and improves tracking performance in complex scenarios by adopting adaptive neuron threshold adjustment in the test application phase, based on the established correlation between the SNN firing threshold and the target scale, even when the target scale is not seen.

[0033] 3. This invention combines the heterogeneous architecture of ANN-SNN, contrastive learning, and adaptive neuron threshold adjustment mechanism to effectively utilize the efficient feature extraction capability of ANN and the temporal processing capability of SNN, thereby achieving stable tracking of targets at different scales. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the framework of a scale-insensitive target tracking method based on pulse mechanism control according to the present invention.

[0035] Figure 2 This is a schematic diagram of a heterogeneous single-target tracking framework of ANN-SNN in an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram illustrating the relationship between the firing threshold and the target scale in an SNN neuron, constructed using a contrastive learning training method in an embodiment of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0038] like Figure 1 As shown, a scale-insensitive target tracking method based on pulse mechanism control includes the following steps:

[0039] S01. Construct a heterogeneous ANN-SNN single-target tracking framework and use feature fusion to obtain more stable target features.

[0040] This invention introduces an SNN branch into the traditional ANN object tracking framework and proposes a novel object tracking architecture, such as... Figure 2 As shown.

[0041] This invention employs a three-layer convolutional SNN structure to ensure computational efficiency and simplicity. The architecture consists of three spiking convolutional blocks, each block including: a convolutional layer; and a LIF (Leaky Integrate-and-Fire) neuron layer.

[0042] During the information flow transmission process, the convolutional layer generates pulse signals, which are processed by LIF neurons to generate a time-series pulse sequence.

[0043] SNNs directly process RGB images, and at the final time step, the membrane potential is used as the output to capture visual features related to the target scale.

[0044] In this embodiment of the invention, the SNN model defined in the literature “Wu, Yujie, et al. Spatio-temporalbackpropagation for training high-performance spiking neural networks. Frontiers in neuroscience 12(2018):331.” is used as the basis.

[0045] This invention employs an end-to-end training framework, thus requiring the resolution of training problems arising from the non-spurious firing function of SNNs. This invention utilizes a surrogate gradient method for optimization.

[0046]

[0047] The surrogate gradient function is defined as follows:

[0048]

[0049] This method effectively solves the gradient propagation problem in SNN training, thus enabling more stable end-to-end optimization capabilities.

[0050] The feature fusion module proposed in this invention enhances features based on self-attention and cross-attention mechanisms.

[0051] Self-Attention: Adaptive features (containing target scale information) generated by the SNN branch are processed by self-attention to improve the discriminative power of the features.

[0052] Cross-Attention mechanism: The enhanced SNN adaptive features are fused with the discriminative features extracted by ANN.

[0053] S02, through a contrastive learning training method, constructs the correlation between the firing threshold and the target scale in SNN neurons.

[0054] The training process of comparative learning, such as Figure 3 As shown. To construct an image sample pool covering a wide range of target scales, this invention employs a specially designed cropping algorithm to ensure that the cropped image completely preserves the entire region of the target object. This cropping algorithm generates a square cropping region centered on the target's bounding box to maintain the target's expected scale within the image. During the cropping process, if the cropping region exceeds the boundaries of the original image, this invention uses the average pixel value of the image to fill it, ensuring image size consistency.

[0055] To further improve model performance, this invention introduces a contrastive learning loss term into the original loss function, constructing an enhanced loss function. The complete loss function definition is as follows:

[0056] L = L ce +λ1L SIoU +λ2L contra

[0057] Among them, L ce This represents Cross-Entropy Loss, used for object classification; L SIoU The scale-invariant object union (SIoU) loss enhances the consistency of the model's objectives across different scales; L contra Contrastive Learning Loss is used to improve the model's ability to distinguish targets at different scales.

[0058] Because this invention uses scale-based similarity to group images, multiple positive pairs are generated during training. This invention employs a contrastive loss strategy, which is more robust to inaccurate samples. Specifically, this contrastive loss can ignore noisy or erroneous positive samples while setting a large similarity threshold, thereby improving the model's stability and generalization ability. The formula for calculating the contrastive loss is as follows:

[0059]

[0060] Where q is the query sample; p∈P represents multiple positive samples; and n∈N represents negative samples.

[0061] To train a feature extractor capable of adapting to targets of different scales, this invention employs the Momentum Contrast (MoCo) framework, systematically selecting positive and negative sample pairs from the cropped image pool for training:

[0062] For small-scale targets, positive sample pairs within a specific range are determined using a linear model;

[0063] For large-scale targets, calculate the upper and lower bounds of their original scale, and determine the selection range of positive samples by adding or subtracting a preset constant.

[0064] This method optimizes the scale-aware feature extractor, enabling it to more accurately identify and process targets of different sizes, thereby improving the model's tracking performance.

[0065] This invention proposes for the first time a dynamic pulse threshold adjustment mechanism, which enables the pulse threshold to adaptively respond to changes in the external environment, especially changes in the target scale.

[0066] In SNN architectures, the spike threshold controls the network's spike firing rate, and an unreasonable spike firing rate can negatively impact the training process. To optimize this process, this invention adjusts the spike threshold of LIF neurons based on the feature encoding trained with MoCo, ensuring that the neuron firing pattern adapts to changes in the target scale.

[0067] When the target size is large, a higher pulse threshold is set to reduce over-excitation;

[0068] When the target size is small, a lower pulse threshold is set to enhance the response capability to small targets.

[0069] This correlation enhances the model's robustness in scale adaptability, enabling it to learn scale information directly from images and effectively integrate this information, thereby improving the accuracy of target tracking.

[0070] S03, during the testing and application phase, adaptive tracking is achieved by manually adjusting the neuron threshold.

[0071] This invention establishes a strong correlation between the pulse firing threshold and the target scale, and utilizes the contrastive learning MoCo framework to optimize the pulse threshold. During the inference phase, this invention further dynamically adjusts the pulse firing threshold of the SNN neurons based on this correlation to improve tracking accuracy.

[0072] In practical applications, if it is necessary to manually adjust the firing threshold of spiking neurons, the function describing the correlation between the pulse threshold and the target scale is first derived using training data. This function can be used to predict the optimal pulse threshold based on the target scale of the first frame of the test sequence.

[0073] Since the target may change scale in the video sequence, the present invention further dynamically adjusts the pulse threshold based on the target bounding box output by the model: First frame: the initial pulse threshold is predicted using the correlation function according to the initial scale of the target; subsequent frames: the pulse threshold is updated using the target scale predicted by the model to adapt to the size change of the target.

[0074] In this process, the present invention directly applies the prediction threshold to the tracking module without relying on the contrastive learning framework for adjustment, thereby improving computational efficiency.

[0075] Based on the same inventive principle, this invention also provides a scale-insensitive target tracking device based on pulse mechanism control, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the scale-insensitive target tracking method described in the above embodiments.

[0076] Based on the same inventive principle, this invention also provides a computer-readable storage medium, characterized in that it stores a program thereon, which, when executed by a processor, implements the scale-insensitive target tracking method described in the above embodiments.

[0077] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A scale-insensitive target tracking method based on pulse mechanism control, characterized in that, Includes the following steps: (1) Introduce an SNN branch into the ANN target tracking framework to construct an ANN-SNN heterogeneous single target tracking framework. This framework uses feature fusion to obtain target features. (2) The single target tracking framework is trained by contrastive learning to construct the correlation between the firing threshold in the SNN neurons and the target scale, so as to utilize the information of target scale change during tracking. The relationship between the firing threshold and the target scale in SNN neurons is constructed as follows: During the training phase, the target scale information in the samples is extracted using the feature extraction module in contrastive learning; the firing threshold of the SNN is obtained through statistical data analysis of this feature and the transmission of a lightweight FC neural network. (3) During the testing and application phase, based on the established correlation between the SNN firing threshold and the target scale, adaptive tracking is achieved by manually adjusting the neuron threshold when the target scale is not visible, thereby improving tracking accuracy; specifically: During the testing phase, given the initial position and scale of the tracking target, a suitable firing threshold is derived based on the existing relationship between the SNN firing threshold and scale. In the subsequent tracking phase, the model's inference results are used to approximate the scale changes in the actual target tracking process. The changed scale is then used as the basis for derivation to change the SNN firing threshold, thereby further improving the robustness at the target scale.

2. The scale-insensitive target tracking method based on pulse mechanism control according to claim 1, characterized in that, In step (1), a heterogeneous ANN-SNN single-target tracking framework is constructed, specifically including: The ANN module is used to extract spatial features and perform preliminary analysis of scale variations. The SNN module is used to process target information in the time domain and achieve scale adaptive adjustment through pulse firing. The fusion module combines the feature representation capabilities of ANN with the impulse dynamic response of SNN to fuse the features obtained from ANN and SNN, resulting in fused target features to improve the tracking robustness of the target.

3. The scale-insensitive target tracking method based on pulse mechanism control according to claim 2, characterized in that, The SNN module contains three pulse convolutional blocks, each of which includes a convolutional layer and a LIF neuron layer. During the information flow, the convolutional layer generates pulse signals, which are then processed by the LIF neuron layer to generate a temporal pulse sequence.

4. The scale-insensitive target tracking method based on pulse mechanism control according to claim 2, characterized in that, During feature fusion, the features of the SNN are superimposed onto the features of the ANN through an attention mechanism.

5. The scale-insensitive target tracking method based on pulse mechanism control according to claim 1, characterized in that, In step (2), the single-target tracking framework is trained using a contrastive learning method, specifically as follows: On the training dataset, a comparative dataset covering different target scales was constructed by cropping different targets and different scenes; During training, after calculating the scale of the target on the current training sample, the target is compared with the samples in the constructed comparison dataset. Targets with similar scales are taken as positive samples, and targets with large scale differences are taken as negative samples.

6. A scale-insensitive target tracking device based on pulse mechanism control, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the scale-insensitive target tracking method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the scale-insensitive target tracking method according to any one of claims 1-5.

Images