Scale-insensitive target tracking method and device based on pulse mechanism control and readable storage medium

Abstract

The invention discloses a scale-insensitive target tracking method and device based on pulse mechanism control and a readable storage medium, and the method comprises the steps: (1) constructing an ANN-SNN heterogeneous single target tracking frame which obtains target features in a feature fusion manner; (2) training a single target tracking framework through a comparative learning method, and constructing association between an issuing threshold value in an SNN neuron and a target scale so as to utilize scale change information of a target in a tracking process; and (3) in a test application stage, according to the established association between the SNN issuing threshold value and the target scale, under the condition that the target scale is not found, self-adaptive tracking is completed by manually adjusting the neuron threshold value, so that the tracking accuracy is improved. According to the method, stable tracking of targets with different scales can be realized by combining an ANN-SNN heterogeneous architecture, comparative learning and a self-adaptive neuron threshold adjustment mechanism.

Description

Technical Field

[0001] The present invention belongs to the field of brain-inspired computing, and in particular, relates to a scale-insensitive target tracking method, device, and readable storage medium based on pulse mechanism control. Background Art

[0002] As the third generation of artificial neural networks, Spiking Neural Networks (SNNs) are widely regarded as one of the key technologies for achieving efficient and low-power intelligent computing due to their biologically inspired computing mechanisms. Compared with the second generation of artificial neural networks (ANNs), SNNs are more similar to the working mode of biological nervous systems. Information is transmitted between neurons through binary pulse signals (0 or 1), constructing an event-driven computing model.

[0003] In SNNs, the activation of neurons is triggered asynchronously without relying on a global clock signal. This asynchronous event-triggering mechanism makes the computing processes of neurons in the network independent of each other, effectively reducing the dependence on computing resources and providing a natural advantage for parallel computing. At the same time, the sparse activation characteristic of SNNs, that is, computing only when events occur, is highly consistent with the energy efficiency of biological nervous systems, so it is regarded as an ideal basis for implementing brain-inspired intelligent systems.

[0004] Combined with emerging neuromorphic hardware, the low-power characteristics of SNNs will show stronger application potential in embedded systems, edge computing devices, and other energy-efficiency-sensitive scenarios, becoming an important direction for promoting the next generation of artificial intelligence systems.

[0005] In the field of computer vision, single object tracking (SOT), as one of the basic tasks, is widely used in scenarios such as autonomous driving, intelligent monitoring, and robotics. Current mainstream SOT methods usually adopt a template matching mechanism, that is, similarity matching is performed between a reference template and a search area, and a feature fusion module is used to integrate multi-scale or multi-channel information to determine the target position. However, such methods often rely on complex post-processing procedures, restricting their performance in terms of real-time and computational efficiency.

[0006] In recent years, to simplify the traditional template matching process, researchers have proposed a new tracking paradigm based on sequence generation, which can achieve more robust tracking performance by explicitly modeling the target motion cues in the time dimension.

[0007] In addition, as application scenarios put forward higher requirements for the tracking accuracy of small targets, related research has gradually focused on the direction of tiny object tracking, and algorithms such as re-detection mechanisms, SmallTrack based on Siamese Network, context-aware and multi-scale feature fusion strategies, and algorithms that combine multi-layer knowledge distillation to improve the representation ability of small targets have been proposed.

[0008] However, existing methods are generally based on traditional ANN architectures, and still face problems such as insufficient adaptability, poor generalization ability, and high energy consumption when facing scenarios such as drastic changes in target scale and drift in image domain distribution. Summary of the Invention

[0009] The present invention provides a scale-insensitive target tracking method, device, and readable storage medium based on pulse mechanism control. By combining the ANN-SNN heterogeneous architecture, contrast learning, and adaptive neuron threshold adjustment mechanism, stable tracking of targets with different scales can be achieved.

[0010] A scale-insensitive target tracking method based on pulse mechanism control includes the following steps:

[0011] (1) Introduce an SNN branch into the ANN target tracking framework to construct an ANN-SNN heterogeneous single-target tracking framework, which obtains target features in a feature fusion manner;

[0012] (2) Train the single-target tracking framework through contrast learning to establish the association between the firing threshold in the SNN neuron and the target scale, so as to utilize the information of scale change during the tracking process of the target;

[0013] (3) In the test application stage, according to the established association between the SNN firing threshold and the target scale, in the case of unseen target scale, complete adaptive tracking by manually adjusting the neuron threshold to improve the tracking accuracy.

[0014] In step (1), constructing the ANN-SNN heterogeneous single-target tracking framework specifically includes:

[0015] The ANN module is used to extract spatial features and perform preliminary analysis of scale changes;

[0016] The SNN module is used to process target information in the time domain and achieve scale self-adaptation adjustment through pulse firing;

[0017] The fusion module combines the feature representation ability of the ANN and the pulse dynamic response of the SNN to perform feature fusion on the features obtained by the ANN and the SNN, obtaining the fused target features and improving the tracking robustness of the target.

[0018] The SNN module includes three pulsed convolutional blocks, and each pulsed convolutional block includes: a convolutional layer and a LIF neuron layer; during the information flow transmission process, the convolutional layer generates pulse signals, which are processed by the LIF neuron layer to generate a temporal pulse sequence.

[0019] During the feature fusion process, the features of the SNN are superimposed on the features of the ANN through an attention mechanism.

[0020] In step (2), the single-object tracking framework is trained by the method of contrastive learning, specifically:

[0021] On the training dataset, by cropping different targets and different scenarios, a contrast dataset covering different target scales is constructed;

[0022] During the training process, after calculating the scale of the target on the current training sample, by comparing with the samples in the constructed contrast dataset, the targets with similar scales are used as positive samples, and the targets with large scale differences are used as negative samples.

[0023] In step (2), the association between the firing threshold in the SNN neuron and the target scale is constructed, specifically:

[0024] In the training stage, using the feature extraction module in contrastive learning, the target scale information in the sample is extracted; through statistical data analysis of this feature and the transmission of a lightweight FC neural network, the firing threshold of the SNN is obtained.

[0025] In step (3), adaptive tracking is completed by manually adjusting the neuron threshold, specifically:

[0026] In the testing stage, given the initial position and scale of the tracking target, based on the established connection between the existing SNN firing threshold and the scale, a suitable firing threshold is derived;

[0027] In the subsequent tracking stage, the scale change in the actual target tracking process is approximated by the inference result of the model, and then the changed scale is used as the basis for derivation to change the SNN firing threshold to further improve the robustness at the target scale.

[0028] A scale-insensitive target tracking device based on pulse mechanism control includes a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above scale-insensitive target tracking method.

[0029] A computer-readable storage medium stores a program thereon, and when the program is executed by a processor, the above-mentioned scale-insensitive object tracking method is implemented.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The present invention constructs an ANN-SNN heterogeneous single-object tracking framework, and adopts a feature fusion method to effectively combine the advantages of the ANN and SNN networks respectively. Based on this tracking framework, combined with contrast learning, a scale and neuron threshold mapping is constructed to improve the scale adaptability of the network model and enhance the tracking stability of the object at different scales.

[0032] 2. In the test application stage of the present invention, according to the established association between the SNN firing threshold and the target scale, in the case of unseen target scales, adaptive neuron threshold adjustment is adopted to enhance the generalization ability and improve the tracking performance in complex scenarios.

[0033] 3. By combining the ANN-SNN heterogeneous architecture, contrast learning, and adaptive neuron threshold adjustment mechanism, the present invention effectively utilizes the efficient feature extraction ability of the ANN and the temporal processing ability of the SNN to achieve stable tracking of objects at different scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of the framework of a scale-insensitive object tracking method based on a pulse mechanism control according to the present invention.

[0035] Figure 2 It is a schematic diagram of an ANN-SNN heterogeneous single-object tracking framework in an embodiment of the present invention.

[0036] Figure 3 It is a schematic diagram of establishing the association between the firing threshold in the SNN neuron and the target scale by adopting a training method of contrast learning in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] The present invention will be further described in detail below with reference to the drawings and embodiments. It should be noted that the following embodiments are intended to facilitate the understanding of the present invention, but do not limit it in any way.

[0038] As Figure 1 shown, a scale-insensitive object tracking method based on a pulse mechanism control includes the following steps:

[0039] S01, construct an ANN-SNN heterogeneous single-object tracking framework, and adopt a feature fusion method to obtain more stable object features.

[0040] The present invention introduces an SNN branch into the traditional ANN target tracking framework and proposes a novel target tracking architecture, as Figure 2 shown.

[0041] The present invention adopts a three-layer convolutional SNN structure to ensure the efficiency and simplicity of calculation; this architecture consists of three spiking convolutional blocks, and each block includes: a convolutional layer; a LIF (Leaky Integrate-and-Fire) neuron layer.

[0042] During the information flow transmission process, the convolutional layer generates spiking signals, which are processed by LIF neurons to generate a temporal spiking sequence.

[0043] The SNN directly processes RGB images, and at the final time step, the membrane potential is used as the output result to capture visual features related to the target scale.

[0044] In the embodiment of the present invention, the SNN model defined in the literature "Wu, Yujie, et al. 'Spatio-temporal backpropagation for training high-performance spiking neural networks.' Frontiers in neuroscience 12(2018):331." is used as the basis.

[0045] The present invention adopts an end-to-end training framework, so it is necessary to solve the training problem brought by the non-differentiable spiking function of the SNN. The present invention adopts the surrogate gradient method for optimization:

[0046]

[0047] Among them, the surrogate gradient function is defined as:

[0048]

[0049] This method effectively solves the gradient propagation problem in SNN training, so as to have a more stable end-to-end optimization ability.

[0050] The feature fusion module proposed by the present invention performs feature enhancement based on the self-attention mechanism and the cross-attention mechanism:

[0051] Self-attention mechanism (Self-Attention): The adaptive features (including target scale information) generated by the SNN branch are processed by self-attention to improve the discrimination ability of the features.

[0052] Cross-Attention mechanism: The enhanced SNN adaptive features are fused with the discriminative features extracted by the ANN.

[0053] S02, through the training method of contrastive learning, constructs the association between the firing threshold in the SNN neuron and the target scale.

[0054] The training process of contrastive learning is as Figure 3 shown. To construct an image sample pool covering a wide range of target scales, the present invention adopts a specially designed cropping algorithm to ensure that the cropped image can completely retain all regions of the target object. The cropping algorithm takes the bounding box of the target as the center to generate a square cropping area to maintain the expected scale of the target in the image. During the cropping process, if the cropping area exceeds the boundary of the original image, the present invention fills it with the average pixel value of the image to ensure the consistency of the image size.

[0055] To further improve the model performance, the present invention introduces a contrastive learning loss term on the basis of the original loss function to construct an enhanced loss function. The complete definition of the loss function is as follows:

[0056] L = L ce + λ1L SIoU + λ2L contra

[0057] where, L ce represents the Cross-Entropy Loss for target classification; L SIoU represents the Scale-Invariant Object Union (SIoU) to enhance the object consistency of the model at different scales; L contra is the Contrastive Learning Loss for improving the model's ability to distinguish targets at different scales.

[0058] Since the present invention groups images based on scale similarity, multiple positive pairs will be generated during the training process. The present invention adopts a contrastive loss strategy, which is more robust to inaccurate samples. Specifically, the contrastive loss can ignore noise samples or incorrect positive samples on the premise of setting a relatively large similarity threshold, thereby improving the stability and generalization ability of the model. The calculation formula of the contrastive loss is as follows:

[0059]

[0060] Among them, q is the query sample; p ∈ P represents multiple positive samples; n ∈ N represents negative samples.

[0061] To train a feature extractor that can adapt to targets of different scales, the present invention adopts the Momentum Contrast (MoCo) framework, and systematically selects positive and negative sample pairs from the cropped image pool for training:

[0062] For small-scale targets, positive sample pairs within a specific range are determined through a linear model;

[0063] For large-scale targets, the upper and lower bounds of their original scales are calculated, and the selection range of positive samples is determined by adding and subtracting a preset constant.

[0064] This method optimizes the scale-aware feature extractor, enabling it to more accurately identify and process targets of different sizes, and improving the tracking performance of the model.

[0065] The present invention first proposes a dynamic pulse threshold adjustment mechanism, enabling the pulse threshold to adaptively respond to changes in the external environment, especially changes in target scale.

[0066] In the SNN structure, the pulse threshold controls the pulse firing rate of the network, and an unreasonable pulse firing rate may have a negative impact on the training process. To optimize this process, the present invention adjusts the pulse threshold of the LIF neuron based on the feature encoding of MoCo training to ensure that the neuron firing pattern can adapt to changes in target scale:

[0067] When the target scale is large, a higher pulse threshold is set to reduce over-excitation;

[0068] When the target scale is small, a lower pulse threshold is set to enhance the response ability to small targets.

[0069] This correlation enhances the robustness of the model in terms of scale adaptability, enabling it to directly learn scale information from images and effectively integrate this information to improve the accuracy of target tracking.

[0070] S03, in the test application stage, adaptive tracking is completed by manually adjusting the neuron threshold.

[0071] The present invention establishes a strong correlation between the pulse firing threshold and the target scale, and uses the contrast learning MoCo framework to optimize the pulse threshold. In the inference stage, the present invention further dynamically adjusts the pulse firing threshold of the SNN neuron based on this correlation to improve the tracking accuracy.

[0072] In practical applications, if it is necessary to manually adjust the firing threshold of a spiking neuron, first, a function describing the correlation between the spiking threshold and the target scale is derived using training data. This function can be used to predict the optimal spiking threshold based on the target scale of the first frame of the test sequence.

[0073] Since the target may change in scale in the video sequence, the present invention further dynamically adjusts the spiking threshold based on the target bounding box output by the model: First frame: According to the initial scale of the target, the initial spiking threshold is predicted using the correlation function; Subsequent frames: The spiking threshold is updated using the target scale predicted by the model to adapt to the size change of the target.

[0074] In this process, the present invention directly applies the predicted threshold to the tracking module without relying on the contrast learning framework for adjustment, thereby improving the calculation efficiency.

[0075] Based on the same inventive principle, an embodiment of the present invention also provides a scale-insensitive target tracking device based on a spiking mechanism control, including a memory and one or more processors. An executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the scale-insensitive target tracking method described in the above embodiment.

[0076] Based on the same inventive principle, an embodiment of the present invention also provides a computer-readable storage medium, characterized in that a program is stored thereon. When the program is executed by a processor, it implements the scale-insensitive target tracking method described in the above embodiment.

[0077] The above embodiments have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, supplements, and equivalent replacements made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A scale-insensitive target tracking method based on pulse mechanism control, characterized in that It includes the following steps: (1) Introduce an SNN branch into the ANN target tracking framework to construct an ANN-SNN heterogeneous single-target tracking framework, which obtains target features through feature fusion; (2) Train the single-target tracking framework by contrastive learning to establish the association between the firing threshold in the SNN neuron and the target scale, so as to utilize the information of the target scale change during tracking; (3) In the test application stage, according to the established association between the SNN firing threshold and the target scale, in the case of unseen target scales, complete adaptive tracking by manually adjusting the neuron threshold to improve the tracking accuracy.

2. The scale-insensitive target tracking method based on pulse mechanism control according to claim 1, wherein In step (1), to construct the ANN-SNN heterogeneous single-target tracking framework, it specifically includes: The ANN module is used to extract spatial features and conduct a preliminary analysis of scale changes; The SNN module is used to process target information in the time domain and achieve scale adaptive adjustment through pulse firing; The fusion module combines the feature representation ability of the ANN and the pulse dynamic response of the SNN to perform feature fusion on the features obtained by the ANN and the SNN, and obtain the fused target features to improve the tracking robustness of the target.

3. The scale-insensitive object tracking method based on pulse mechanism control according to claim 2, wherein The SNN module contains three pulse convolution blocks, and each pulse convolution block includes: a convolutional layer and a LIF neuron layer; during the information flow transmission process, the convolutional layer generates pulse signals, which are processed by the LIF neuron layer to generate a temporal pulse sequence.

4. The scale-insensitive object tracking method based on pulse mechanism control according to claim 2, characterized in that, In the process of feature fusion, the features of the SNN are superimposed on the features of the ANN through the attention mechanism.

5. The scale-insensitive target tracking method based on pulse mechanism control according to claim 1, wherein In step (2), the single-target tracking framework is trained by contrastive learning, specifically: On the training dataset, by cropping different targets and different scenarios, a contrast dataset covering different target scales is constructed; During the training process, after calculating the scale of the target on the current training sample, by comparing with the samples in the constructed contrast dataset, the targets with similar scales are used as positive samples, and the targets with large scale differences are used as negative samples.

6. The scale-insensitive target tracking method based on pulse mechanism control according to claim 1, wherein In step (2), to establish the association between the firing threshold in the SNN neuron and the target scale, specifically: In the training stage, use the feature extraction module in contrastive learning to extract the target scale information in the sample; through statistical data analysis of this feature and the transmission of a lightweight FC neural network, obtain the firing threshold of the SNN.

7. The scale-insensitive target tracking method based on pulse mechanism control according to claim 1, characterized in that, In step (3), to complete adaptive tracking by manually adjusting the neuron threshold, specifically: In the test stage, based on the initial position and scale of the given tracking target, on the established connection between the existing SNN firing threshold and scale, deduce an appropriate firing threshold; In the subsequent tracking stage, approximate the scale change in the actual target tracking process through the inference result of the model, and then use the changed scale as the basis for deduction to change the SNN firing threshold to further improve the robustness at the target scale.

8. A scale-insensitive target tracking device based on pulse mechanism control, characterized in that It includes a memory and one or more processors. Executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the scale-insensitive object tracking method described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, A program is stored thereon. When the program is executed by a processor, it implements the scale-insensitive object tracking method described in any one of claims 1 to 7.

Images