Target tracking methods, devices, electronic equipment, and computer storage media

By combining feature extraction and fusion techniques from event cameras and RGB cameras, the problem of multi-scene adaptability of visual target tracking in complex environments was solved, achieving more efficient target tracking results.

CN120298456BActive Publication Date: 2026-11-17CHANGCHUN UNIV OF SCI & TECH +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510548340.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-04-28
Publication Date
2026-11-17
Estimated Expiration
2045-04-28

AI Technical Summary

Technical Problem

Existing single-vision target tracking technologies cannot meet the target tracking needs of multiple scenarios in complex electromagnetic warfare environments, especially in harsh environments such as fast movement, excessive darkness, and overexposure.

Method used

By combining event cameras and RGB cameras, feature maps of event streams and RGB frame streams are extracted and fused through event modules, RGB modules, cross-modal fusion modules, and classification and regression modules, respectively, for target tracking.

Benefits of technology

It improves the adaptability and accuracy of target tracking, and can effectively handle challenges such as high-speed motion, excessive darkness and overexposure, and interference from similar objects in multiple scenarios, providing real-time tracking support.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120298456B_ABST
    Figure CN120298456B_ABST
Patent Text Reader

Abstract

This invention relates to a target tracking method, apparatus, electronic device, and computer storage medium, belonging to the field of image recognition technology. The target tracking method includes acquiring an event stream and an RGB frame stream of the target to be tracked; using an event module to extract features from the event stream to obtain an event feature map; using an RGB module to extract features from the RGB frame stream to obtain an RGB feature map; using a cross-modal fusion module to fuse the event feature map and the RGB feature map to obtain a fused feature map; and using a classification and regression module to perform classification and regression analysis on the fused feature map to determine the target to be tracked. This invention combines an RGB camera with an event camera to meet the multi-scene target tracking needs of existing technologies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a target tracking method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] With the rapid development of computer vision technology, vision-based target tracking technology has become the most intuitive form of tracking, closely resembling human behavior. Visual target tracking refers to continuously extracting features from subsequent video frames given any target region in the initial frame of a video, and using the extracted features to determine the position and size of the moving target in subsequent frames, thereby completing the task of tracking the target object.

[0003] Although visual target tracking faces interference from factors such as short detection range, cluttered backgrounds, rapid movement, and excessive darkness or overexposure, in the complex electromagnetic warfare environment of modern times, after various advanced electromagnetic technologies have failed and become chaotic, visual target tracking technology will demonstrate its unique autonomy and anti-interference capabilities. Compared with radar tracking, laser-guided tracking, and infrared tracking technologies, it is irreplaceable and can fully utilize its advantages as one of the terminal guidance technologies. The field of visual target tracking mainly uses traditional RGB cameras as sensors. They can effectively measure absolute light intensity and provide rich characterization of the corresponding scene, but they are difficult to work in some harsh scene environments, such as rapid movement and excessive darkness or overexposure.

[0004] This shows that existing single-vision target tracking technologies cannot meet current needs. Summary of the Invention

[0005] In view of this, it is necessary to provide a target tracking method, apparatus, electronic device, and computer storage medium to solve the problem that existing single-vision target tracking technologies cannot meet current needs.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a target tracking method, comprising:

[0007] The event stream and RGB frame stream of the target to be tracked are obtained. The event module is used to extract features from the event stream to obtain an event feature map, and the RGB module is used to extract features from the RGB frame stream to obtain an RGB feature map.

[0008] A cross-modal fusion module is used to fuse the event feature map and the RGB feature map to obtain a fused feature map;

[0009] The classification and regression modules are used to perform classification and regression analysis on the fused feature map to determine the target to be tracked.

[0010] In one possible embodiment of the present invention, the event module includes a first template branch and a first search branch. The event module is used to extract features from the event stream to obtain an event feature map, including:

[0011] The event stream is feature extracted using the first template branch and the first search branch to obtain the event feature map of the first template branch and the event feature map of the first search branch.

[0012] Multiple deep cross-correlation tests are performed on the event feature maps of the first template branch and the first search branch to obtain the event feature maps.

[0013] In one possible embodiment of the present invention, both the first template branch and the first search branch include a pulse self-attention feature extractor and a spatiotemporal feature fusion module. The first template branch and the first search branch are used to extract features from the event stream, resulting in an event feature map of the first template branch and an event feature map of the first search branch. This includes:

[0014] The event stream is extracted using a deep learning network and a spiking neural network in a spiking self-attention feature extractor to obtain spatiotemporal feature information. The spiking neural network dynamically adjusts the spiking threshold through a dynamic spiking threshold adjustment module.

[0015] The spatiotemporal feature fusion module employs multiple dynamic attention mechanisms to fuse spatiotemporal feature information, resulting in a first template branch event feature map and a first search branch event feature map. The multiple dynamic attention mechanisms include a spatiotemporal attention module, a cross-domain aggregator, and a spatial attention module.

[0016] In one possible embodiment of the present invention, the RGB module includes a second search branch and a second template branch. Both the second search branch and the second template branch include a feature extraction submodule and a feature fusion submodule. The RGB module is used to extract features from the RGB frame stream to obtain an RGB feature map, including:

[0017] The feature extraction submodule is used to extract features from the RGB frame stream to obtain RGB frame stream feature information;

[0018] The feature fusion submodule is used to fuse the feature information of the RGB frame stream to obtain the RGB feature map.

[0019] In one possible embodiment of the present invention, the cross-modal fusion module includes two cross-modal modulation and selection modules, each including two cross-attention modules and two adaptive weight modules. The cross-modal fusion module is used to fuse the event feature map and the RGB feature map to obtain a fused feature map, including:

[0020] The event feature map and the RGB feature map are initially fused by the cross-attention module to obtain the preliminary fused feature map.

[0021] An adaptive weighting module is used to determine the weights of the initial fused feature map, and the initial fused feature map is fused based on the weights to obtain the fused feature map.

[0022] In one possible embodiment of the present invention, the classification and regression module includes a classification branch and a regression branch jointly trained with classification and IOU. The classification and regression module is used to perform classification and regression analysis on the fused feature map to determine the target to be tracked, including:

[0023] The feature fusion map is classified using a classification branch to obtain a classification feature map of the target to be tracked;

[0024] The feature fusion map is classified using a regression branch to obtain the regression feature map of the target to be tracked.

[0025] Secondly, the present invention also provides a target tracking device, comprising:

[0026] The feature extraction module is used to acquire the event stream and RGB frame stream of the target to be tracked. The event module is used to extract features from the event stream to obtain an event feature map, and the RGB module is used to extract features from the RGB frame stream to obtain an RGB feature map.

[0027] The feature fusion module is used to fuse the event feature map and the RGB feature map using a cross-modal fusion module to obtain a fused feature map;

[0028] The target recognition module is used to perform classification and regression analysis on the fused feature map using the classification and regression module to determine the target to be tracked.

[0029] In one possible embodiment of the present invention, the cross-modal fusion module includes two cross-modal modulation and selection modules, each including two cross-attention modules and two adaptive weighting modules. The step of fusing the event feature map and the RGB feature map using the cross-modal fusion module to obtain a fused feature map includes:

[0030] The event feature map and the RGB feature map are initially fused using a cross-attention module to obtain a preliminary fused feature map.

[0031] An adaptive weighting module is used to determine the weights of the preliminary fused feature map, and the preliminary fused feature map is fused based on the weights to obtain a fused feature map.

[0032] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein,

[0033] The memory is used to store programs;

[0034] The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the target tracking method described in any of the above embodiments.

[0035] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps of the target tracking method described in any of the above embodiments.

[0036] The beneficial effects of this invention are as follows: The target recognition method provided by this invention uses four modules—an event module, an RGB module, a cross-modal fusion module, and a classification and regression module—to extract and fuse features from the event stream and RGB frame stream of the target to be tracked. The event stream can record image information, and its low latency, high dynamic range, and low power consumption make it more suitable for some challenging scenarios in target tracking. Combining an RGB camera with an event camera meets the multi-scenario target tracking needs of existing technologies. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a target tracking method provided in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram illustrating the framework of a target tracking algorithm provided in an embodiment of the present invention;

[0040] Figure 3 A flowchart illustrating an event feature map extraction method provided in an embodiment of the present invention;

[0041] Figure 4 This is a flowchart illustrating a feature extraction method provided in an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of an SFE structure provided in an embodiment of the present invention;

[0043] Figure 6 A schematic diagram of a TSFF structure provided in an embodiment of the present invention;

[0044] Figure 7This is a flowchart illustrating an RGB feature map extraction method provided in an embodiment of the present invention.

[0045] Figure 8 This is a schematic diagram of the structure of a feature extraction subnetwork provided in an embodiment of the present invention;

[0046] Figure 9 This is a schematic diagram of a feature fusion module structure provided in an embodiment of the present invention;

[0047] Figure 10 A schematic flowchart of a fusion feature map extraction method provided in an embodiment of the present invention;

[0048] Figure 11 This is a framework structure diagram of a CDMS provided in an embodiment of the present invention;

[0049] Figure 12 A flowchart illustrating a method for determining a target to be tracked, provided in an embodiment of the present invention;

[0050] Figure 13 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of the present invention;

[0051] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which constitute a part of the present invention and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0053] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0054] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] A specific embodiment of the present invention, such as Figure 1 As shown, a target tracking method is disclosed, including:

[0056] S101, acquire the event stream and RGB frame stream of the target to be tracked, use the event module to extract features from the event stream to obtain the event feature map, and use the RGB module to extract features from the RGB frame stream to obtain the RGB feature map;

[0057] S102, the cross-modal fusion module is used to fuse the event feature map and the RGB feature map to obtain the fused feature map;

[0058] S103 uses the classification and regression module to perform classification and regression analysis on the fused feature map to determine the target to be tracked.

[0059] In this embodiment of the invention, a target tracking algorithm is provided to track the target. The algorithm framework mainly consists of four parts, such as... Figure 2 As shown, these are the event module, RGB module, cross-modal fusion module, and classification / regression module, respectively. The event stream of the target to be tracked refers to the image information acquired by an event camera (such as a dynamic vision sensor). The event camera can record image information by outputting an asynchronous event stream. The RGB frame stream refers to the image stream information acquired by a traditional RGB camera. By inputting the event stream and RGB frame stream into the event module and RGB module respectively, feature extraction can be performed on the event stream and RGB frame stream to obtain event feature maps and RGB feature maps. The specific feature extraction process will be described in detail later in this invention.

[0060] In this embodiment of the invention, the cross-modal fusion module is a key module in the target tracking algorithm that fuses the event domain and the RGB domain. It fuses the event feature maps and RGB feature maps obtained from the event module and the RGB module to obtain the final fused feature map, which is then input into the classification and regression module to achieve feature fusion. The specific structure of the cross-modal fusion module will be described in detail later in this invention.

[0061] In this embodiment of the invention, the classification and regression module is used to predict the location of the target bounding box, thereby performing target tracking. The tracking task is divided into two sub-tasks: a classification task to predict each location category and a regression task to calculate the target bounding box. The specific structure of the classification and regression module will be described in detail later in this invention.

[0062] The target recognition method provided by this invention uses four modules—an event module, an RGB module, a cross-modal fusion module, and a classification and regression module—to extract and fuse features from the event stream and RGB frame stream of the target to be tracked. The event stream can record image information, and its low latency, high dynamic range, and low power consumption make it more suitable for some challenging scenarios in target tracking. By combining an RGB camera with an event camera, it meets the multi-scenario target tracking needs of existing technologies.

[0063] In some possible embodiments of the present invention, such as Figure 3 As shown, the event module includes a first template branch and a first search branch. The event module is used to extract features from the event stream to obtain an event feature map, including:

[0064] S301, the first template branch and the first search branch are used to extract features from the event stream to obtain the first template branch event feature map and the first search branch event feature map;

[0065] S302, perform multiple deep cross-correlation operations on the event feature map of the first template branch and the event feature map of the first search branch to obtain the event feature map.

[0066] In embodiments of the present invention, such as Figure 2 As shown, the event module includes a first template branch and a first search branch. The first template branch and the first search branch are used to extract features from the event stream to obtain the first template branch event feature map and the first search branch event feature map. Multiple deep cross-correlation operations are performed on the first template branch event feature map and the first search branch event feature map to obtain the event feature map.

[0067] Specifically, such as Figure 4 As shown, both the first template branch and the first search branch include a pulse self-attention feature extractor and a spatiotemporal feature fusion module. The first template branch and the first search branch are used to extract features from the event stream, resulting in an event feature map for the first template branch and an event feature map for the first search branch, including:

[0068] S401 uses a deep learning network and a spiking neural network in the spiking self-attention feature extractor to extract features from the event stream and obtain spatiotemporal feature information. The spiking neural network achieves dynamic adjustment of the spiking threshold through a dynamic spiking threshold adjustment module.

[0069] S402, multiple dynamic attention mechanisms of the spatiotemporal feature fusion module are used to fuse spatiotemporal feature information to obtain the first template branch event feature map and the first search branch event feature map. The multiple dynamic attention mechanisms include a spatiotemporal attention module, a cross-domain aggregator and a spatial attention module.

[0070] In embodiments of the present invention, such as Figure 5 The diagram shows the specific structure of the Pulse Self-Attention Feature Extractor (SFE). The SFE module has two branches: a Transformer (deep learning) branch and an SNN (spiking neural network) branch. Between these branches, the SFE provides a dynamic pulse threshold adjustment module to dynamically adjust the SNN pulse threshold. The SFE module relies on a 3-layer LIF-based...

[0071] A SNN (Leakage Integral-Trigger Neuron Model) and a simplified Swin-transformer (a deep learning model) are used to capture information cues from the temporal and spatial domains. Specifically, given a sequence of events over a period of time, this sequence of events is discretized into an n-bin voxel grid to represent the temporal dimension. For each bin, two frames E are generated by recording the spatial locations of all negative and positive events occurring during this period. neg and E pos The Transformer branch is used to extract global spatial features from events. Given n pairs of positive and negative cumulative event frames, the global spatial features FG can be extracted using the following formula:

[0072] F G =ψ(C3(F ST ))

[0073] F ST =T(C1(Ec))

[0074] Ec = [E 1 pos E 2 pos ,…,E n pos E 1 neg E 2 neg ,…,E n neg ]

[0075] Where [] indicates connection by channel, C k Let represent a convolutional layer with a k×k convolutional kernel, T represent a simplified Swin-transformer, and ψ is an operator consisting of batch normalization and ReLU activation function.

[0076] The SNN network is based on a Spiking CNN architecture and consists of three conv-SNN blocks. Within each conv-SNN block, a convolutional layer follows a LIF-based SNN layer, where the convolutional layer converts impulses into membrane potentials, which serve as input to the SNN layer. Using n pairs of positive and negative event frames, the SNN branches process them sequentially in chronological order. For each new input pair, the membrane potentials of all SNN neurons remain in their previous state rather than being reset to their initial state, allowing for the capture of temporal cues. After processing all n pairs, the membrane potentials are reset. Mathematically, for a conv-SNN-based network, at timestamp t, the membrane potential V of the neuron in layer l is... t,l It can be represented as:

[0077] V t,l =Ht-1,l +C(Z t,l-1 )

[0078] Z t,l =f(V t,l -V th )

[0079] H t,l =αV t,l (1-Z t,l )

[0080] Z t,0 =E t =∨(E) t pos E t neg )

[0081] Where f() is the Heaviside step function; V th α is the membrane potential threshold; α is the leakage factor of LIF neurons; ∨ is the element-wise OR operator. At the last time stamp n, the average value of the last layer membrane potential over all time stamps (1 to n) is calculated and used to estimate the temporal characteristics. This process is defined as:

[0082]

[0083] Dynamic impulse thresholding (LIF) is a spontaneous regulatory mechanism that inhibits overexcited (high firing rate) neurons and dead (low firing rate) neurons. It adaptively adjusts the impulse threshold of LIF neurons based on the spatial characteristics of the scene. To dynamically define the impulse threshold V... t th This mechanism uses the global space feature F provided by the Transformer branch. G The representative value K and its entropy value ε are given. The formula for calculating the pulse threshold is as follows:

[0084] V t th =(K+ε)V th

[0085] Among them, V th The threshold is predefined, K is a representative value, and ε is the entropy value. The values ​​of K and ε are calculated as follows:

[0086] K=σ(A(S m t ))

[0087]

[0088] Where σ is a sigmoid function, A is adaptive average pooling, and S... mt It is a matrix that records the average values ​​of the channels for the transformer output characteristics, N is the normalization operation that transforms the values ​​to the range [0,255], and P i Indicates in S m t The upper value represents the number of pixels i, and ξ is a numerical value ε. Intuitively, K reflects contextual information; a higher K value indicates more spatial contextual information in the scene. When K is high, the firing rate of neurons should be reduced to better accumulate and capture temporal cues from longer windows, and vice versa. On the other hand, ε reflects the randomness of features, which may be due to spatial noise. For observations with high entropy, it is desirable to increase the impulse threshold to filter out potential noise. Conversely, if impulses may be triggered by informational cues, the threshold should be decreased to increase sensitivity to these impulses.

[0089] In this embodiment of the invention, the TSFF module consists of three key parts: a temporal attention module (TA), a cross-domain aggregator (CI), and a spatial attention module (SA), such as... Figure 6 As shown,

[0090] Temporal Attention Module (TA): It takes the temporal features obtained through SFE as input and uses channel-level dependencies to generate corresponding enhanced temporal information, given the input temporal features F. T ∈R C×H×W The TA module can be represented by the formula:

[0091] F' T =R ((C H ,W)) (GY)+F T

[0092]

[0093] Where Y = Q = R ((C,HW)) (F T ), G=K=R ((HW,C)) (F T ); R ((·)) It is a shape transformation function with a target shape (·); This represents a softmax function.

[0094] Cross-Domain Aggregator (CI): CI is a cross-domain attention mechanism designed to fuse spatial and temporal cues. Specifically, we generate an attention map based on the spatial and temporal information of the input and then apply it to the temporal cues. Therefore, we can suppress low-entropy information from both spatial and temporal perspectives. CI then fuses the input spatial features and the optimized temporal features through element-wise addition. Given the input temporal features F'...T and spatial features F G The formula can be expressed as:

[0095]

[0096] β=σ(M(A([F G ,F' T ])))

[0097] M is a three-layer multilayer perceptron (MLP) operator, consisting of a linear input layer, a ReLU activation function, and a linear output layer.

[0098] Spatial Attention Module (SA): The fused features are input into a self-attention-based SA module to further enhance the discriminative spatial features. This SA module is similar to the TA module, but its attention map is based on pixel-to-pixel dependencies rather than channel-to-channel dependencies. Formally, this SA module can be represented as:

[0099] F” G =R ((C,H,W)) (Y”G”)+F' G

[0100]

[0101] Where Y” = R ((C,HW)) (F'G),Q”=R ((HW,C)) (F'G), and K” = Y”.

[0102] The event module in this embodiment of the invention can accurately identify feature information in the event stream and extract features, ensuring the accuracy of feature extraction and facilitating subsequent target tracking.

[0103] In some possible embodiments of the present invention, such as Figure 7 As shown, the RGB module includes a second search branch and a second template branch. Both the second search branch and the second template branch include a feature extraction submodule and a feature fusion submodule. The RGB module is used to extract features from the RGB frame stream to obtain an RGB feature map, including:

[0104] S701 uses a feature extraction submodule to extract features from the RGB frame stream to obtain RGB frame stream feature information;

[0105] S702 uses a feature fusion submodule to fuse the feature information of the RGB frame stream to obtain an RGB feature map.

[0106] In this embodiment of the invention, the RGB module inputs the RGB images into the search branch and template branch respectively. After passing through the feature extraction subnetwork and the feature fusion module, RGB fused feature maps are obtained for the classification branch and the regression branch respectively. After adjusting the number of channels of the obtained RGB fused feature maps through 1x1 convolution, they are input into the CDMS (Cross-Modal Fusion) modules of the corresponding classification branch and regression branch respectively. The framework diagram of the feature extraction subnetwork and the feature fusion module is shown below. Figure 8 and 9 As shown, the feature extraction subnetwork consists of two branches: the template image Z serves as the input to the template branch, and the search region image X serves as the input to the search branch. Both branches share the AlexNet convolutional neural network as their backbone, and each branch outputs corresponding feature maps φ(z) and φ(x). To fuse the information from the two branches, φ(z) is used as the convolution kernel of φ(x) to perform a cross-correlation operation, resulting in the corresponding response map R. In subsequent network frameworks, the scale and localization information of the tracked target can be obtained by decoding the response map R; therefore, the response map R should contain relatively rich information. Using deep cross-correlation to generate multi-channel response maps provides rich feature information.

[0107]

[0108] In this diagram, ★ represents a deep cross-correlation operation, a lightweight cross-correlation operation of a transformation type. The generated response map R has the same number of channels as φ(x) and φ(z), containing a large amount of semantic and spatial information. Shallow networks have small receptive fields and small overlapping areas, containing more pixel information, allowing the network to capture more details. They are fine-grained and better at capturing small targets. Shallow features generated by shallow networks contain information such as edges, corners, colors, and shapes, rich in spatial information, making them better for target localization. However, because they undergo fewer convolutions, their semantics are lower and they have more noise. Deep networks have large receptive fields and large overlapping areas, containing information from a whole region. They are fine-grained and capture feature regions between a region and its neighbors. Features extracted by deep networks are closer to the output, containing more abstract information, i.e., semantic information, making them better at target classification. However, because of their lower resolution, they have poorer detail perception and lack spatial awareness. Therefore, convolutional neural networks can capture both shallow and deep features, and can capture features regardless of location. Furthermore, due to the continuous emergence and application of deep networks, deep convolutional networks can obtain more optimized feature maps for target tracking by fusing shallow and deep features, thereby achieving better tracking accuracy.

[0109] The framework selects the last layer features from the last three blocks of the modified AlexNet for feature fusion between shallow and deep layers, labeling them F3, F4, and F5. The template branches correspond to F3(z), F4(z), and F5(z), while the search branches correspond to F3(x), F4(x), and F5(x). This is because the classification branch requires more semantic information features, while the regression branch requires more spatial information features. The three different feature maps used for the classification branch have an increasing proportion of channels (from shallow to deep), while the three different feature maps used for the regression branch have an decreasing proportion of channels (from shallow to deep). This chapter uses 1x1 convolutions to adjust the number of channels corresponding to F3(z), F4(z), and F5(z) and F3(x), F4(x), and F5(x), thus obtaining F3(z) for the classification branch. cls F4(z) cls F5(z) cls and F3(x) cls F4(x) cls F5(x) cls At the same time, F3(z) is obtained for the regression branch. reg F4(z) reg F5(z) reg and F3(x) reg F4(x) reg F5(x) reg .

[0110] In the process of tuning the channel ratio between deep and shallow feature maps, we compared multiple sets of parameters with the 1:1:1 feature fusion method (classification, regression). Based on the experimental tracking results, we ultimately selected 2:3:4 (classification) and 4:3:2 (regression) as the channel ratio parameters. Then, the framework of this chapter will use the F3(z) corresponding to the template branch. cls F4(z) cls F5(z) cls F3(z) reg F4(z) reg F5(z) reg F3(x) corresponding to the regression branch cls F4(x) cls F5(x) cls F3(x) reg F4(x) reg F5(x) reg Perform a deep cross-correlation operation to obtain the corresponding F. 3_cls F 3_reg F 4_cls F 4_reg F 5_cls F 5_regThese six feature maps.

[0111] Feature fusion methods can be divided into early fusion and late fusion. Early fusion methods include FCN, Inside-OutsideNet, and HyperNet, while late fusion methods include FPN and DenseNet. Because the goal is to verify the effectiveness of the proposed idea with minimal increase in computation, and late fusion is more complex than early fusion, this chapter adopts the early fusion approach. Basic feature fusion methods in early fusion include pointwise addition and concatenation, as well as a series of special fusion methods. Since pointwise addition is a special form of concatenation (which includes pointwise addition), and other special fusion methods are too complex and not compatible with the computational power of the loitering munition platform, this chapter chooses the concatenation feature fusion method to verify and determine the feasibility of the idea. The feature fusion module of the RGB framework processes the three feature maps F corresponding to the obtained classification branches. 3_cls F 4_cls F 5_cls The three feature maps F corresponding to the regression branch 3_reg F 4_reg F 5_reg Each feature map is then fused separately. Based on the preceding analysis, this chapter's framework employs a feature concatenation fusion method to perform three-by-three fusion of the feature maps, resulting in fused feature maps F for the classification and regression branches, respectively. cls With F reg .

[0112] In this embodiment of the invention, the RGB module is used to extract features from the RGB frame stream, which facilitates subsequent feature fusion and target tracking.

[0113] In some possible embodiments of the present invention, such as Figure 10 As shown, the cross-modal fusion module includes two cross-modal modulation and selection modules, which in turn include two cross-attention modules and two adaptive weight modules. The cross-modal fusion module fuses the event feature map and the RGB feature map to obtain a fused feature map, including:

[0114] S1001, the event feature map and RGB feature map are initially fused by the cross-attention module to obtain the preliminary fused feature map;

[0115] S1002, the adaptive weight module is used to remove the weights of the preliminary fusion feature map, and the preliminary fusion feature map is fused based on the weights to obtain the fusion feature map.

[0116] In this embodiment of the invention, the cross-modal fusion module is a key module of the target tracking algorithm that fuses the event domain and the RGB domain. It fuses the event feature maps and RGB feature maps obtained from the event module and the RGB module to obtain the final fused feature map, and then inputs it into the classification branch and the regression branch respectively.

[0117] The cross-modal fusion module, used to fuse the obtained RGB features and event features, is a cross-modal attention mechanism. The cross-modal fusion module consists of two CDMS (Cross-Modal Modulation and Selection Modules), each composed of two Cross-Attention Modules (CAB) and two Adaptive Weight Modules (AW). The framework structure of the CDMS is as follows: Figure 11 As shown.

[0118] In some possible embodiments of the present invention, such as Figure 12 As shown, the classification and regression module includes a classification branch and a regression branch jointly trained with IOU. The classification and regression module is used to perform classification and regression analysis on the fused feature map to determine the target to be tracked, including:

[0119] S1201, the feature fusion map is classified using a classification branch to obtain the classification feature map of the target to be tracked;

[0120] S1203 uses regression branching to classify the feature fusion map, obtaining the regression feature map of the target to be tracked.

[0121] In this embodiment of the invention, the classification and regression sub-network includes a classification branch and a regression branch jointly trained with IOU, which work together to predict the location of the target bounding box, thereby performing target tracking. The tracking task is divided into two sub-tasks: a classification task to predict the category of each location and a regression task to calculate the target bounding box. The final fused feature map obtained by feature fusion is input into the classification branch to obtain a classification feature map for target classification. The final fused feature map obtained by feature fusion is input into the regression branch to obtain the regression feature map used for target localization.

[0122] The target tracking method provided by this invention combines the advantages of event cameras and RGB cameras, while mitigating and avoiding the shortcomings of each camera. It can better handle various challenging scenarios such as high-speed movement, excessive darkness and overexposure, and interference from similar objects, resulting in a significant improvement in overall performance. It is suitable for more adverse challenging environments and ensures a certain real-time tracking speed. When facing complex combat missions in all weather and multiple fields, it can provide technical support for accurate target tracking.

[0123] To better implement the target tracking method in the embodiments of the present invention, based on the target tracking method, correspondingly, as follows: Figure 13As shown, this embodiment of the invention also provides a target tracking device, the target tracking device 1300 including:

[0124] The feature extraction module 1301 is used to acquire the event stream and RGB frame stream of the target to be tracked. The event module is used to extract features from the event stream to obtain an event feature map, and the RGB module is used to extract features from the RGB frame stream to obtain an RGB feature map.

[0125] The feature fusion module 1302 is used to fuse the event feature map and the RGB feature map using a cross-modal fusion module to obtain a fused feature map;

[0126] The target recognition module 1303 is used to perform classification and regression analysis on the fused feature map using the classification and regression module to determine the target to be tracked.

[0127] The target tracking device 1300 provided in the above embodiments can realize the technical solutions described in the above target tracking method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above target tracking method embodiments, and will not be repeated here.

[0128] like Figure 14 As shown, the present invention also provides an electronic device 1400. The electronic device 1400 includes a processor 1401, a memory 1402, and a display 1403. Figure 14 Only some components of the electronic device 1400 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0129] In some embodiments, processor 1401 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in memory 1402 or process data, such as the target tracking method of the present invention.

[0130] In some embodiments, processor 1401 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 1401 may be local or remote. In some embodiments, processor 1401 may be implemented on a cloud platform. In some embodiments, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, or any combination thereof.

[0131] In some embodiments, memory 1402 may be an internal storage unit of electronic device 1400, such as a hard disk or memory of electronic device 1400. In other embodiments, memory 1402 may also be an external storage device of electronic device 1400, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on electronic device 1400.

[0132] Furthermore, the memory 1402 may include both internal storage units of the electronic device 1400 and external storage devices. The memory 1402 is used to store application software and various types of data installed on the electronic device 1400.

[0133] In some embodiments, display 1403 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 1403 is used to display information from electronic device 1400 and to display a visual user interface. Components 1401-1403 of electronic device 1400 communicate with each other via a system bus.

[0134] In some embodiments, when processor 1401 executes a target tracking program in memory 1402, the following steps may be performed:

[0135] The event stream and RGB frame stream of the target to be tracked are obtained. The event module is used to extract features from the event stream to obtain an event feature map, and the RGB module is used to extract features from the RGB frame stream to obtain an RGB feature map.

[0136] A cross-modal fusion module is used to fuse the event feature map and the RGB feature map to obtain a fused feature map;

[0137] The classification and regression modules are used to perform classification and regression analysis on the fused feature map to determine the target to be tracked.

[0138] It should be understood that when the processor 1401 executes the target tracking program in the memory 1402, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0139] Accordingly, embodiments of the present invention also provide a computer-readable storage medium for storing a computer-readable program or instruction. When the program or instruction is executed by a processor, it can implement the steps or functions in the target tracking method provided in the above-described method embodiments.

[0140] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0141] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A target tracking method, characterized in that, include: The event stream and RGB frame stream of the target to be tracked are acquired. The event module is used to extract features from the event stream to obtain an event feature map. The RGB module is used to extract features from the RGB frame stream to obtain an RGB feature map. A cross-modal fusion module is used to fuse the event feature map and the RGB feature map; The cross-modal fusion module includes two CDMS modules, which input the feature information obtained from the event module and the RGB module into the classification CDMS module and the regression CDMS module, respectively, and obtain the fusion feature map of the event and the RGB domain through CAB and AW in the CDMS module. Input the fused feature map output by the classification CDMS module into the classification branch, and input the fused feature map output by the regression CDMS module into the regression branch; The fusion feature maps of the input classification branch and the input regression branch are processed by classification and regression modules respectively to determine the target to be tracked. The classification branch obtains a classification feature map for target classification, and the regression branch obtains a regression feature map for target localization. The event module includes a first template branch and a first search branch; The event module is used to extract features from the event stream to obtain an event feature map, including: The event stream is feature extracted using the first template branch and the first search branch to obtain the first template branch event feature map and the first search branch event feature map. Multiple deep cross-correlation tests are performed on the event feature map of the first template branch and the event feature map of the first search branch to obtain the event feature map; Both the first template branch and the first search branch include a pulse self-attention feature extractor and a spatiotemporal feature fusion module. The step of using the first template branch and the first search branch to extract features from the event stream to obtain a first template branch event feature map and a first search branch event feature map includes: The event stream is feature extracted using a deep learning network and a spiking neural network in a spiking self-attention feature extractor to obtain spatiotemporal feature information. The spiking neural network dynamically adjusts the spiking threshold through a dynamic spiking threshold adjustment module. The spatiotemporal feature information is fused using multiple dynamic attention mechanisms of the spatiotemporal feature fusion module to obtain a first template branch event feature map and a first search branch event feature map. The multiple dynamic attention mechanisms include a spatiotemporal attention module, a cross-domain aggregator, and a spatial attention module. The RGB module inputs the RGB image into the search branch and the template branch respectively, and obtains RGB fused feature maps for the classification branch and the regression branch respectively through the feature extraction sub-network and the feature fusion module; The feature extraction subnetwork consists of a template branch and a search branch. The template image is used as the input of the template branch, and the search region image is used as the input of the search branch. The template branch and the search branch share the AlexNet convolutional neural network as the backbone network, and output the feature map φ(z) corresponding to the template image and the feature map φ(x) corresponding to the search region image, respectively. By using φ(z) as the convolution kernel of φ(x) to perform a deep cross-correlation operation, the corresponding response map R is obtained. The scale information and localization information of the tracked target are obtained by decoding the response map R.

2. The target tracking method according to claim 1, characterized in that, The RGB module includes a second search branch and a second template branch. Both the second search branch and the second template branch include a feature extraction submodule and a feature fusion submodule. The RGB module is used to extract features from the RGB frame stream to obtain an RGB feature map, including: The feature extraction submodule is used to extract features from the RGB frame stream to obtain RGB frame stream feature information; The feature fusion submodule is used to fuse the feature information of the RGB frame stream to obtain an RGB feature map.

3. The target tracking method according to claim 1, characterized in that, The cross-modal fusion module includes two cross-modal modulation and selection modules, each comprising two cross-attention modules and two adaptive weighting modules. The cross-modal fusion module is used to fuse the event feature map and the RGB feature map to obtain a fused feature map, including: The event feature map and the RGB feature map are initially fused using a cross-attention module to obtain a preliminary fused feature map. An adaptive weighting module is used to determine the weights of the preliminary fused feature map, and the preliminary fused feature map is fused based on the weights to obtain a fused feature map.

4. The target tracking method according to claim 1, characterized in that, The classification and regression module includes a classification branch and a regression branch jointly trained with classification and IOU. The classification and regression module is used to perform classification and regression analysis on the fused feature map to determine the target to be tracked, including: The feature fusion map is classified using the classification branch to obtain the classification feature map of the target to be tracked; The regression branch is used to classify the feature fusion map to obtain the regression feature map of the target to be tracked.

5. A target tracking device, characterized in that, When the apparatus is operated, it performs the method of claim 1, and the apparatus comprises: The feature extraction module is used to acquire the event stream and RGB frame stream of the target to be tracked. The event module is used to extract features from the event stream to obtain an event feature map, and the RGB module is used to extract features from the RGB frame stream to obtain an RGB feature map. The feature fusion module is used to fuse the event feature map and the RGB feature map using a cross-modal fusion module to obtain a fused feature map; The target recognition module is used to perform classification and regression analysis on the fused feature map using the classification and regression module to determine the target to be tracked.

6. The target tracking device according to claim 5, characterized in that, The cross-modal fusion module includes two cross-modal modulation and selection modules, each comprising two cross-attention modules and two adaptive weighting modules. The cross-modal fusion module is used to fuse the event feature map and the RGB feature map to obtain a fused feature map, including: The event feature map and the RGB feature map are initially fused using a cross-attention module to obtain a preliminary fused feature map. An adaptive weighting module is used to determine the weights of the preliminary fused feature map, and the preliminary fused feature map is fused based on the weights to obtain a fused feature map.

7. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory is used to store a program; and the processor is coupled to the memory to execute the program stored in the memory to implement the steps in the target tracking method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the target tracking method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • High-frame-rate target tracking method based on deep pulse neural network

    CN117876927A

  • Target tracking method fusing RGB / EVENT bimodal samples

    CN118334078A