Bidirectional Adapter-based multi-modal and multi-unmanned aerial vehicle single-target tracking method

Abstract

The invention discloses a multi-mode and multi-unmanned aerial vehicle single-target tracking method based on bidirectional Adapter. The method comprises the steps that S1, input information of each mode / view angle is processed through a double-flow encoder; s2, embedding a bidirectional Adapter module into each layer in each encoder branch, fusing the feature information of any mode with the feature information of another mode from the previous layer by using the bidirectional Adapter module, and transmitting the hidden state of a state space model SSM to a subsequent layer to record the time sequence information of the mode of the current frame; s3, performing time sequence feature fusion on the output of the double-flow encoder, integrating the output of different modes, and sending the output into a prediction head for calculation to obtain a target tracking result; s4, taking the time sequence token of each mode and the hidden state as initialization parameters of a next frame so as to transmit time sequence information of a target tracking trajectory and time sequence information of the modes; and S5, based on the double-flow encoder and the bidirectional Adapter module, realizing multi-mode and multi-unmanned aerial vehicle tracking of a single target.

Description

Technical Field

[0001] The present invention relates to the field of single target tracking, and in particular to a multi-modal and multi-UAV single target tracking method based on a bidirectional adapter. Background Art

[0002] Object tracking is a fundamental visual task in computer vision. Single object tracking (SOT) algorithms are designed to capture the position of a given object in subsequent frames of a given video sequence, given a tracking bounding box for the specified object at the beginning of the video. Single object tracking algorithms must maintain track of any given object, even objects of unknown categories for which they were not trained. Although object tracking algorithms based on RGB images perform well under ideal conditions, illumination variations, shadows, and low light in complex scenes can make it difficult for RGB cameras to effectively capture target information. These factors limit the performance of RGB tracking algorithms, impacting robustness and accuracy. To improve object tracking, multimodal complementary information has been introduced, such as multi-camera collaboration, thermal infrared technology, event cameras, and natural language descriptions, to overcome environmental interference and enhance algorithm robustness and adaptability. Thermal infrared sensors capture the heat emitted by objects, providing clear visualization of targets in low-light or dark environments. Although their resolution is low and lacks texture information, their complementarity with RGB images makes multimodal object tracking possible. Event cameras capture pixel-level visual intensity changes, providing high temporal resolution data, making them particularly suitable for tracking fast-moving targets. In scenarios where the target is obscured, multi-camera collaborative tracking can compensate for the shortcomings of a single perspective by leveraging information from different viewpoints. Natural language text, by describing the semantic attributes of the target, combined with visual information, improves the accuracy of target tracking algorithms in complex backgrounds or with interference from similar targets. This combination of multimodal and multi-view data significantly enhances the adaptability of target tracking algorithms in complex scenarios.

[0003] However, the field lacks a unified network framework for multimodal image tracking (such as thermal infrared and event cameras), multimodal image and text tracking, and multi-UAV single-target tracking. Different network structures are required for different multimodal / multi-UAV target tracking tasks, which not only increases the complexity of model development but also limits the algorithm's versatility and scalability. Furthermore, the high cost of labeling multimodal / multi-UAV viewpoint data makes it difficult to fully train these methods on limited data, limiting algorithm performance. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a multi-modal and multi-UAV single target tracking method based on a bidirectional adapter. Based on the existing single-modal target tracking model, a visual cue fine-tuning framework is designed. This framework can perceive the dynamic changes of the dominant modality in open scenes and efficiently fuse the information of multi-modal / multi-UAV perspectives in an adaptive manner. The bidirectional adapter module designed in the present invention adopts a simple and efficient cross-cue structure, introduces a state-space model (SSM) to perceive the changes in modal dominance between recorded frames, and by modeling temporal dependencies and utilizing the temporal characteristics of the latent state transfer modality, it can better handle complex and changeable multi-modal / multi-perspective tracking tasks. By introducing only 0.36M learnable parameters, the designed model exhibits strong environmental adaptability in open scenes and achieves robust multi-modal and multi-UAV single target tracking.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A multi-modal and multi-UAV single target tracking method based on a bidirectional adapter, including:

[0007] S1. Process the input information of each modality / viewpoint through a dual-stream encoder. The dual-stream encoder includes two encoder branches, one for extracting features of the RGB image modality and the other for extracting features of the other modalities / viewpoints. A learnable temporal token is added to the features of each modality. The temporal information of the target tracking trajectory in the sampled video sequence is stored.

[0008] S2. Each layer in each encoder branch is embedded with a bidirectional adapter module. For each layer in each encoder branch, the bidirectional adapter module is used to fuse the feature information of any modality with the feature information of another modality from the previous layer, and pass the hidden state of the state space model (SSM) to the subsequent layer to record the temporal information of the modality of the current frame. The bidirectional adapter consists of two linear projection layers that change the feature dimension and a Mamba module.

[0009] S3: The output of the dual-stream encoder is subjected to temporal feature fusion, and the outputs of different modalities are integrated and sent to the prediction head for calculation to obtain the target tracking result;

[0010] S4: Use the timing token of each modality and the hidden state of the state space model (SSM) as the initialization parameters of the next frame to transmit the timing information of the target tracking trajectory and the timing information of the modality;

[0011] S5. Based on the above dual-stream encoder and bidirectional adapter module, multi-modal and multi-UAV tracking of a single target is achieved.

[0012] Furthermore, in step S1:

[0013] The input of each modality / viewpoint includes the initial template image of the target to be tracked Historical template images in time series And the image to be tracked in the current frame, that is, the search frame First, block encoding and position encoding are used to convert the RGB image modality input into features Convert thermal infrared, event map, or other drone-view input into features Among them, an additional learnable time sequence token is added to the features of each modality

[0014] Among them, for text input, the text encoder of the CLIP model is used to convert the text input into text tokens, and then a learnable linear projection layer is used to make the text input become Multiply the text input with the features of the RGB image modality to obtain the modality features with text information

[0015] Furthermore, the dual-stream encoder uses Vision Transformer as the backbone network, and each encoder branch consists of 12 layers of Transformer blocks to perform joint feature extraction on the template frame and search frame of each modality; each encoder branch is initialized using the network parameters of ODTrack and all its parameters are frozen during training.

[0016] Furthermore, the bidirectional adapter is modularly embedded into the attention layer and feed-forward layer MLP of each layer in each encoder branch to provide cross-modal / view feature prompts; it collaboratively enhances the target features of each encoder branch while transferring and updating the hidden state of each encoder branch.

[0017] Furthermore, taking the processing of a certain encoder branch M1 as an example, the i-th layer feature is updated as follows:

[0018]

[0019] Among them F Att is the multi-head attention layer, h i-1 is the hidden state calculated by the previous layer of the dual-stream encoder, F Ada Output the cross-modal hint of the other encoder branch M2 mode to obtain cross-modal features and hidden state h′ i-1 ; and in the feedforward stage FMLP Further integrate cross-modal features to achieve bidirectional collaborative optimization of cross-modal features; while transmitting feature hints, it also outputs the hidden state h calculated by feature hints i , and pass it to the bidirectional Adapter module in the next layer of the dual-stream encoder;

[0020] At any layer of any branch of the two-stream encoder, the input feature dimension is first reduced to d by one of the linear projection layers. e , sent to the Mamba module for processing; let the input feature be First, normalize it by RMSNorm to get

[0021] Then it will Through the two parallel branches of the Mamba module, one of the parallel branch features F is activated using the SiLU function after passing through the linear projection layer inside the Mamba module to generate the gating signal;

[0022] Another parallel branch feature F is first dimensionally expanded through the linear projection layer within the Mamba module, and then the local spatial features are extracted through the convolution layer. After being processed by the SiLU activation function, the corresponding parallel branch feature F is input into the state space model SSM for dynamic extraction and update of context information to obtain the modal feature information of the corresponding layer.

[0023] For the state space model SSM, the double-stream encoder is used to correspond to the hidden state h of the previous layer in the branch i-1 As input, it is used to pass the modal feature information perceived by the previous layer; the features of the two encoder branches are weighted and projected down to the dimension d of the original input features e ;

[0024] Finally with Add up the output of the Mamba module Like other Adapter modules, a linear projection layer is used to transform The processed cross-modal feature hints are finally obtained by projecting it onto the original dimension of the input feature.

[0025] Furthermore, the prediction head is initialized using the same structure and parameters as in ODTrack; the prediction head uses a fully convolutional network (FCN) to convert the features output by the dual-stream encoder into a two-dimensional spatial feature map, and outputs the target classification score map, offset, and normalized bounding box.

[0026] Furthermore, for the initialization of the timing tokens of each encoder branch, the timing token initialized with all zeros is added to the timing token output by the encoder branch of the previous frame, and the timing information of the target tracking trajectory is propagated in this autoregressive manner.

[0027] Furthermore, for the initial input hidden state of each encoder branch or Initialize the first frame of the video with all zeros; when processing the hidden state of the kth frame, use the hidden state of the k-1th frame output by the other encoder branch image through the dual-stream encoder or Parameter initialization, exchanging the hidden states of the two encoder branches enables different encoder branches to refer to each other's modal temporal features;

[0028]

[0029] During training, each branch of the dual-stream encoder and all parameters of the prediction head are frozen, and only the parameters of the bidirectional adapter module are trained to achieve cross-modal / viewpoint feature hints. are the branch timing tokens of the kth frame of the two encoder branches respectively, They are the branch timing tokens of the k-1th frame of the two encoder branches respectively.

[0030] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the multi-modal and multi-UAV single-target tracking method based on a bidirectional adapter are implemented.

[0031] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-modal and multi-UAV single target tracking method based on a bidirectional adapter.

[0032] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0033] 1. Multimodal and multi-UAV single target tracking visual cueing framework based on bidirectional adapter module:

[0034] The present invention adopts a dynamic modality prompt fusion strategy. Through a non-fixed dominant modality perception mechanism, it can detect and capture the dynamic changes of the dominant modality in real time in open scenes, adaptively adjust multimodal data fusion, and significantly improve the accuracy and robustness of target tracking.

[0035] In addition, the present invention adopts a dual-stream encoder structure with multi-stream input, which can extract features from RGB images and other modalities (such as thermal infrared, event camera, text and drone perspective) respectively, and realize collaborative enhancement of cross-modal information through visual cues.

[0036] This solution can adaptively adjust the information fusion weights in different scenarios (such as low light, partial occlusion of the target or rapid motion), effectively overcoming the problem that a single modality (such as RGB images) is easily limited by lighting, shadows, low resolution, etc. in complex backgrounds, thereby greatly improving the robustness and accuracy of target tracking.

[0037] By complementing multimodal data and integrating multiple drone perspectives, it is possible to provide more comprehensive information when the target is obscured or the background changes, making the tracking results more stable and accurate.

[0038] 2. Bidirectional Adapter Design Based on State Space Model (SSM):

[0039] A bidirectional Adapter module is embedded in each layer of each encoder branch of the dual-stream encoder, allowing features from either modality to be complementary to features from the other modality (or drone view).

[0040] By using the state space model, the temporal information between modalities is recorded and transmitted by passing the hidden state of the previous frame, thereby capturing the dynamic changes of the dominant modality between frames.

[0041] Only 0.36M learnable parameters are added, which ensures the high efficiency of the system without significantly burdening the original model structure.

[0042] By utilizing a concise and efficient cross-cue mechanism and hidden state transfer, temporal information of different modalities and perspectives can be efficiently shared within the network, thereby significantly improving the ability to capture the dynamic features of the target during tracking.

[0043] Since only a small number of additional parameters are introduced, the system ensures reasonable control of computational complexity and model complexity while fusing multimodal information, making it suitable for real-time tracking tasks in actual complex environments.

[0044] 3. Unified network structure and multi-dimensional input processing:

[0045] A dual-stream encoder based on VisionTransformer is used as the backbone network. Each encoder branch consists of 12 layers of Transformer blocks, and inputs of different modalities (including vision and text) are processed uniformly.

[0046] The time series token of each modality is initialized and updated through autoregression to achieve dynamic propagation of the target time series features.

[0047] A freezing strategy is adopted for the network parameters of single-modal basic feature extraction to ensure the utilization of mature model features, while only fine-tuning for multi-modal / multi-UAV tasks is performed on the Adapter module.

[0048] This unified network design eliminates the tediousness of designing different network structures for different multimodal or multi-UAV missions, improves the versatility and scalability of model development, and reduces the difficulty of training caused by the high cost of data annotation.

[0049] While integrating multiple data sources such as vision, thermal infrared, event cameras, and text, it can also take into account feature cues from multiple drone perspectives, fully utilize the feature extraction capabilities of existing single-modal models, and realize cross-modal feature optimization in multi-task scenarios, thereby demonstrating excellent tracking effects on multiple datasets (such as RGBT234, LasHeR, VisEvent, TNL2K, and MDOT2).

[0050] In summary, the present invention can timely capture the changes in the dominant mode in open scenes, adaptively adjust the fusion of multimodal data, and significantly improve the accuracy and robustness of target tracking. The bidirectional Adapter module based on the state-space model is not only simple in structure and clear in cross-prompt, but also requires only a small number of additional parameters (0.36M), so that the overall model can achieve both high performance and efficient calculation. The unified network framework realizes the fusion processing of visual, textual and multi-UAV perspective data, reduces the complexity of the architectural design between different tasks, has great potential for promotion and application, and can achieve stable and accurate single target tracking in a variety of complex scenarios.

[0051] With its innovative bidirectional adapter design and unified multimodal network architecture, this invention achieves efficient fusion and robust tracking of multimodal and multi-UAV data in complex scenarios. It not only achieves theoretical and technological breakthroughs, but also verifies its practical application advantages in experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 and Figure 2 Schematic diagram of the flow of a single target tracking method in a specific embodiment;

[0053] Figure 3 FIG. 4 is a schematic diagram of a bidirectional adapter module in a specific embodiment. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] Example 1

[0056] This embodiment proposes a multimodal and multi-UAV single-target tracking network model based on a bidirectional adapter. This network model uses a dual-stream encoder architecture with a non-fixed dominant modality, adapting to scenarios in open worlds where the dominant modality dynamically changes. Furthermore, the bidirectional adapter module enables efficient feature prompting across different branches of the dual-stream encoder. This network model uses hidden states and temporal tokens to transfer tracking and modality temporal information. This solves and circumvents technical issues existing in the background art, enabling better utilization of existing data and improving target tracking performance.

[0057] like Figure 2 As shown in the figure, a multi-modal and multi-UAV single target tracking method based on a bidirectional adapter consists of three parts, and the functions of each part are as follows:

[0058] In the encoder part of the dual-stream structure, this embodiment uses a pre-trained Vision Transformer to perform joint feature extraction on the template frame sequence and the search frame.

[0059] In each layer of the dual-stream encoder, a bidirectional Adapter module is used for cross-modal and cross-view feature prompts, and efficient parameter fine-tuning is used to achieve multi-modal and drone-perspective single target tracking.

[0060] The output features of the encoder are calculated using the pre-trained prediction head to obtain the resulting box coordinates of the target tracking.

[0061] The task definition specifically includes:

[0062] The multi-modal and multi-UAV single target tracking task based on the bidirectional adapter module is to calculate the position of the template target in each subsequent search image after the initial template frame image of the target to be tracked is given for the video sequence to be tracked. Specifically, for a given video image [I1,I2,……,I k ]First frame I in different modes / views template The initial bounding box position of the target object is B0, and the single target tracking algorithm learns to search Search for the object and calculate its bounding box position B k

[0063] The specific steps of implementing the single target tracking method based on the above single target tracking network model are as follows:

[0064] 101: The dual-stream encoder processes input information from each modality, extracting features from the RGB image and the other modality / viewpoint separately. The dual-stream encoder uses the Vision Transformer as the backbone network, initialized with the network weights from ODTrack. Each branch consists of 12 Transformer layers, jointly extracting features from the search frame and the template.

[0065] 102: The bidirectional adapter module fuses feature information from different modalities within each feature layer and transfers hidden state information. The bidirectional adapter module is embedded in the encoder's attention layer and feedforward layer to perform cross-modal / viewpoint feature cues, collaboratively enhancing the target features of each branch while simultaneously transferring and updating the hidden state of each branch.

[0066] 103: Perform temporal feature fusion on the features output by the dual-stream encoder, and then integrate the outputs of each modality and send them to the prediction head to calculate the target tracking result.

[0067] 104: The time series token and hidden state of each modality are used as initialization parameters for the next frame to transmit the time series information of the target tracking trajectory and the time series information of the modality. The time series token and hidden state are initialized and propagated through autoregression to transmit and update the time series trajectory characteristics of the target and the time series information of the modality.

[0068] Based on the dual-stream encoder and bidirectional adapter module, multi-modal and multi-UAV tracking of a single target is achieved. The prediction head uses a fully convolutional network to convert the encoder output features into a two-dimensional spatial feature map, outputting the target classification, offset, and bounding box, and summarizing them to obtain the final tracking result.

[0069] This embodiment implements robust multimodal and multi-UAV target tracking through a bidirectional adapter module. It can effectively fuse features from different modalities and perspectives in complex environments and transmit temporal information, thereby achieving efficient multimodal / perspective feature fusion and accurate target tracking results.

[0070] Example 2

[0071] This embodiment further introduces the technical solution of the above-mentioned single target tracking method in combination with specific data and calculation formulas, as follows:

[0072] 1. Data Preparation

[0073] This embodiment can support image multimodality such as RGBT / E, image and text multimodality, and single target tracking tasks from multiple drone perspectives. This embodiment verifies the effectiveness of the proposed method on the RGBT234, LasHeR, VisEvent, TNL2K, and MDOT datasets.

[0074] RGBT234 provides 234 aligned video sequences of visible light and thermal infrared. It provides 12 attributes, including LI (low illumination), occlusion, DEF (deformation), motion, etc. The total number of frames is approximately 234K, and the maximum number of frames per sequence is 8K. The RGBT234 dataset provides ground truth labels for both RGB and TIR modalities, which can be used to evaluate the multimodal performance of the tracker. Since RGBT234 uses a visible light-thermal infrared imaging system with parallel optical axes, no pre-processing or post-processing (such as stereo matching and color correction) is required. Its cross-modal alignment is more accurate, but due to physical factors such as the imaging mechanism, the true values of RGB and TIR are still not completely consistent. Therefore, for fair comparison, this embodiment follows the evaluation method of multimodal target tracking and uses MPR (maximum precision rate) and MSR (maximum success rate) instead of PR and SR of single-modal target tracking as evaluation indicators. Specifically, for each frame, the Euclidean distance between the predicted result box and the true value is calculated in RGB and TIR modes respectively, and the smaller distance is used to calculate the accuracy.

[0075] LasHeR is also an RGBT single object tracking dataset, containing 1224 RGBT sequences and 730K frames, captured on various types of imaging platforms. It includes 19 video attributes, adding 7 new attributes such as "HI" (high illumination), "FL" (frame dropout), and "AIV" (abrupt illumination change) based on the previous RGBT234, making it a more challenging RGBT object tracking task dataset.

[0076] VisEvent aims to provide a dedicated training and evaluation platform for visible light event tracking algorithms. The dataset contains 820 video pairs, with an average of 450 frames per video, providing a large number of frame-by-frame annotated video sequences. The dataset includes 709 short-term tracking videos and 111 long-term tracking videos, primarily focusing on pedestrians and vehicles, which conform to a long-tail distribution. This dataset realistically reflects scenes closely related to daily life and facilitates the construction of robust and flexible trackers.

[0077] TNL2K is an image-text dataset for natural language tracking, containing 1,244,340 2k resolution images and 630 words. The text labels for video sequences tend to describe the attributes and initial position of the target. The sequences are mostly sourced from images captured by YouTube, surveillance cameras, and mobile phones. The dataset provides the categorical attributes of the tracked objects and includes labels for when the target disappears from view. The dataset provides 17 attribute labels for the video sequences, introducing challenging attributes such as adversarial examples (AS) and modality swapping (MS) for the object tracking task for the first time.

[0078] MDOT is a multi-UAV single-target tracking dataset based on drone perspectives, the first in its field. It contains a total of 155 video clips and 259,793 high-resolution images. In each video clip, the same target is tracked by multiple drones. The dataset annotates each video clip with 10 different attribute types, providing strong data support for multi-UAV single-target tracking tasks. It covers nine target categories: pedestrians, cars, horse-drawn carriages, motorcycles, bicycles, tricycles, trucks, dogs, and buses, with a high degree of diversity within each category. The same target is recorded by multiple drones from different perspectives and heights to ensure diversity. The dataset is divided into two subsets: the two-UAV dataset (MDOT-2) contains 92 video clips and 113,918 high-resolution images, while the three-UAV dataset (MDOT-3) contains 63 video clips and 145,875 high-resolution images.

[0079] 2. Multimodal and Multi-UAV Single Target Tracking Network Structure (MAT) Based on Bidirectional Adapter

[0080] This embodiment provides a multi-modal and multi-UAV single target tracking method (MAT) based on a bidirectional adapter, which specifically includes:

[0081] First, the input information of different modalities or perspectives is subjected to feature extraction through a dual-stream encoder, and the features of different modalities are fused using a bidirectional adapter in each encoding layer. At the same time, the modality-dominant information of the current frame is transmitted through the hidden state of the state space model (SSM).

[0082] Secondly, the features output by the dual-stream encoder are temporally fused and fed into the prediction head to generate the target tracking results. The temporal tokens and hidden states of each modality are used as initialization parameters for the next frame in an autoregressive manner, enabling the transfer of temporal information across frames.

[0083] Specifically, each modal input includes the initial template of the target to be tracked, historical templates, and the search image of the current frame. Feature encoding is performed on the image input, and learnable temporal tokens are additionally introduced. Text input is converted by the CLIP text encoder and fused with image features to obtain modal features that carry textual semantic information.

[0084] like Figure 2 As shown in Figure 2, the two-stream encoder uses the Vision Transformer as the backbone, and each branch consists of a 12-layer Transformer block, which is initialized with pre-trained parameters and has its parameters frozen during training.

[0085] The bidirectional adapter is embedded in the attention layer and feedforward layer of the encoder in a modular form, and realizes the bidirectional collaborative optimization of modal features through cross-modal feature prompts. Figure 3 As shown in the figure, the specific structure includes a linear projection layer and a Mamba module, which dynamically extracts context information and updates the modal hidden state through a state space model.

[0086]

[0087] Among them F Att is the multi-head attention layer, h i-1 is the hidden state calculated by the previous layer of the dual-stream encoder. Ada Output the cross-modal hint of the other encoder branch M2 mode to obtain cross-modal features and hidden state h′ i-1 . And in the feedforward stage F MLP Further integrate cross-modal features to achieve bidirectional collaborative optimization of cross-modal features. While transmitting feature hints, it also outputs the hidden state h calculated by feature hints. i , and pass it to the Adapter module in the next layer of the dual-stream encoder;

[0088] The prediction head uses a series of fully convolutional networks (FCNs) to convert the output of the two-stream encoder into a two-dimensional spatial feature map and output the target location, offset and bounding box.

[0089] During training, by freezing each branch of the dual-stream encoder and all parameters of the prediction head, only the bidirectional Adapter module is optimized to achieve efficient cross-modal feature prompts. The additional parameters are less than 0.5% of the basic network, showing good target tracking performance in complex environments.

[0090] 3. Evaluation Metrics and Protocols

[0091] This example uses precision (PR) and area under the curve (SR) to evaluate the performance of the tracking method. Specifically:

[0092] Precision: The center position error is defined as the average Euclidean distance between the center position of the tracked object and the true value. The average center position error of all frames of a sequence is used to summarize the overall performance of the sequence. However, when the tracker loses the target, the output position may be random and the average error value may not correctly measure the tracking performance. Therefore, the precision map shows the percentage of frames whose estimated position is within a given threshold distance of the true label. The score with a threshold of 20 pixels is used as the precision to measure the overall tracking performance.

[0093] AUC: Another evaluation metric is bounding box overlap. Given a tracked bounding box and a ground-truth bounding box, we count the number of successful frames where the overlap is greater than a given threshold. The success rate graph shows the proportion of successful frames at thresholds varying from 0 to 1. Using a single success rate value at a specific threshold for tracker evaluation may be unfair or unrepresentative. Tracking algorithms are ranked using the area under the curve (AUC) for each success rate graph.

[0094] 4. Details of model usage

[0095] 1. Model optimization: In the training process, the batch size of the embodiment of the present invention is 8, and the AdamW optimization algorithm is used for training for 60 cycles with a learning rate of 1×10 -4 , which decreases at epoch 48. During training, all parameters of the dual-stream encoder and prediction head are frozen, and only the bidirectional Adapter module is fine-tuned.

[0096] 2. Hyperparameter settings: In this example, the template frame size is set to 192*192; the search frame size is 384*384. The historical template frame sequence used is 3.

[0097] Example 3

[0098] This example applies the aforementioned single-target tracking method to the visible-thermal-infrared RGBT single-target tracking task and compares it with multiple methods on the RGBT234 and LasHeR datasets. The results are shown in Table 1.

[0099] On RGBT234, the MAT of the embodiment of the present invention achieved the best performance; the MPR and MSR scores of 92.1% and 68.6% were higher than MambaVT by 1.4% and 1.3%. These data fully demonstrate the effectiveness of the MAT framework. Like MambaVT, it introduces multiple frames to supplement the contextual information of the target; based on the efficient fine-tuning of parameters using the Transformer framework, the performance is not inferior to or even better than related methods that use a stronger feature extraction module and perform full parameter fine-tuning. On the LasHeR dataset, which introduces more extreme attributes, the gap between various competing methods is further widened. In contrast, MAT leads the pack, achieving a PR of 76.7% and an SR score of 61.2%, which is 2.5% and 2.1% higher than the second-place competing method. This fully demonstrates the concise and efficient cross-modal feature extraction capabilities of the present invention's example.

[0100] Table 1

[0101]

[0102]

[0103] Example 4

[0104] This example applies the aforementioned single-target tracking method to the RGBE single-target tracking task in visible light event maps and compares it with multiple methods on the VisEvent dataset. The results are shown in Table 2. The performance ratio (PR) and performance ratio (SR) scores of 79.5 and 62.5, respectively, are superior to those of the compared methods. However, RGBE data does not offer the same strong modality-dominant differences and complementary properties as RGBT data, so the MAT method does not significantly outperform SeqTrackv2.

[0105] Table 2

[0106]

[0107]

[0108] Example 5

[0109] This embodiment applies the aforementioned single target tracking method to the image-text single target tracking task, and compares it with multiple methods on the TNL2K dataset. The results are shown in Table 3. Among them, 'BB' means that only the template frame is used for tracking. 'BB+NL' means multimodal tracking using the template frame and text description. On a more challenging dataset such as TNL-2K, the fusion of text and image features is particularly important, and a higher score cannot be achieved by relying solely on the excellent design of the target tracking network. MAT achieved an AUC of 63.7% and an accuracy of 67.8%, which is slightly better than UVLTrack, achieving an AUC score of 0.6% and an accuracy lead of 1.1%. Experiments have proved that the embodiment of the present invention has the ability to extract image and text feature cues at the same time, and can achieve performance that is no less than that of other specially designed image and text feature extraction networks.

[0110] Table 3

[0111]

[0112]

[0113] Example 6

[0114] This embodiment applies the aforementioned single target tracking method to the multi-UAV single target tracking task, and compares it with multiple methods on the MDOT2 dataset. For the multi-UAV single target tracking scenario, there are not many multi-UAV tracking methods that can be compared due to limited related research in the field. The results were evaluated using Lasot's evaluation criteria, and the experimental results are shown in Table 4. The MAT of this embodiment achieved an AUC of 53% and an accuracy of 70.8% by using a bidirectional Adapter for joint feature extraction. In particular, it was 5% higher than the accuracy of TransMDOT in the perspective of Drone1, leading all other methods, fully demonstrating the effectiveness of the method of this embodiment, and that it can also achieve "homogeneous heterogeneous" cross-perspective feature prompts in the multi-UAV target tracking scenario.

[0115] Table 4

[0116]

[0117] In summary, this paper proposes a multimodal and multi-UAV single-target tracking visual cueing framework based on a bidirectional adapter. This framework adaptively perceives and effectively integrates the dynamic changes of the dominant modality in open scenes through a dynamic modal cue fusion strategy, thereby improving the accuracy and robustness of multimodal information fusion and optimizing the multi-UAV target tracking performance.

[0118] This paper designs a bidirectional adapter module based on a state-space model. This module employs a simple and efficient cross-cue architecture, leveraging the latent state of the state-space model to convey temporal information about the dominant modality. By adding only 0.36M learnable parameters, it improves the robustness and accuracy of multimodal and multi-UAV target tracking in complex environments.

[0119] This paper proposes a unified network structure for multi-dimensional input: this unified network structure supports visual multimodality, multi-drone perspective and visual-text multimodal tasks. By maximizing the feature extraction capabilities of existing unimodal models, it achieves fine-tuning of feature prompts for different tasks. Effective experimental verification has been carried out on multiple datasets, demonstrating its wide application potential.

[0120] Therefore, this paper adopts a dynamic modality fusion strategy to improve the robustness and accuracy of information fusion. At the same time, a unified network structure is designed to support multi-dimensional input tasks. Through feature-cue fine-tuning, it demonstrates excellent performance on multiple datasets.

[0121] Preferably, the embodiments of the present application further provide a specific implementation of an electronic device capable of implementing all steps of the multi-modal and multi-UAV single target tracking method based on a bidirectional adapter in the above embodiment, wherein the electronic device specifically includes the following contents:

[0122] Processor (processor), memory (memory), communication interface (CommunicationsInterface) and bus;

[0123] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0124] The processor is used to call the computer program in the memory. When the processor executes the computer program, all steps of the multi-modal and multi-UAV single target tracking method based on the bidirectional adapter in the above embodiment are implemented.

[0125] Preferably, an embodiment of the present application also provides a computer-readable storage medium capable of implementing all steps of the multi-modal and multi-UAV single target tracking method based on a bidirectional adapter in the above embodiment. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements all steps of the multi-modal and multi-UAV single target tracking method based on a bidirectional adapter in the above embodiment.

[0126] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the hardware + program embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0127] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0128] Although the present application provides method operation steps such as embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many steps and does not represent the only execution order. When an actual device or client product is executed, it can be executed in the order shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0129] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0132] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, those skilled in the art may make various specific modifications based on the teachings of the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. A multi-modal and multi-UAV single target tracking method based on a bidirectional adapter, characterized by: include: S1. Processing the input information of each modality / viewpoint through a dual-stream encoder, wherein the dual-stream encoder includes two encoder branches, one of which is used to extract features of the RGB image modality and the other is used to extract features of other modalities / viewpoints; An additional learnable time sequence token is added to the features of each modality; the time sequence information of the target tracking trajectory in the sampled video sequence is stored; S2. Each layer in each encoder branch is embedded with a bidirectional adapter module. For each layer in each encoder branch, the bidirectional adapter module is used to fuse the feature information of any modality with the feature information of another modality from the previous layer, and pass the hidden state of the state space model (SSM) to the subsequent layer to record the temporal information of the modality of the current frame. The bidirectional adapter consists of two linear projection layers that change the feature dimension and a Mamba module. S3: The output of the dual-stream encoder is subjected to temporal feature fusion, and the outputs of different modalities are integrated and sent to the prediction head for calculation to obtain the target tracking result; S4: Use the timing token of each modality and the hidden state of the state space model (SSM) as the initialization parameters of the next frame to transmit the timing information of the target tracking trajectory and the timing information of the modality; S5. Based on the above dual-stream encoder and bidirectional adapter module, multi-modal and multi-UAV tracking of a single target is achieved.

2. The method for multi-modal and multi-UAV single target tracking based on a bidirectional adapter according to claim 1, characterized in that: In step S1: The input of each modality / viewpoint includes the initial template image of the target to be tracked Historical template images in time series And the image to be tracked in the current frame, that is, the search frame First, block encoding and position encoding are used to convert the RGB image modality input into features Convert thermal infrared, event map, or other drone-view input into features Among them, an additional learnable time sequence token is added to the features of each modality Among them, for text input, the text encoder of the CLIP model is used to convert the text input into text tokens, and then a learnable linear projection layer is used to make the text input become Multiply the text input with the features of the RGB image modality to obtain the modality features with text information 3. The multi-modal and multi-UAV single target tracking method based on a bidirectional adapter according to claim 1, characterized in that: The dual-stream encoder uses the Vision Transformer as the backbone network. Each encoder branch consists of 12 layers of Transformer blocks to perform joint feature extraction on the template frame and search frame of each modality. Each encoder branch is initialized with the network parameters of ODTrack and all its parameters are frozen during training.

4. The method for multi-modal and multi-UAV single target tracking based on a bidirectional adapter according to claim 1, characterized in that: The bidirectional adapter is modularly embedded into the attention layer and feed-forward layer MLP of each layer in each encoder branch to provide cross-modal / view feature prompts; it collaboratively enhances the target features of each encoder branch while transferring and updating the hidden state of each encoder branch.

5. The multi-modal and multi-UAV single target tracking method based on a bidirectional adapter according to claim 1, characterized in that: Taking the processing of a certain encoder branch M1 as an example, the feature update of the i-th layer is: Among them F Att is the multi-head attention layer, h i-1 is the hidden state calculated by the previous layer of the dual-stream encoder, F Ada Output the cross-modal hint of the other encoder branch M2 mode to obtain cross-modal features and hidden state h′ i-1 ; and in the feedforward stage F MLP Further integrate cross-modal features to achieve bidirectional collaborative optimization of cross-modal features; while transmitting feature hints, it also outputs the hidden state h calculated by feature hints i , and pass it to the bidirectional Adapter module in the next layer of the dual-stream encoder; At any layer of any branch of the two-stream encoder, the input feature dimension is first reduced to d by one of the linear projection layers. e , sent to the Mamba module for processing; let the input feature be First, normalize it by RMSNorm to get Then it will Through the two parallel branches of the Mamba module, one of the parallel branch features F is activated using the SiLU function after passing through the linear projection layer inside the Mamba module to generate the gating signal; Another parallel branch feature F is first dimensionally expanded through the linear projection layer within the Mamba module, and then the local spatial features are extracted through the convolution layer. After being processed by the SiLU activation function, the corresponding parallel branch feature F is input into the state space model SSM for dynamic extraction and update of context information to obtain the modal feature information of the corresponding layer. For the state space model SSM, the double-stream encoder is used to correspond to the hidden state h of the previous layer in the branch i-1 As input, it is used to transmit the modal feature information perceived by the previous layer; The features of the two encoder branches are weighted and projected down to the dimension d of the original input features e ; Finally with Add up the output of the Mamba module Like other Adapter modules, a linear projection layer is used to transform The processed cross-modal feature hints are finally obtained by projecting it onto the original dimension of the input feature.

6. The method for multi-modal and multi-UAV single target tracking based on a bidirectional adapter according to claim 1, characterized in that: For the prediction head, the same structure and parameters as in ODTrack are used for initialization; the prediction head uses a fully convolutional network (FCN) to convert the features output by the dual-stream encoder into a two-dimensional spatial feature map, and outputs the target classification score map, offset, and normalized bounding box.

7. The method for multi-modal and multi-UAV single target tracking based on a bidirectional adapter according to claim 1, characterized in that: For the initialization of the timing token of each encoder branch, the timing token initialized with all zeros is added to the timing token output by the encoder branch of the previous frame, and the timing information of the target tracking trajectory is propagated in this autoregressive manner.

8. The method for multi-modal and multi-UAV single target tracking based on a bidirectional adapter according to claim 1, characterized in that: For each encoder branch, the initial input hidden state or Initialize the first frame of the video with all zeros; when processing the hidden state of the kth frame, use the hidden state of the k-1th frame output by the other encoder branch image through the dual-stream encoder or Parameter initialization, exchanging the hidden states of the two encoder branches enables different encoder branches to refer to each other's modal temporal features; During training, each branch of the dual-stream encoder and all parameters of the prediction head are frozen, and only the parameters of the bidirectional adapter module are trained to achieve cross-modal / viewpoint feature hints. are the branch timing tokens of the kth frame of the two encoder branches respectively, They are the branch timing tokens of the k-1th frame of the two encoder branches respectively.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the multi-modal and multi-UAV single target tracking method based on a bidirectional adapter are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-modal and multi-UAV single target tracking method based on a bidirectional adapter are implemented.

Images