A unified multi-modal visual tracking method based on hybrid expert model

Abstract

This application belongs to the field of multimodal visual recognition technology. It provides a unified multimodal visual tracking method based on a hybrid expert model. The embodiments disclosed herein utilize the MAEF module and adaptive weight generation to dynamically balance heterogeneous modal contributions, solving the modality fusion problem and improving tracking robustness in complex scenes. By leveraging the USMoE module and the UTD task-aware routing strategy, modality-specific, collaborative, and shared representations are decoupled and dynamically fused, mitigating modal interference and balancing structural uniformity and functional specificity. A hybrid mini-batch joint training strategy is adopted to reduce dependence on large-scale single-modal data, enabling the model to adaptively adapt to multiple modality combinations and enhancing scalability and generalization ability. Sparse expert activation and lightweight module design are employed to improve tracking accuracy and ensure computational efficiency, allowing direct integration into existing RGB-based trackers without large-scale reconstruction.

Description

Technical Field

[0001] This disclosure relates to the field of multimodal visual recognition technology, and in particular to a unified multimodal visual tracking method based on a hybrid expert model. Background Technology

[0002] Visual target tracking is a core task in computer vision, aiming to continuously estimate the position of targets in video frames, providing technical support for key applications such as autonomous driving, robotics, and surveillance. With the development of deep learning and the Transformer architecture, red-green-blue (RGB) trackers have made significant progress, but their performance tends to degrade significantly in complex scenarios such as varying lighting, occlusion, or motion blur. To improve tracking robustness, multimodal visual tracking technology has emerged, optimizing performance by fusing complementary sensor signals such as thermal imaging (T), depth (D), and event (E): thermal cameras provide temperature cues under low-light conditions, depth sensors reveal target geometry, and event cameras capture high-speed motion. The combination of these heterogeneous cues enables the system to have more reliable perception capabilities in diverse environments. Currently, multimodal target tracking tasks mostly adopt specific pairwise modal tracking methods, which manually design fusion strategies for specific modal combinations. Essentially, these methods are task-specific, which limits the versatility of the models and makes it difficult to extend them to other modal combinations or hybrid modal scenarios.

[0003] Therefore, many scholars have explored a unified framework that can handle multiple modality pairs in a single model to improve the problem of insufficient model generality caused by design fragmentation. Representative methods such as ViPT and XTrack employ a Transformer-based backbone network and introduce modality cues, labels, or adapters to adapt to different auxiliary modalities. At the same time, they improve training efficiency and architectural consistency by sharing most of the network parameters across tasks.

[0004] However, although existing unified multimodal visual tracking methods effectively address design fragmentation, they still face the following three problems: 1) Significant modal differences: The sensing principles, signal densities, noise patterns, and time synchronization characteristics of different modalities are fundamentally different, resulting in highly inconsistent feature distributions and hindering the extraction and alignment of cross-modal semantically consistent representations; 2) Unified training is prone to modal interference: When training multiple modalities jointly, the dominant modality (usually RGB) will dominate the optimization process, inhibiting the learning of auxiliary modalities and causing cross-modal performance imbalance; 3) Over-reliance on large-scale single-modal data (especially pure RGB datasets): Without such auxiliary supervision, the unified model is difficult to converge and has poor scalability to new modal combinations, limiting practical deployment capabilities.

[0005] Existing unified multimodal visual tracking methods primarily focus on structural uniformity, employing homogeneous feature extractors and fusion mechanisms across modalities, often neglecting modal heterogeneity. Furthermore, existing methods often lack mechanisms to dynamically adjust modal contribution weights based on data quality or task context, resulting in insufficient robustness and scalability in complex scenarios. Therefore, this application proposes a unified multimodal visual tracking method based on a hybrid expert model. This method first employs a hybrid mini-batch joint training mode, allowing the model to simultaneously learn cross-modal commonalities and modality-specific discriminative properties within a single parameter space, improving the model's generalization ability. Next, a modality-aware expert fusion module is constructed, generating dynamically balanced heterogeneous modal contributions through adaptive weights. Finally, a unified architecture hybrid expert model and task-aware routing mechanism are used to achieve decoupled modeling of modality-specific, collaborative, and shared representations.

[0006] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0007] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0008] The purpose of this disclosure is to provide a unified multimodal visual tracking method based on a hybrid expert model, thereby overcoming, to at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0009] According to embodiments of this disclosure, a unified multimodal visual tracking method based on a hybrid expert model is provided, comprising: Step S1: Construct mixed mini-batch training data, where each mini-batch sample contains pure RGB data, RGB-T data, RGB-D data, and RGB-E data; Step S2: Construct a target tracking framework based on a hybrid expert model. This framework includes a MAEF module, a UTD module, a backbone network, and a decoder. The backbone network comprises multiple Transformer blocks containing USMoE modules and self-attention mechanisms. The MAEF module includes average pooling layers, fully connected layers, and a feature refinement expert module. Pooling layers are used to compute global feature representations, fully connected layers are used to generate modality fusion weights, and the feature refinement expert module is used to refine the weighted fusion features. The UTD module includes a classifier composed of average pooling layers and fully connected layers. The generated global descriptors are then subjected to softmax to obtain probabilities, which are used to compute cross-entropy loss and generate Top-K routing signals. The USMoE module includes modality-specific experts, collaborative experts, and shared experts. Modality-specific experts are used to model the unique features of each modality, collaborative experts are used to capture cross-modality shared interactions, and shared experts encode modality-invariant information. Step S3: Input the mixed mini-batch training data into the network, wherein the template frame and search frame of the mixed mini-batch training data are converted into RGB tokens and auxiliary modality tokens through the projection layer, respectively; Step S4: Input the RGB token and auxiliary modal token into the MAEF module, dynamically calculate the modal weights, and perform token-level fusion of the two types of tokens to obtain the initial fusion features; Step S5: Input the initial fusion features into the UTD module to generate a guiding routing signal for identifying the current sample modality type; Step S6: Input the initial fused features into the USMoE module, calculate the score based on the guiding routing signal using the Top-K routing strategy, and aggregate the subsets of modality-specific experts, collaborative experts, and shared experts after adaptive activation. Then, obtain the final feature representation through a self-attention mechanism. Step S7: Input the final feature representation into the decoder, and output the target classification score and bounding box coordinates to obtain the tracking result.

[0010] Furthermore, step S4 specifically includes: The global features of the RGB token and the auxiliary modality token are calculated separately by average pooling, concatenated and input into the fully connected layer to generate a normalized modality weight vector. The RGB tokens and auxiliary modal tokens are weighted and fused based on the modal weight vector to obtain the initial fused features:

[0011] in, For RGB weight vectors, For auxiliary modal weight vectors, For RGB tokens, For auxiliary modal tokens; By sparsely activating the corresponding experts through the expert refinement module, the obtained features are refined to obtain the refined initial feature fusion representation.

[0012] Furthermore, step S5 specifically includes: The final fused representation is input to the average pooling and classifier to generate a global descriptor, which is then used to obtain the prediction task index. Use the prediction task index as a guiding routing signal.

[0013] Furthermore, step S6 specifically includes: The initial fused features are input into the sparse hybrid expert module. Based on the fused global representation, the Top-K experts are selected and combined to generate the final fused representation.

[0014] in, K The total number of experts, For the first i One expert, Set its routing weight.

[0015] Furthermore, step S7 specifically includes: Based on the guiding routing signal generated by the UTD module, the Top-K routing strategy is used to calculate the score of each expert for the current input feature:

[0016] in, These are learnable weight parameters. Input the features of the b-th mini-batch training sample; Adaptively activate the K experts with the highest scores based on each expert's rating of the current input feature to obtain aggregated feature representations of modality-specific, collaborative, and shared expert outputs:

[0017]

[0018]

[0019] in, Output for modality-specific experts Output for collaborative experts To share expert output, The first route weight, As the second route weight, For the mode-specific expert assigned to the current mode, To assign collaborative experts to the current modality, Generate modality-independent shared knowledge for shared experts; For each mini-batch sample, the final feature representation is obtained as follows: .

[0020] Furthermore, the total loss function is:

[0021] in, For classifying losses, For bounding box regression loss, For generalized IoU loss, To predict cross-entropy loss for the task, For load balancing losses, Weights for bounding box regression loss. For generalized IoU loss weights, Weighting for load balancing loss; The task prediction cross-entropy loss is:

[0022] in, b For the current sample being processed, B The total number of samples to be processed. For modal prediction probabilities, For real modal indexing The load balancing loss is:

[0023] in, Let b be the probability of assigning sample b to expert e. E This represents the total number of experts.

[0024] Furthermore, the method also includes: Execution training and reasoning: During the training phase, according to the total loss function End-to-end training was performed, freezing all parameters of the backbone network and optimizing only the parameters of the MAEF, UTD, and USMoE modules to obtain a well-trained hybrid expert model. During the inference phase, the RGB template, search region, and target auxiliary modality are input into the trained hybrid expert model. After projection, MAEF fusion, and Transformer+USMoE feature processing, the UTD module automatically identifies the modality type and guides the expert route. Finally, the tracking result is output through the bounding box header.

[0025] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, the unified multimodal visual tracking method based on the hybrid expert model described above, in a first aspect, the method uses the MAEF module and adaptive weight generation to dynamically balance the contributions of heterogeneous modalities, effectively solving the fusion problem caused by modal differences and improving the tracking robustness in complex scenarios.

[0026] Secondly, this method, with the help of the USMoE module and the UTD task-aware routing strategy, achieves the decoupling and dynamic fusion of modality-specific, cooperative and shared representations, which significantly alleviates modality interference and balances structural uniformity and functional specificity within a unified parameter space.

[0027] Thirdly, this method adopts a hybrid mini-batch joint training strategy, which reduces the dependence on large-scale single-modal data. The model can adaptively adapt to multiple modal combinations and has stronger scalability and generalization ability.

[0028] Fourthly, this method employs sparse expert activation and lightweight module design, which improves tracking accuracy while ensuring computational efficiency. It can be directly integrated into existing RGB-based trackers without large-scale reconstruction. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0030] Figure 1 The diagram illustrates the steps of a unified multimodal visual tracking method based on a hybrid expert model in an exemplary embodiment of this disclosure. Figure 2 A flowchart illustrating the unified multimodal visual tracking method based on a hybrid expert model in an exemplary embodiment of this disclosure is shown. Figure 3 This diagram illustrates the structure of a target tracking framework based on a hybrid expert model in an exemplary embodiment of this disclosure. Figure 4 This illustrates a specific implementation process of the unified multimodal visual tracking method in an exemplary embodiment of this disclosure; Figure 5 The visualization results of this application and various comparison algorithms on a multimodal dataset are shown in exemplary embodiments of this disclosure. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0032] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0033] This example implementation provides a unified multimodal visual tracking method based on a hybrid expert model. (Reference) Figure 1 As shown, this unified multimodal visual tracking method based on a hybrid expert model may include: Step S1: Construct mixed mini-batch training data, where each mini-batch sample contains pure RGB data, RGB-T data, RGB-D data, and RGB-E data; Step S2: Construct a target tracking framework based on a hybrid expert model. This framework includes a MAEF module, a UTD module, a backbone network, and a decoder. The backbone network comprises multiple Transformer blocks containing USMoE modules and self-attention mechanisms. The MAEF module includes average pooling layers, fully connected layers, and a feature refinement expert module. Pooling layers are used to compute global feature representations, fully connected layers are used to generate modality fusion weights, and the feature refinement expert module is used to refine the weighted fusion features. The UTD module includes a classifier composed of average pooling layers and fully connected layers. The generated global descriptors are then subjected to softmax to obtain probabilities, which are used to compute cross-entropy loss and generate Top-K routing signals. The USMoE module includes modality-specific experts, collaborative experts, and shared experts. Modality-specific experts are used to model the unique features of each modality, collaborative experts are used to capture cross-modality shared interactions, and shared experts encode modality-invariant information. Step S3: Input the mixed mini-batch training data into the network, wherein the template frame and search frame of the mixed mini-batch training data are converted into RGB tokens and auxiliary modality tokens through the projection layer, respectively; Step S4: Input the RGB token and auxiliary modal token into the MAEF module, dynamically calculate the modal weights, and perform token-level fusion of the two types of tokens to obtain the initial fusion features; Step S5: Input the initial fusion features into the UTD module to generate a guiding routing signal for identifying the current sample modality type; Step S6: Input the initial fused features into the USMoE module, calculate the score based on the guiding routing signal using the Top-K routing strategy, and aggregate the subsets of modality-specific experts, collaborative experts, and shared experts after adaptive activation. Then, obtain the final feature representation through a self-attention mechanism. Step S7: Input the final feature representation into the decoder, and output the target classification score and bounding box coordinates to obtain the tracking result.

[0034] The unified multimodal visual tracking method based on the above-mentioned hybrid expert model has the following advantages: First, by using the MAEF module and adaptive weight generation, the method dynamically balances the contributions of heterogeneous modalities, effectively solving the fusion problem caused by modal differences and improving the tracking robustness in complex scenarios.

[0035] Secondly, this method, with the help of the USMoE module and the UTD task-aware routing strategy, achieves the decoupling and dynamic fusion of modality-specific, cooperative and shared representations, which significantly alleviates modality interference and balances structural uniformity and functional specificity within a unified parameter space.

[0036] Thirdly, this method adopts a hybrid mini-batch joint training strategy, which reduces the dependence on large-scale single-modal data. The model can adaptively adapt to multiple modal combinations and has stronger scalability and generalization ability.

[0037] Fourthly, this method employs sparse expert activation and lightweight module design, which improves tracking accuracy while ensuring computational efficiency. It can be directly integrated into existing RGB-based trackers without large-scale reconstruction.

[0038] Below, we will refer to Figures 1 to 5 The steps of the unified multimodal visual tracking method based on a hybrid expert model described in this example implementation will be explained in more detail.

[0039] In one embodiment, the basic flowchart of multimodal target tracking in this application is as follows: Figure 2 As shown, the specific steps are as follows: Step S1: Define the multimodal visual tracking task and input format. Given a video sequence ,in RGB frames For auxiliary modal frames, where These correspond to thermal imaging (T), depth (D), and event (E) data, respectively. The tracker receives the initial bounding box annotations for the first frame. The goal is to predict the bounding box of the target in subsequent frames. .

[0040] Construct hybrid mini-batch training data. During the training phase, each mini-batch sample simultaneously contains four classes of data: pure RGB, RGB-T, RGB-D, and RGB-E, enabling the model to learn both cross-modal commonality and modality-specific discriminability under a single network parameter θ. The unified tracking function is expressed as: (1) The same parameter θ is jointly optimized through heterogeneous mode pairs. Step S2: Construct a target tracking framework based on a hybrid expert model; such as Figure 3 As shown, the target tracking framework includes a MAEF module, a UTD module, a backbone network, and a decoder. The backbone network comprises multiple Transformer blocks containing USMoE modules and self-attention mechanisms. The MAEF module includes average pooling layers, fully connected layers, and a feature refinement expert module. The pooling layers are used to compute global feature representations, the fully connected layers are used to generate modality fusion weights, and the feature refinement expert module is used to refine the weighted fused features. The UTD module includes a classifier composed of average pooling layers and fully connected layers. The generated global descriptors are subjected to softmax to obtain probabilities, which are used to compute cross-entropy loss and generate Top-K routing signals. The USMoE module includes modality-specific experts, collaborative experts, and shared experts. Modality-specific experts are used to model the unique features of each modality, collaborative experts are used to capture cross-modality shared interactions, and shared experts encode modality-invariant information.

[0041] Step S3: Feature Projection and Modality Separation. The search frames and template frames for RGB and auxiliary modalities are converted into patch embeddings through a lightweight projection layer. The generated tokens are divided into RGB tokens. and auxiliary modal tokens Two streams are used to achieve preliminary separation and characterization of modal features.

[0042] Steps S4 to S6: Modal-Aware Expert Fusion (MAEF). The MAEF module dynamically balances the contributions of RGB and auxiliary modalities. First, global features of the two types of tokens are calculated using average pooling. These features are then concatenated and input into two fully connected layers to generate a normalized weight vector. This characterizes the relative importance of each modality in the current sample; then, initial fused features are obtained through token-level fusion. (2) Finally, the Sparse Hybrid Expert (MoE) module selects the Top-K experts based on the fused global representation and generates the final fused representation through weighted combination: (3) in For the i-th expert, Set its routing weight.

[0043] The Universal Task Decoder (UTD) generates guiding routing signals. UTD is introduced to achieve unified supervision and task-aware routing for mixed-modality batches. The fused feature representation output from MAEF is input to average pooling and an MLP classifier to generate a global descriptor, resulting in the predicted task index. As a guiding routing signal, it is used to automatically identify the mode type and select the corresponding mode-specific router. During the UTD training phase, prediction is supervised by a task using cross-entropy loss. (4) in, It is the modal prediction probability. It is a true modality index.

[0044] The Transformer backbone network is integrated with Unified-Specialized Mixture of Experts (USMoE). In the backbone network, each Transformer block embeds a USMoE module, which contains three types of experts: modality-specific experts (preserving modality-specific features), collaborative experts (modeling cross-modal interactions), and shared experts (capturing modality-invariant knowledge). For the b-th mini-batch training sample feature input obtained from MAEF... Based on the guiding routing signal generated by UTD, a score is calculated using the Top-K routing strategy, adaptively activating a subset containing three types of experts. The formula for calculating the score using the Top-K strategy is: (5) in, These are learnable weight parameters.

[0045] In the activated subset of experts, the aggregated feature representations of the three types of expert outputs are as follows: (6) (7) (8) in, Output for modality-specific experts Output for collaborative experts To share expert output, , For routing weights, , This indicates the expert assigned to the current modality. This represents the generation of modality-independent knowledge.

[0046] Finally, for each mini-batch of samples, the final feature representation is as follows: (9) Meanwhile, to prevent expert model collapse, load balancing loss is used to supervise the predicted route distribution: (10) in, Let b be the probability of assigning sample b to expert e.

[0047] Step S7: Calculate the multi-task training loss function for backpropagation. The loss function used in this application is derived from the multi-task training objective, including classification loss ( ), bounding box regression loss ( ), generalized IoU loss ( ), task prediction cross-entropy loss ( ) and load balancing losses ( The total loss function is defined as: (11) Decoder classification and regression. The final feature representation is input to the decoder, and the target classification score and bounding box coordinates are output through the bounding box header. This aspect can be adapted to three tracking scenarios: RGB-T, RGB-D, and RGB-E, with only one training iteration.

[0048] In one specific embodiment, this application proposes a unified multimodal visual tracking method based on a hybrid expert mechanism. The implementation steps are described in detail below with specific experimental configurations and datasets, but the technical scope of this application is not limited to the content described therein.

[0049] The specific implementation process is as follows: Figure 4 As shown: Define the basic model configuration. Build the USTrack framework based on four representative RGB single-modal trackers (OSTrack, SeqTrack, ODTrack, and AQATrack), use ViT-B as the backbone network, embed USMoE modules in each Transformer block, and deploy MAEF modules in front of the Transformer backbone network to unify and integrate cross-modal commonality and modality-specific discriminative modeling.

[0050] Dataset and training environment were constructed. Three major multimodal benchmark datasets were used jointly for training: the LaSHeR dataset for RGB-T tracking, the DepthTrack dataset for RGB-D tracking, and the VisEvent dataset for RGB-E tracking, allowing the model to encounter heterogeneous modalities during a unified optimization process. The AdamW optimizer (with weight decay of 10) was used during training. 4) The model is trained for 60 epochs with an initial learning rate of 1×10⁻⁶. 4. The learning rate decays by a factor of 10 after 48 epochs; the batch size is set to 32, and the template and search image resolutions are 128×128 and 256×256, respectively.

[0051] Constructing a hybrid mini-batch mode. This application constructs a joint mini-batch training mode based on a hybrid dataset. Each training mini-batch contains pure RGB, RGB-T, RGB-D, and RGB-E, enabling the model to learn modality-specific and cross-modal representations simultaneously while sharing parameters.

[0052] Configure the parameters of each module of the model. In the MAEF module, the modality weight generation module adopts two fully connected layers, and K is selected as an appropriate value in the Top-K routing of sparse hybrid experts to balance efficiency and specificity. In the USMoE module, each modality is configured with a dedicated modality-specific expert, and the number of collaborative experts and shared experts is adapted according to the backbone network capacity. The UTD module adopts a simple MLP classifier to output the predicted probabilities of the three modalities.

[0053] Set the weights of the loss function. For the loss function given by formula (7), set the weights of the bounding box regression loss based on experience and hyperparameter experimental results. Generalized IoU loss weights Load balancing loss weight The remaining loss weights are set to 1.

[0054] Perform training and inference. During the training phase, follow the multi-task loss function. End-to-end training is performed, freezing all parameters of the backbone network and optimizing only the parameters of the MAEF and USMoE modules. During the inference phase, the RGB template, search region, and target auxiliary modalities (thermal imaging / depth / event) are input into the model. After projection, MAEF fusion, and Transformer+USMoE feature processing, the UTD automatically identifies the modal type and guides expert routing. Finally, the tracking results are output through the bounding box header.

[0055] This application evaluates RGB-T tracking performance on the LaSHeR and RGBT234 datasets, RGB-D tracking on the DepthTrack and VOTRGBD2022 datasets, and RGB-E tracking on the VisEvent dataset. All evaluations use standard evaluation protocols, and the specific results and comparisons with other algorithms are shown in Table 1.

[0056] Table 1. Experimental results of this application and various comparison algorithms on multimodal datasets.

[0057] like Figure 5 The image shows the visualization results of this application and various comparison algorithms on a multimodal dataset; among them, Figure 5 a, b, and c in the figure represent the visualized tracking results of the algorithm in this application on the three multimodal datasets Lahser, DepthTrack, and VisEvent, respectively, as well as the cross-union ratio (CUP) curves of the algorithm in this application and other algorithms on the frame sequence.

[0058] As can be seen, in the RGB-T sequence, this application can still maintain robust tracking performance under illumination changes and thermal noise interference; when a certain mode becomes unreliable, it can avoid the phenomenon of confidence collapse.

[0059] On the RGB-D benchmark, this application exhibits stronger robustness to depth ambiguity and background clutter, thereby significantly reducing target drift during long-sequence tracking.

[0060] In RGB-E tracking tasks, since event data is usually sparse and noisy, this application is still able to generate significantly more stable confidence change trajectories and achieve more accurate target localization.

[0061] Overall, USTrack demonstrates excellent tracking performance across all modal datasets. Specifically, this application achieves the highest PR and SR on LaSHeR, the best MPR and MSR on RGBT234, and leads on DepthTrack and VisEvent. In RGB-T tracking benchmarks, this application shows a significant advantage over RGB-based trackers, especially when extended to multimodal inputs and RGB-T-specific methods. For example, on LaSHeR, this application improves the PR to 72.0 and the SR to 55.9, outperforming powerful RGB-T trackers such as XTrack and CMDTrack. A similar trend is observed on RGBT234, where this application achieves the highest MPR of 90.3, indicating greater robustness and tracking stability under thermal changes. In RGB-D tracking benchmarks, this application ranks among the top. On DepthTrack, this application achieves the best F-score of 63.8. In the widely considered challenging VOT-RGBD2022 benchmark, this application achieved the highest EAO score of 75.0 and robustness score of 91.8, demonstrating strong resilience against depth noise and sudden appearance changes. This application also maintains its advantage in the RGB-E tracking benchmark. On VisEvent, this application achieves the best MPR and MSR, outperforming comparative dedicated RGB and RGB-E based trackers. This result is particularly significant because event data exhibits sparse and asynchronous characteristics, often hindering unified modeling.

[0062] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0063] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A unified multimodal visual tracking method based on a hybrid expert model, characterized in that, include: Step S1: Construct mixed mini-batch training data, where each mini-batch sample contains pure RGB data, RGB-T data, RGB-D data, and RGB-E data; Step S2: Construct a target tracking framework based on a hybrid expert model. This framework includes a MAEF module, a UTD module, a backbone network, and a decoder. The backbone network comprises multiple Transformer blocks containing USMoE modules and self-attention mechanisms. The MAEF module includes average pooling layers, fully connected layers, and a feature refinement expert module. Pooling layers are used to compute global feature representations, fully connected layers are used to generate modality fusion weights, and the feature refinement expert module is used to refine the weighted fusion features. The UTD module includes a classifier composed of average pooling layers and fully connected layers. The generated global descriptors are then subjected to softmax to obtain probabilities, which are used to compute cross-entropy loss and generate Top-K routing signals. The USMoE module includes modality-specific experts, collaborative experts, and shared experts. Modality-specific experts are used to model the unique features of each modality, collaborative experts are used to capture cross-modality shared interactions, and shared experts encode modality-invariant information. Step S3: Input the mixed mini-batch training data into the network, wherein the template frame and search frame of the mixed mini-batch training data are converted into RGB tokens and auxiliary modality tokens through the projection layer, respectively; Step S4: Input the RGB token and auxiliary modal token into the MAEF module, dynamically calculate the modal weights, and perform token-level fusion of the two types of tokens to obtain the initial fusion features; Step S5: Input the initial fusion features into the UTD module to generate a guiding routing signal for identifying the current sample modality type; Step S6: Input the initial fused features into the USMoE module, calculate the score based on the guiding routing signal using the Top-K routing strategy, and aggregate the subsets of modality-specific experts, collaborative experts, and shared experts after adaptive activation. Then, obtain the final feature representation through a self-attention mechanism. Step S7: Input the final feature representation into the decoder, and output the target classification score and bounding box coordinates to obtain the tracking result.

2. The unified multimodal visual tracking method based on a hybrid expert model according to claim 1, characterized in that, Step S4 specifically includes: The global features of the RGB token and the auxiliary modality token are calculated separately by average pooling, concatenated and input into the fully connected layer to generate a normalized modality weight vector. The RGB tokens and auxiliary modal tokens are weighted and fused based on the modal weight vector to obtain the initial fused features: in, For RGB weight vectors, For auxiliary modal weight vectors, For RGB tokens, For auxiliary modal tokens; By sparsely activating the corresponding experts through the expert refinement module, the obtained features are refined to obtain the refined initial feature fusion representation.

3. The unified multimodal visual tracking method based on a hybrid expert model according to claim 2, characterized in that, Step S5 specifically includes: The final fused representation is input to the average pooling and classifier to generate a global descriptor, which is then used to obtain the prediction task index. Use the prediction task index as a guiding routing signal.

4. The unified multimodal visual tracking method based on a hybrid expert model according to claim 3, characterized in that, Step S6 specifically includes: The initial fused features are input into the sparse hybrid expert module. Based on the fused global representation, the Top-K experts are selected and combined to generate the final fused representation. in, K The total number of experts, For the first i One expert, Set its routing weight.

5. The unified multimodal visual tracking method based on a hybrid expert model according to claim 4, characterized in that, Step S7 specifically includes: Based on the guiding routing signal generated by the UTD module, the Top-K routing strategy is used to calculate the score of each expert for the current input feature: in, These are learnable weight parameters. Input the features of the b-th mini-batch training sample; Adaptively activate the K experts with the highest scores based on each expert's rating of the current input feature to obtain aggregated feature representations of modality-specific, collaborative, and shared expert outputs: in, Output for modality-specific experts Output for collaborative experts To share expert output, The first route weight, As the second route weight, For the mode-specific expert assigned to the current mode, To assign collaborative experts to the current modality, Generate modality-independent shared knowledge for shared experts; For each mini-batch of samples, the final feature representation is obtained as follows: 。 6. The unified multimodal visual tracking method based on a hybrid expert model according to claim 5, characterized in that, The total loss function is: in, For classifying losses, For bounding box regression loss, For generalized IoU loss, To predict cross-entropy loss for the task, For load balancing losses, Weights for bounding box regression loss. For generalized IoU loss weights, Weighting for load balancing loss; The task prediction cross-entropy loss is: in, b For the current sample being processed, B The total number of samples to be processed. For modal prediction probabilities, For real modal indexing The load balancing loss is: in, Let b be the probability of assigning sample b to expert e. E This represents the total number of experts.

7. The unified multimodal visual tracking method based on a hybrid expert model according to claim 6, characterized in that, The method also includes: Execution training and reasoning: During the training phase, according to the total loss function End-to-end training was performed, freezing all parameters of the backbone network and optimizing only the parameters of the MAEF, UTD, and USMoE modules to obtain a well-trained hybrid expert model. During the inference phase, the RGB template, search region, and target auxiliary modality are input into the trained hybrid expert model. After projection, MAEF fusion, and Transformer+USMoE feature processing, the UTD module automatically identifies the modality type and guides the expert route. Finally, the tracking result is output through the bounding box header.

Images