An arbitrary modality target tracking method, device, equipment and storage medium

Abstract

The application discloses an arbitrary modality target tracking method and device, equipment and storage medium, and belongs to the technical field of computer vision. The method comprises the following steps: obtaining a trained arbitrary modality target tracking model; inputting a token sequence of a template image based on each modality of a current video frame, a token sequence of a search image and a time sequence token of each modality into a visual coding module shared by modes to perform feature extraction processing, and obtaining visual features of each modality; inputting the visual features of each modality into a modality perception interaction module to perform feature interaction processing, and generating unified features; inputting the unified features and multi-modality prompt features into a context understanding module to perform bidirectional information propagation processing, and generating refined unified features; and inputting the refined unified features into a prediction module to perform target detection processing, and generating target tracking information. The technical scheme provided by the application can realize processing of target tracking in any number and any modality combination within a single model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and storage medium for tracking targets of any modality. Background Technology

[0002] Visual target tracking aims to continuously locate specific targets in video sequences and is a fundamental task in the field of computer vision, widely used in scenarios such as autonomous driving, intelligent monitoring, and robot navigation. With the development of perception technology, target tracking has gradually evolved from relying on a single modality (such as visible light, depth, and thermal infrared) to a multimodal fusion paradigm, which improves tracking robustness in complex scenes by integrating complementary information. However, existing multimodal target tracking methods are usually designed with fixed modal combinations, such as visible light and depth, visible light and thermal infrared, visible light and event stream, and visible light and speech, etc. Each modal combination requires separate model training and storage of specific parameters, which leads to low resource utilization and complex training processes. At the same time, the mode of training separately for isolated datasets hinders cross-modal knowledge transfer and further exacerbates the risk of overfitting. In addition, existing methods generally require visible light modality as a required input and cannot flexibly handle arbitrary modal combinations that are not dominated by visible light. Furthermore, in practical applications, multimodal data is often incomplete due to sensor failure, synchronization problems, or signal loss. Existing task-specific models lack robustness in handling modal missing or quality degradation and have poor adaptability to modal missing data. Therefore, a more reliable solution is needed. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide an arbitrary modal target tracking method, device, equipment and storage medium, which can flexibly handle target tracking of any number and type of modal combinations within a single model, and improve the accuracy, robustness and efficiency of target tracking in complex scenarios.

[0004] To achieve the above objectives, the present invention employs the following technical solution:

[0005] On one hand, the present invention provides an arbitrary modal target tracking method, the method comprising:

[0006] The system acquires the template input image and search input image of the current video frame, as well as a trained arbitrary modality target tracking model. The search input image includes a search image of at least one modality, and the template input image includes a template image of at least one modality. The arbitrary modality target tracking model includes a visual encoding module, a modality-aware interaction module, a context understanding module, and a prediction module.

[0007] The spatiotemporal token of each modality is input into the modality-shared visual encoding module for feature extraction processing to obtain the visual features of each modality; the spatiotemporal token of each modality is obtained based on the token sequence of the template image of each modality, the token sequence of the search image, and the temporal token; the temporal token is a temporal token generated from the previous video frame;

[0008] The visual features of each modality are input into the modality perception interaction module for feature interaction processing to generate unified features;

[0009] The unified feature and the multimodal cue feature are input into the context understanding module for bidirectional information propagation processing to generate a refined unified feature; the multimodal cue feature is obtained based on the acquired multimodal cue information;

[0010] The refined and unified features are input into the prediction module for target detection processing to generate target tracking information.

[0011] In some possible implementations, the modality-aware interaction module includes multiple appearance experts and a shared semantic expert.

[0012] The step of inputting the visual features of each modality into the modality-aware interaction module for feature interaction processing to generate unified features includes:

[0013] Select a target number of target appearance experts for each modality from the plurality of appearance experts;

[0014] The visual features of each modality are input to the target number of target appearance experts to perform modality-specific feature extraction processing, thereby generating the target number of specific appearance features for each modality;

[0015] The visual features of each modality are input into the shared semantic expert for cross-modal shared feature extraction processing to generate cross-modal shared semantic features for each modality.

[0016] Based on the gating weights corresponding to each target appearance expert in each modality, feature fusion processing is performed on each specific appearance feature, cross-modal shared semantic feature and visual feature of each modality to generate specific enhanced visual features for each modality.

[0017] Cross-modal temporal aggregation processing is performed on specific enhanced visual features and temporal tokens for each modality to generate the unified feature.

[0018] In some possible implementations, the modality-aware interaction module further includes a modality-aware router;

[0019] The step of selecting a target number of target appearance experts for each modality from the plurality of appearance experts includes:

[0020] The compressed visual features of each modality are input into the modality-aware router, and the routing logic value of each appearance expert corresponding to each modality is output.

[0021] A noise gating mechanism is adopted to inject learnable noise into the routing logic value of each appearance expert corresponding to each mode, thereby generating the gated routing value of each appearance expert corresponding to each mode.

[0022] The gated routing values ​​of multiple appearance experts corresponding to each modality are sorted in descending order to generate the gated routing value sorting result for each modality.

[0023] The appearance experts corresponding to the top number of gated route values ​​in the sorting results of the gated route values ​​for each modality are determined as the target number of target appearance experts for each modality.

[0024] The gating weights corresponding to each target appearance expert in each modality are determined in the following manner:

[0025] The gating routing values ​​of each target appearance expert in each modality are normalized to generate the gating weights corresponding to each target appearance expert in each modality.

[0026] In some possible implementations, the cross-modal temporal aggregation processing of specific enhanced visual features and temporal tokens for each modality to generate the unified feature includes:

[0027] Through a multi-head cross-attention mechanism, attention interaction processing is performed on specific enhanced visual features and temporal tokens of each modality to generate visually enhanced temporal tokens for each modality.

[0028] The visual enhancement timing token of each modality is subjected to nonlinear transformation processing to generate a refined timing token for each modality, so as to use the refined timing token of each modality as the timing token of the next video frame.

[0029] Determine the affinity matrix between the specific enhanced visual features of each modality and the refined temporal token;

[0030] Based on the affinity matrix of each modality, the specific enhanced visual features of each modality are weighted and fused to generate the unified feature.

[0031] In some possible implementations, the context understanding module includes Asymmetric bidirectional attention layer;

[0032] The step of inputting the unified feature and multimodal cue feature into the context understanding module for bidirectional information propagation processing to generate refined unified features includes:

[0033] The unified feature and the multimodal cue feature are input into the current layer asymmetric bidirectional attention layer. Based on the multi-head self-attention mechanism, the multimodal cue feature is subjected to context aggregation processing to generate cue interaction features.

[0034] Based on the multi-head cross-attention mechanism, the cue interaction features and the mask features obtained from the mask memory are subjected to cross-context aggregation processing to generate initial memory-enhanced cue features;

[0035] The initial memory enhancement cue features are subjected to nonlinear transformation processing to generate memory enhancement cue features;

[0036] Based on a multi-head cross-attention mechanism, cross-context aggregation processing is performed on the unified feature and the memory-enhancing cue feature to generate a cue-enhancing unified feature.

[0037] The cue enhancement unified feature output from the current asymmetric bidirectional attention layer, along with the multimodal cue feature and the mask feature, are input into the next asymmetric bidirectional attention layer. The steps of performing context aggregation processing on the multimodal cue feature based on a multi-head self-attention mechanism to generate cue interaction features, and then performing cross-context aggregation processing on the unified feature and the memory-enhanced cue feature based on a multi-head cross-attention mechanism to generate cue enhancement unified features, are repeated until the process is complete. Layer-by-layer processing of the asymmetric bidirectional attention layer generates refined and unified features.

[0038] In some possible implementations, the method further includes:

[0039] Based on the target tracking information, a target tracking bounding box is determined, wherein the target tracking bounding box includes a bounding box center, a bounding box width, and a bounding box height;

[0040] The first initial Gaussian variance and the second initial Gaussian variance are determined based on the bounding box width and the bounding box height, respectively.

[0041] Based on the refined unified features, the variance estimation network predicts the first and second offsets.

[0042] A Gaussian pseudomask is determined based on the bounding box center, the first initialized Gaussian variance, the second initialized Gaussian variance, the first offset, and the second offset.

[0043] The Gaussian pseudo-mask is encoded to generate initial mask features;

[0044] The initial mask features and the refined unified features are subjected to feature fusion processing to generate refined mask features, which are then stored in the mask memory so that the refined mask features can be used as the mask features for the next video frame.

[0045] In some possible implementations, the trained arbitrary modal target tracking model is trained based on a multi-task loss, wherein the multi-task loss is based on focus loss, The loss, generalized intersection-union ratio loss, and equilibrium loss are determined as shown in the following formula:

[0046] ;

[0047] in, Indicates multi-tasking loss. Indicates focal loss. This indicates the generalized intersection and comparison loss. express loss, Indicates the loss of equilibrium; , , and These represent the first hyperparameter, the second hyperparameter, the third hyperparameter, and the fourth hyperparameter, respectively.

[0048] The balancing loss is determined based on the importance score and load of each apparent expert for each mode, as shown in the following formula:

[0049] ;

[0050] in, Representing modes The The importance score of an expert in appearance, Representing modes The The load of an appearance expert; Representing modes The total number of appearance experts; It represents the ratio of the standard deviation to the mean.

[0051] On the other hand, the present invention provides an arbitrary modal target tracking device, the device comprising:

[0052] The data acquisition unit is configured to acquire the template input image and search input image of the current video frame, and to acquire a trained arbitrary modality target tracking model; the search input image includes a search image of at least one modality, and the template input image includes a template image of at least one modality; the arbitrary modality target tracking model includes a visual encoding module, a modality-aware interaction module, a context understanding module, and a prediction module;

[0053] The target tracking unit is configured to input the spatiotemporal tokens of each modality into a modality-shared visual encoding module for feature extraction processing to obtain visual features for each modality. The spatiotemporal tokens for each modality are obtained based on the token sequence of the template image, the token sequence of the search image, and the temporal tokens. The temporal tokens are generated from the previous video frame. The visual features of each modality are input into the modality perception interaction module for feature interaction processing to generate unified features. The unified features and multimodal cue features are input into the context understanding module for bidirectional information propagation processing to generate refined unified features. The multimodal cue features are obtained based on acquired multimodal cue information. The refined unified features are input into the prediction module for target detection processing to generate target tracking information.

[0054] On the other hand, an electronic device is provided, the device including a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the arbitrary modal target tracking method as described above.

[0055] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction and at least one program are stored therein, the at least one instruction and the at least one program being loaded and executed by a processor to implement the arbitrary modal target tracking method as described above.

[0056] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0057] In this invention, the template input image and search input image of the current video frame are acquired, along with a trained arbitrary modality target tracking model. The search input image includes a search image of at least one modality, and the template input image includes a template image of at least one modality. The arbitrary modality target tracking model includes a visual encoding module, a modality-aware interaction module, a context understanding module, and a prediction module. The spatiotemporal token of each modality is input into the modality-shared visual encoding module for feature extraction processing to obtain the visual features of each modality. The spatiotemporal token of each modality is obtained based on the token sequence of the template image, the token sequence of the search image, and the temporal token of each modality. The temporal token is a temporal token generated from the previous video frame. Then, the visual features of each modality are input into the modality-aware interaction module for feature interaction processing to generate... Unified features enable fine-grained decoupling and semantic alignment of heterogeneous modal features, achieving excellent performance in both complete and missing modal scenarios. The unified features and multimodal cue features are then input into the context understanding module for bidirectional information propagation, generating refined unified features. These features establish accurate spatial correspondences to effectively address scale variations, motion blur, and interference from similar objects, thereby improving the accuracy and robustness of target tracking in complex scenarios. The multimodal cue features are derived from the acquired multimodal cue information. The refined unified features are then input into the prediction module for target detection, generating target tracking information. This allows for flexible handling of target tracking with any number and type of modal combinations within a single model, improving the accuracy, robustness, and efficiency of target tracking in complex scenarios. Attached Figure Description

[0058] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating an arbitrary modal target tracking method provided in an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of an arbitrary modal target tracking method provided in an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of a context understanding module processing procedure provided in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the structure of an arbitrary modal target tracking device provided in an embodiment of the present invention. Detailed Implementation

[0063] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0064] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0065] In this embodiment of the invention, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0066] Various exemplary embodiments, features, and aspects of the present invention will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0067] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0068] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0069] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without certain specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the spirit of the invention.

[0070] Figure 1 This is a flowchart illustrating an arbitrary modal target tracking method provided by an embodiment of the present invention. This specification provides the method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive methods, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the method can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 1 As shown, the above method may include:

[0071] S101: Obtain the template input image and search input image of the current video frame, and obtain the trained arbitrary modality target tracking model; the search input image includes a search image of at least one modality, and the template input image includes a template image of at least one modality; the arbitrary modality target tracking model includes a visual encoding module, a modality-aware interaction module, a context understanding module, and a prediction module;

[0072] In one specific embodiment, the arbitrary modal target tracking method of the present invention is used in a visual target tracking task, which is to continuously locate the target of interest in subsequent video frames and determine the position and scale state of the target of interest, given an initial target template.

[0073] Optionally, the current video frame can be a template image of the video frame currently being tracked in the video sequence; the search image of at least one modality can be a large image centered on the tracking target position of at least one modality; optionally, the search image of at least one modality can be a video frame image of at least one modality to be tracked; the template image of at least one modality can be an image of the tracking target area extracted from the image of at least one modality according to a preset ratio; optionally, at least one modality can include, but is not limited to, visible light, grayscale, depth, thermal infrared, and event stream; optionally, the preset ratio can be set according to the actual application.

[0074] Specifically, at the current moment The template input image and search input image of the current video frame are obtained; the template input image may include template images of at least one modality. The search input image can include at least one modality of search image. ,in, This represents any mode, including but not limited to visible light, grayscale, depth, thermal infrared, event stream, etc. This represents the channel dimension of the template image. Indicates the height of the template image. Indicates the width of the template image; Indicates the channel dimension of the search image. Indicates the height of the search image. This indicates the width of the search image.

[0075] In one specific embodiment, the template input image and the search input image are input into a pre-trained arbitrary modal target tracking model for target detection processing to generate target tracking information. Optionally, the images input into the pre-trained arbitrary modal target tracking model can be template and search images corresponding to any single modality, such as template and search images corresponding to visible light, or template and search images corresponding to grayscale; or template and search images corresponding to any combination of dual modal types, such as template and search images corresponding to visible light and grayscale, or template and search images corresponding to thermal infrared and grayscale; or template and search images corresponding to any combination of multimodal types, such as template and search images corresponding to visible light, thermal infrared, and grayscale, or template and search images corresponding to thermal infrared, grayscale, and depth.

[0076] S102: Input the spatiotemporal token of each modality into the modality-shared visual encoding module for feature extraction processing to obtain the visual features of each modality; the spatiotemporal token of each modality is obtained based on the token sequence of the template image of each modality, the token sequence of the search image and the temporal token, and the temporal token is the temporal token generated from the previous video frame.

[0077] In one specific embodiment, a three-stage downsampling network is used to convert the template input image and the search input image into token sequences of the template input image and the search input image, respectively. The token sequence of the template input image includes token sequences of template images of at least one modality, and the token sequence of the search input image includes token sequences of search images of at least one modality. Based on the token sequence of the template image, the token sequence of the search image, and the temporal token of each modality, a spatiotemporal token of each modality is generated. Optionally, the spatiotemporal token of each modality can be a concatenated token obtained by concatenating the token sequence of the template image of each modality with the token sequence of the search image of each modality, and a token obtained by concatenating the temporal token generated in the previous video frame with the token sequence of the previous video frame. The spatiotemporal token of each modality is a complete representation containing spatiotemporal information.

[0078] Specifically, the above-mentioned conversion of the template input image and the search input image into token sequences of the template input image and the search input image through a three-stage downsampling network may include: performing downsampling and primary feature extraction processing on the template input image and the search input image to generate primary template features and primary search features; then performing downsampling and intermediate feature extraction processing on the primary template features and the primary search features to generate intermediate template features and intermediate search features; finally, performing downsampling and advanced feature extraction processing on the intermediate template features and the intermediate search features to generate token sequences of the template input image and the search input image.

[0079] Optionally, generating a spatiotemporal token for each modality based on the token sequence of the template image, the token sequence of the search image, and the temporal token for each modality may include: concatenating the token sequence of the template image and the token sequence of the search image for each modality along the token dimension to obtain a concatenated token for each modality; and concatenating the concatenated token for each modality with the temporal token generated in the previous video frame to obtain a spatiotemporal token for each modality.

[0080] In one specific embodiment, a three-stage downsampling network is used to convert the template input image and the search input image into token sequences of the template input image and the search input image, respectively; the token sequence of the template input image may include token sequences of at least one modality of the template image. The token sequence for searching the input image can include at least one modality of the token sequence for searching the image. ,in, and These represent the number of tokens for the template image and the number of tokens for the search image in each modality, respectively. This indicates the feature dimension; specifically, the feature dimension can be set to 512.

[0081] Along the token dimension, the token sequence of the template image for each modality and the token sequence of the search image for each modality are concatenated to obtain the concatenated token for each modality. As shown in the following formula:

[0082] ;

[0083] The concatenation token of each modality is combined with the timing token generated from the corresponding previous video frame. By concatenating the tokens, we can obtain the spacetime tokens for each modality. As shown in the following formula:

[0084] ,in, This indicates the number of tokens for each mode of timing tokens; specifically, the number of tokens for timing tokens is set to 1.

[0085] In one specific embodiment, the visual encoding module is used to perform feature extraction processing on the spatiotemporal tokens of each modality. The visual encoding module can be configured according to actual application requirements; specifically, the visual encoding module can be a visual encoder implemented based on the HiViT-B architecture. Optionally, by using a modality-shared visual encoder to extract features from the spatiotemporal tokens of each modality, unified high-dimensional features can be extracted from heterogeneous modal inputs. Specifically, the spatiotemporal tokens of each modality... Input modality sharing visual encoder Feature extraction is performed to obtain the visual features of each modality. As shown in the following formula:

[0086] .

[0087] S103: Input the visual features of each modality into the modality perception interaction module for feature interaction processing to generate unified features;

[0088] In one specific embodiment, the modality-aware interaction module is built on a hybrid expert model to perform dynamic feature interaction and bridge representation differences between different modalities based on a dynamic routing mechanism, and aggregates temporal cues using a cross-modal attention mechanism to ensure spatiotemporal consistency. Figure 2 This is a schematic diagram of an arbitrary modal target tracking method provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the process of inputting the visual features of each modality into the modality-aware interaction module for processing can be seen. Indicates the number of epigenetic experts.

[0089] In an optional embodiment, the modality-aware interaction module described above may include multiple appearance experts and a shared semantic expert.

[0090] The above-mentioned input of the visual features of each modality into the modality-aware interaction module for feature interaction processing to generate unified features may include:

[0091] Select a target number of target appearance experts for each modality from a pool of appearance experts;

[0092] The visual features of each modality are input to a target number of target appearance experts to perform modality-specific feature extraction processing, generating a target number of specific appearance features for each modality;

[0093] The visual features of each modality are input into a shared semantic expert for cross-modal shared feature extraction, generating cross-modal shared semantic features for each modality;

[0094] Based on the gating weights corresponding to each target appearance expert in each modality, feature fusion processing is performed on each specific appearance feature, cross-modal shared semantic feature and visual feature of each modality to generate specific enhanced visual features for each modality.

[0095] Cross-modal temporal aggregation is performed on specific enhanced visual features and temporal tokens for each modality to generate unified features.

[0096] In one specific embodiment, multiple appearance experts are used to capture modality-specific appearance features; a shared semantic expert is used to align and enhance modality-invariant semantic features; and a temporal cue propagation module is used to integrate spatial representations with historical temporal context. Optionally, the number of appearance experts can be interpreted according to the actual application settings; specifically, the number of appearance experts can be set to 6.

[0097] In an optional embodiment, the modality-aware interaction module described above may further include a modality-aware router;

[0098] The above selection of a target number of target appearance experts for each modality from multiple appearance experts may include:

[0099] The compressed visual features of each modality are input into the modality-aware router, and the routing logic value of each appearance expert corresponding to each modality is output.

[0100] A noise gating mechanism is adopted to inject learnable noise into the routing logic value of each appearance expert corresponding to each mode, thereby generating the gated routing value of each appearance expert corresponding to each mode.

[0101] The gated routing values ​​of multiple appearance experts corresponding to each modality are sorted in descending order to generate the gated routing value sorting result for each modality.

[0102] The appearance experts corresponding to the top number of gated route values ​​in the sorting results of gated route values ​​for each modality are determined as the target appearance experts for each modality.

[0103] The gating weights for each target appearance expert in each modality are determined as follows:

[0104] The gating routing values ​​of each target appearance expert in each modality are normalized to generate the gating weights corresponding to each target appearance expert in each modality.

[0105] In one specific embodiment, the modality-aware router is used to determine the routing logical values ​​for each appearance expert; the channel-compressed visual features of each modality can be features obtained by averaging the channel dimensions of the visual features of each modality. Optionally, for each modality's visual features... Perform average pooling operation along the channel dimension Generate channel-compressed visual features for each modality; input the channel-compressed visual features for each modality into the modality-aware router. Output the routing logic value for each appearance expert corresponding to each modality. As shown in the following formula:

[0106] ;

[0107] Optionally, the routing logic value for each apparent expert corresponding to each modality can characterize the probability of each apparent expert being selected for each modality. Optionally, the modality-aware router can be implemented using a multilayer perceptron.

[0108] In one specific embodiment, a noise gating mechanism is employed to ensure diverse utilization of experts; simultaneously, during model training, a noise gating mechanism is employed to ensure diverse utilization of each expert. Optionally, the noise gating mechanism is used to route logical values ​​to each apparent expert corresponding to each modality. Inject learnable noise to generate gated routing values ​​for each appearance expert corresponding to each modality. Specifically, the visual features for each modality. A nonlinear transformation is performed to obtain the nonlinearly transformed visual features of each modality. These features are then processed using an activation function and multiplied element-wise with standard Gaussian random noise to generate learnable noise. Next, the learnable noise is injected into the routing logic value of each appearance expert corresponding to each modality to generate the gated routing value for each appearance expert, as shown in the following equation:

[0109] ;

[0110] in, This represents standard Gaussian random noise. Represents a normal distribution; This represents the activation function. This indicates element-wise multiplication. This represents a constant used to ensure numerical stability.

[0111] In one specific embodiment, a target subset of appearance experts is adaptively selected from multiple appearance experts for each modality based on a Top-K sparse selection mechanism. Specifically, the gated routing values ​​of multiple appearance experts corresponding to each modality are sorted in descending order to generate a gated routing ranking result for each modality. Based on the gated routing value ranking result for each modality, the top-ranked appearance experts with the highest gated routing values ​​are determined. An appearance expert for each modality The number of target appearance experts per modality can be selected appearance experts used to process the visual features of each modality to capture modality-specific appearance features; the number of targets can be set according to actual application requirements, specifically, the number of targets can be 2.

[0112] In a specific embodiment, the gating weights corresponding to each target appearance expert in each modality can be used to balance the contributions of each target appearance expert in each modality; the gating routing values ​​of each target appearance expert in each modality can be normalized by Softmax to generate the gating weights corresponding to each target appearance expert in each modality, as shown in the following formula:

[0113] ;

[0114] in, Representing modes The The gating weights corresponding to the target appearance experts This indicates the number of experts targeting the objective. This represents the set of indices for the selected target appearance experts. The visual features of each modality are processed only by the selected target appearance experts, achieving sparse adaptive selection, thereby avoiding activation of the entire expert pool and adapting to the features of different modalities.

[0115] In a specific embodiment, the target number of specific appearance features for each modality can be features obtained by performing modality-specific feature extraction processing on the visual features of each modality by target number of target appearance experts; the cross-modal shared semantic features for each modality can be features obtained by performing cross-modal shared feature extraction processing on the visual features of each modality by shared semantic experts.

[0116] To simultaneously capture modality-specific features and cross-modality shared features, the visual features of each modality can be input into a target number of target appearance experts and shared semantic experts for processing. Optionally, the visual features of each modality can be input into a target number of target appearance experts for modality-specific feature extraction. By learning emerging features through data-driven learning, heterogeneous modality features are decoupled to a fine-grained representation subspace, generating a target number of specific appearance features for each modality, as shown in the following equation:

[0117] ;

[0118] in, Representing modes The A specific phenotypic feature, Indicates the first A target appearance expert, implemented by a multilayer perceptron.

[0119] Optionally, while the target appearance varies significantly across modalities, the semantic information remains modally invariant. To leverage this invariance and capture cross-modal shared features, all modalities share a semantic expert to align and enhance modality-invariant semantic cues. This shared semantic expert employs a gating mechanism to emphasize salient semantics and suppress irrelevant background. The visual features of each modality are input into the shared semantic expert for cross-modal shared feature processing, generating cross-modal shared semantic features for each modality, as shown in the following equation:

[0120] ;

[0121] in, Representing modes Cross-modal shared semantic features Represents shared semantic experts;

[0122] in, ; This represents a multilayer perceptron. This represents the convolution operation. Representation layer normalization, This represents the GELU activation function. Specifically, the processing of visual features for each modality by a shared semantic expert can include: performing layer normalization on the visual features of each modality to obtain layer-normalized visual features for each modality; then performing nonlinear transformation on the layer-normalized visual features of each modality using a multilayer perceptron to obtain nonlinearly transformed visual features for each modality; performing convolution on the nonlinearly transformed visual features of each modality to obtain convolutionally transformed visual features for each modality; processing the nonlinearly transformed visual features of each modality using an activation function to obtain processed visual features for each modality; performing element-wise multiplication on the convolutionally multiplied visual features of each modality and the processed visual features of each modality to obtain element-wise multiplied visual features for each modality; and performing nonlinear transformation on the element-wise multiplied visual features of each modality using a multilayer perceptron to obtain cross-modal shared semantic features for each modality.

[0123] In a specific embodiment, the specific enhanced visual feature of each modality can be a feature obtained by fusing the specific appearance features, cross-modal shared semantic features, and visual features of each modality based on the gating weights corresponding to each target appearance expert of each modality; optionally, the above-mentioned feature fusion processing of the specific appearance features, cross-modal shared semantic features, and visual features of each modality based on the gating weights corresponding to each target appearance expert of each modality to generate the specific enhanced visual feature of each modality may include:

[0124] Based on the gating weights corresponding to each appearance expert in each modality, feature fusion processing is performed on each specific appearance feature of each modality to generate a specific fused appearance feature for each modality.

[0125] The specific fusion appearance features of each modality and the cross-modal shared semantic features are fused to generate the fusion visual features of each modality.

[0126] The visual features of each modality and the fused visual features are fused together to generate specific enhanced visual features for each modality.

[0127] Specifically, semantically guided expert fusion can be performed to integrate fine-grained appearance features from activated target appearance experts with cross-modal shared semantic features from shared semantic experts. Optionally, to effectively combine complementary features, the contributions of each selected target appearance expert are dynamically balanced based on the gating weights corresponding to each target appearance expert in each modality. Furthermore, this is fused with cross-modal semantic features of each modality, while simultaneously preserving the original feature flow through fusion with visual features of each modality, resulting in specific enhanced visual features for each modality. This yields modality-specific and enhanced visual features for each modality, as shown in the following equation:

[0128] ;

[0129] in, Representing modes Specific enhanced visual features;

[0130] in, , Indicates the feature With features The fusion process involves fusing the visual features of each modality with the fused visual features to generate specific enhanced visual features for each modality. This can include: performing layer normalization on the visual features of each modality to obtain layer-normalized visual features for each modality; performing layer normalization on the fused visual features of each modality to obtain layer-normalized fused visual features for each modality; then performing nonlinear transformation on the layer-normalized visual features of each modality using a multilayer perceptron to obtain nonlinearly transformed visual features for each modality; and finally performing nonlinear transformation on the layer-normalized fused visual features of each modality using a multilayer perceptron to obtain... The nonlinear transformation of each modality is processed to obtain the fused visual features. Then, a multilayer perceptron is used to process the nonlinear transformation of the visual features of each modality to obtain the processed visual features of each modality. The activation function is used to process the fused visual features of each modality to obtain the processed fused visual features of each modality. The processed visual features of each modality and the processed fused visual features of each modality are multiplied element-wise to obtain the element-wise multiplied features of each modality. The visual features of each modality and the element-wise multiplied features of each modality are connected by residuals to obtain the specific enhanced visual features of each modality.

[0131] In an optional embodiment, the cross-modal temporal aggregation processing of the specific enhanced visual features and temporal tokens for each modality to generate unified features may include:

[0132] Through a multi-head cross-attention mechanism, attention interaction processing is performed on specific enhanced visual features and temporal tokens of each modality to generate visually enhanced temporal tokens for each modality.

[0133] The visual enhancement timing token of each modality is processed by nonlinear transformation to generate a refined timing token for each modality, and the refined timing token of each modality is used as the timing token of the next video frame.

[0134] Determine the affinity matrix between specific enhanced visual features for each modality and refined temporal tokens;

[0135] Based on the affinity matrix of each modality, the specific enhanced visual features of each modality are weighted and fused to generate unified features.

[0136] In one specific embodiment, cross-modal temporal aggregation processing of specific enhanced visual features and temporal tokens for each modality can complete context-based temporal cue propagation processing. Optionally, to maintain cross-modal temporal consistency, a multi-head cross-attention mechanism is used to propagate the specific enhanced visual features of each modality to the temporal tokens of each modality. The specific process may include: using a multi-head cross-attention mechanism... The temporal token of each modality is used as the query, and the specific augmented visual features of each modality are used as the key and value. For each modality's specific augmented visual features... and timing tokens Attentional interaction processing is performed to generate visually enhanced temporal tokens for each modality; then, a multilayer perceptron can be used. A nonlinear transformation is performed on the visual enhancement timing tokens for each modality to generate refined timing tokens for each modality. As shown in the following formula:

[0137] .

[0138] Optionally, the refined timing token of each modality is used as the timing token of the next video frame. Optionally, the timing token of each modality in the current video frame can be determined as follows: For the previous video frame, through a multi-head cross-attention mechanism, attention interaction processing is performed on the specific enhanced visual features and timing tokens of each modality in the previous video frame to generate visually enhanced timing tokens for each modality in the previous video frame; nonlinear transformation processing is performed on the visually enhanced timing tokens of each modality in the previous video frame to generate refined timing tokens for each modality in the previous video frame; and the refined timing tokens of each modality in the previous video frame are used as the timing tokens for each modality in the current video frame.

[0139] In one specific embodiment, cross-modal temporal aggregation can be performed, integrating spatial representations with historical temporal context. Specifically, this involves determining specific enhanced visual features for each modality. With refined timing tokens The affinity matrix of each modality is then used; subsequently, based on the affinity matrix of each modality, the specific enhanced visual features of each modality are weighted and fused to generate a unified feature. :

[0140] ;

[0141] in, Represents matrix multiplication. This represents the transpose of the refined timing token.

[0142] Optionally, the affinity matrix between specific enhanced visual features and refined temporal tokens for each modality can characterize the correlation strength between specific enhanced visual features and refined temporal tokens for each modality.

[0143] In the above embodiments, it can flexibly adapt to arbitrary modal inputs, process inputs of any number, modality type, and combination of modalities, generate routing logic values ​​for each appearance expert through a modality-aware router, and introduce learnable noise for noise gating. Based on the Top-K sparse selection mechanism, a subset of target appearance experts is adaptively selected for each modality. The visual features of each modality are processed by the selected subset of target appearance experts and shared semantic experts to capture modality-specific appearance features and extract cross-modality shared semantically invariant features. Fine-grained feature learning and cross-modal alignment are achieved through semantically guided fusion, thereby realizing fine-grained decoupling and semantic alignment of heterogeneous modal features. This effectively solves the problem of performance degradation caused by blind fusion, and achieves excellent performance in both complete and missing modality scenarios. At the same time, the temporal cues are aggregated through a cross-modal attention mechanism to ensure spatiotemporal consistency in heterogeneous fusion.

[0144] S104: Input the unified feature and multimodal cue feature into the context understanding module for bidirectional information propagation processing to generate refined unified features. The multimodal cue feature is obtained based on the acquired multimodal cue information.

[0145] In one specific embodiment Figure 3 This is a schematic diagram of a context understanding module processing procedure provided in an embodiment of the present invention, such as... Figure 3 As shown, the processing of unified features and multimodal features into the context understanding module can be seen.

[0146] In one specific embodiment, multimodal cue features can be determined based on the acquired multimodal cue information. Specifically, this can include: acquiring multimodal cue information, which includes the tracking result of the previous video frame, language description, and audio segments; performing feature extraction processing on the multimodal cue information to generate initial multimodal cue features; and concatenating the initial multimodal cue features along the token dimension to generate additional multimodal cue features. Optionally, the context understanding module can include a multimodal cue encoder, which can perform feature extraction processing on the multimodal cue information to generate the initial multimodal cue features.

[0147] Specifically, obtain the tracking results of the previous video frame. Language description and audio clips The language description and audio segment can be respectively the prompt text and prompt audio to facilitate target localization; the multimodal prompt encoder can include a motion trajectory encoder, a text encoder, and an audio encoder. Then, the motion trajectory encoder, text encoder, and audio encoder process the tracking result, language description, and audio segment of the previous video frame, respectively, to generate initial motion trajectory features, initial text features, and initial audio features; then, a multilayer perceptron performs dimension alignment processing on the initial motion trajectory features, initial text features, and initial audio features to generate motion trajectory features. Text features and audio features The above process can be represented by the following formula:

[0148] ;

[0149] in, This refers to a multimodal cue encoder, which includes a motion trajectory encoder, a text encoder, and an audio encoder. A multilayer perceptron is used for dimension alignment. To distinguish corner types in motion tokens, learnable point embeddings are introduced. Then, motion trajectory features, text features, and audio features are concatenated along the token dimension to form multimodal cue features. :

[0150] ;

[0151] in, The token dimension represents the characteristics of the motion trajectory. The token dimension represents the text features. The token dimension represents the audio feature.

[0152] Optionally, multimodal information can be used to provide spatial priors and semantic guidance to achieve precise target localization. By establishing a spatial correspondence between visual features and target location through the aforementioned determined global-local cues (i.e., multimodal cue features), contextual modeling of target perception is achieved, resulting in clear target boundaries. Optionally, global-local cues act as a bridge between historical spatial context and current visual features (i.e., unified features), providing necessary priors for precise localization.

[0153] In an optional embodiment, the above-described context understanding module may include Asymmetric bidirectional attention layer;

[0154] The above-mentioned input of unified features and multimodal cue features into the context understanding module for bidirectional information propagation processing, generating refined unified features, may include:

[0155] The unified features and multimodal cue features are input into the current layer's asymmetric bidirectional attention layer. Based on the multi-head self-attention mechanism, the multimodal cue features are processed by context aggregation to generate cue interaction features.

[0156] Based on the multi-head cross-attention mechanism, cross-context aggregation processing is performed on the cue interaction features and the mask features obtained from the mask memory to generate initial memory-enhanced cue features;

[0157] The initial memory enhancement cue features are subjected to a nonlinear transformation to generate memory enhancement cue features;

[0158] Based on the multi-head cross-attention mechanism, cross-context aggregation processing is performed on unified features and memory-enhanced cue features to generate cue-enhanced unified features;

[0159] The unified cue enhancement feature output from the current asymmetric bidirectional attention layer, along with the multimodal cue features and mask features, are input into the next asymmetric bidirectional attention layer. This process repeats the steps from multi-head self-attention mechanism-based context aggregation of the multimodal cue features to generate cue interaction features, to multi-head cross-attention mechanism-based cross-context aggregation of the unified feature and memory-enhanced cue features to generate the unified cue enhancement feature, until completion. Layer-by-layer processing of the asymmetric bidirectional attention layer generates refined and unified features.

[0160] In one specific embodiment, target cue features and unified features are input to... A layer-by-layer asymmetric bidirectional attention layer is used for processing to obtain refined and unified features; optionally, The number of asymmetric bidirectional attention layers can be set according to the actual application requirements. Specifically, The processing of inputting unified features and multimodal cue features into the current layer's asymmetric bidirectional attention layer includes: based on a multi-head self-attention mechanism, using multimodal cue features as queries, keys, and values, performing context aggregation on the multimodal cue features to generate cue interaction features, which can explore their inherent relationships; then, performing propagation from memory to cue, based on a multi-head cross-attention mechanism, using multimodal cue features as queries, and starting from the mask memory... The mask features obtained are used as keys and values. Cross-context aggregation is performed on the cue interaction features and mask features to generate initial memory-enhanced cue features. Then, nonlinear transformation is applied to the initial memory-enhanced cue features to generate memory-enhanced cue features. As shown in the following formula:

[0161] ;

[0162] in, This indicates the bulls' self-attention. This represents multi-head cross-attention. Optionally, the encoded target representation of historical video frames, i.e., mask features, can be stored using a dimensional mask memory to maintain spatial coherence. Then, the cues are propagated from the viewpoint. Based on the multi-head cross-attention mechanism, the unified feature is used as the query, and the memory-enhanced cue features are used as the key and value. Cross-context aggregation processing is performed on the same feature and the memory-enhanced features to generate a unified cue-enhanced feature. As shown in the following formula:

[0163] ;

[0164] At this point, the current asymmetric bidirectional attention layer processing is complete. The unified cue enhancement feature output by the current asymmetric bidirectional attention layer, along with the unchanged multimodal cue features and mask features, can be input into the next asymmetric bidirectional attention layer. The steps described above—using the multimodal cue features as queries, keys, and values ​​based on a multi-head self-attention mechanism to perform contextual aggregation processing on the multimodal cue features to generate cue interaction features—continue until the steps described above—using the unified feature and memory-enhanced cue features to perform cross-contextual aggregation processing based on a multi-head cross-attention mechanism to generate cue enhancement unified features—are completed. Layer-by-layer processing of the asymmetric bidirectional attention layer generates refined unified features, specifically using the cue enhancement unified features output by the last asymmetric bidirectional attention layer as the refined unified features. Optionally, in each cross-attention step, a dense spatial graph is introduced into the visual features and mask memory as a positional encoding to ensure spatial correspondence between modalities.

[0165] In the above embodiments, by modeling the different information flows between global-local cues and visual features through asymmetric bidirectional attention, cross-modal information can be fully utilized. The asymmetric bidirectional attention mechanism enables bidirectional information propagation between global-local cues and visual features: global-local cues receive the spatial context of historical mask memory, allowing cues to adapt to spatial details in visual features; and visual features receive semantic guidance from global-local cues, enabling visual features to perceive the semantic guidance from the cues.

[0166] In an optional embodiment, the above method may further include:

[0167] Based on the target tracking information, the target tracking bounding box is determined. The target tracking bounding box includes the bounding box center, bounding box width, and bounding box height.

[0168] The first initial Gaussian variance and the second initial Gaussian variance are determined based on the bounding box width and the bounding box height, respectively.

[0169] The first and second offsets are predicted using a variance estimation network based on refined unified features.

[0170] The Gaussian pseudomask is determined based on the bounding box center, the first initialized Gaussian variance, the second initialized Gaussian variance, the first offset, and the second offset.

[0171] The Gaussian pseudomask is encoded to generate the initial mask features;

[0172] The initial mask features and the refined unified features are fused to generate refined mask features, which are then stored in the mask memory so that the refined mask features can be used as the mask features for the next video frame.

[0173] In one specific embodiment, a position-adaptive Gaussian pseudomask can be generated to provide flexible spatial guidance for target localization, adapting to sequence-specific target variations. Optionally, the aforementioned context understanding module may also include a dynamic mask generator, which can generate refined mask features and store them in mask memory.

[0174] In a specific embodiment, the target tracking information includes a classification score tensor, a spatial offset tensor, and a normalized scale tensor. The classification score tensor represents the probability that the target to be tracked exists at each location in the search image; the spatial offset tensor is used to compensate for discretization errors; and the normalized scale tensor represents the width and height of the target to be tracked. Determining the target tracking bounding box based on the target tracking information may include: determining the location with the highest classification score based on the classification score tensor; and constructing the target tracking bounding box at the location with the highest classification score by combining the spatial offset tensor and the normalized scale tensor. Its bounding box center is The bounding box width is The bounding box height is .

[0175] In one specific embodiment, based on the bounding box width and bounding box height Determine the first initial Gaussian variance respectively. Second initial Gaussian variance While this static initialization provides a reasonable spatial prior, a variance estimation network is needed to adapt to sequence-specific target variations. Based on refined and unified features Predict the first offset Second offset As shown in the following formula:

[0176] ;

[0177] in, This represents the pooling operation; specifically, the refined unified features are pooled to obtain the pooled refined unified features; the pooled refined unified features are then processed by a variance estimation network to predict the offsets, generating the first offset and the second offset.

[0178] In a specific embodiment, the first initialized Gaussian variance can be used. and the first offset Indeed, when the first variance According to the second initial Gaussian variance Second offset Determine the second variance Determine the Gaussian pseudomask based on the bounding box center, the first variance, and the second variance. As shown in the following formula:

[0179] .

[0180] In one specific embodiment, a mask encoder can be used. Gaussian pseudomask Encoding is performed to generate initial mask features; then a feature fusion network can be used. Initial mask features and refined unified features Perform feature fusion processing to generate refined mask features. As shown in the following formula:

[0181] The refined mask features can be stored in the mask memory. The refined mask features are used as the mask features for the next video frame.

[0182] in, ; ;

[0183] Specifically, through the mask encoder Gaussian pseudomask Encoding processes to generate initial mask features may include: Gaussian pseudomask. The initial mask features are generated by sequentially performing convolution operations, layer normalization, GELU activation function processing, and convolution operations; then, a feature fusion network is used. Initial mask features and refined unified features The feature fusion process to generate refined mask features may include: sequentially performing extraction and operation on the initial mask features and refined unified features, layer normalization, convolution operation, GELU activation function processing, and convolution operation to generate refined mask features.

[0184] Optionally, the mask features of the current video frame are determined as follows: For the previous video frame, based on the target tracking information of the previous video frame, the target tracking bounding box of the previous video frame is determined. The target tracking bounding box of the previous video frame includes the center, width, and height of the bounding box. Based on the width and height of the bounding box of the previous video frame, the first and second initialized Gaussian variances of the previous video frame are determined respectively. A variance estimation network is then used to predict the mask features of the previous video frame based on the refined unified features of the previous video frame. First offset and second offset; based on the bounding box center of the previous video frame, first initial Gaussian variance, second initial Gaussian variance, first offset and second offset, determine the Gaussian pseudo-mask of the previous video frame; encode the Gaussian pseudo-mask of the previous video frame to generate the initial mask feature of the previous video frame; perform feature fusion processing on the initial mask feature and refined unified feature of the previous video frame to generate the refined mask feature of the previous video frame, and store it in the mask memory, so as to use the refined mask feature of the previous video frame as the mask feature of the current video frame.

[0185] In the above embodiments, the generation of Gaussian pseudomasks is guided by the target tracking bounding box, and the spatial prior of the dynamically generated Gaussian pseudomask is adapted to the position. This enables contextual modeling of target perception and enhances foreground-background discrimination. Thus, the context understanding module can establish the spatial correspondence between visual features and target positions through multimodal cue encoding, asymmetric bidirectional attention, and dynamic mask generation mechanisms. It can also achieve bidirectional semantic calibration through hierarchical information propagation and refine the contextual modeling of target perception. This can effectively enhance the discrimination ability between foreground and background, thereby effectively addressing problems such as scale changes, motion blur, and interference from similar objects, and thus improving tracking accuracy in complex scenes.

[0186] S105: Input the refined and unified features into the prediction module for target detection processing to generate target tracking information.

[0187] In one specific embodiment, the prediction module can be configured according to the actual application. The prediction module can be a prediction head. Refined and unified features are input into the prediction head for target detection processing to generate target tracking information. The target tracking information may include a classification score tensor, a spatial offset tensor, and a normalized scale tensor. Optionally, the target tracking result of the current video frame can be determined based on the target tracking information of the current video frame. Specifically, the search region features are extracted from the refined unified features output by the context understanding module and input into the prediction head for target detection processing to generate target tracking information.

[0188] In an optional embodiment, the pre-trained arbitrary modality target tracking model is trained based on multi-task loss, where the multi-task loss is based on focus loss, ... The loss, generalized intersection-union ratio loss, and equilibrium loss are determined as shown in the following formula:

[0189] ;

[0190] in, Indicates multi-tasking loss. Indicates focal loss. This indicates the generalized intersection and comparison loss. express loss, Indicates the loss of equilibrium; , , and These represent the first hyperparameter, the second hyperparameter, the third hyperparameter, and the fourth hyperparameter, respectively.

[0191] The balancing loss is determined based on the importance score and load of each apparent expert for each mode, as shown in the following equation:

[0192] ;

[0193] in, Representing modes The The importance score of an expert in appearance, Representing modes The The load of an appearance expert; Representing modes The total number of appearance experts; It represents the ratio of the standard deviation to the mean.

[0194] In one specific embodiment, to ensure balanced utilization of various appearance experts, a balancing loss is introduced. Specifically, determine the mode. The An expert on appearance Importance score Modality The An expert on appearance Importance score This can be the sum of the gating weights of the apparent expert on the batch samples, as shown in the following formula:

[0195] ;

[0196] in, Indicates the batch sample index. Indicates the first An appearance expert in Gating weights on batch samples;

[0197] Determine the mode The An expert on appearance load Modality The An expert on appearance The load can be the activation frequency of the apparent expert, as shown in the following formula:

[0198] ;

[0199] in, Indicates the first An appearance expert in Activation frequency on batch samples;

[0200] Since discrete TOP-K operations are not differentiable, a Gaussian cumulative distribution function can be used based on the routing logic value. Sure It can represent the first An expert on appearance The likelihood of falling into the TOP-K selection. Specifically, the balancing loss can be the sum of the importance scores of all modes and the coefficients of variation of the loading terms, as described above. As shown in the formula;

[0201] in, It can be modal The coefficient of variation of importance scores, It can be modal The coefficient of variation of the loading terms;

[0202] in, ;

[0203] in, This represents the mean. This represents a small constant, specifically it can be set as... Optionally, the coefficient of variation can be used to measure the dispersion of data. Specifically, to facilitate the determination of the coefficient of variation, both the numerator and denominator are squared. In this case, the definition of the coefficient of variation changes from the ratio of the standard deviation to the mean to the ratio of the variance to the square of the mean. This balanced loss can promote the balanced utilization of experts across modalities, thereby enhancing the model's training stability and generalization ability.

[0204] In one specific embodiment, for the overall optimization of model training, a multi-task loss is adopted, which is based on focus loss. The losses, generalized intersection-union ratio losses, and equilibrium losses are determined as described above. As shown, the first, second, third, and fourth hyperparameters can be set according to the actual application. , , and .

[0205] In the above embodiments, the stability and robustness of the hybrid expert model training are ensured by balancing the loss, and the accuracy, reliability and generalization ability of the model training can be improved by designing multi-task loss, thereby improving the accuracy of target location prediction.

[0206] This invention also provides an arbitrary modal target tracking device, accordingly, Figure 4 This is a schematic diagram of the structure of an arbitrary modal target tracking device provided in an embodiment of the present invention; as shown below. Figure 4 As shown, the above-mentioned device includes:

[0207] The data acquisition unit 410 is configured to acquire the template input image and search input image of the current video frame, and to acquire a trained arbitrary modality target tracking model; the search input image includes a search image of at least one modality, and the template input image includes a template image of at least one modality; the arbitrary modality target tracking model includes a visual encoding module, a modality-aware interaction module, a context understanding module, and a prediction module;

[0208] The target tracking unit 420 is configured to input the spatiotemporal token of each modality into a modality-shared visual encoding module for feature extraction processing to obtain visual features for each modality; the spatiotemporal token of each modality is obtained based on the token sequence of the template image, the token sequence of the search image, and the temporal token of each modality; the temporal token is a temporal token generated from the previous video frame; the visual features of each modality are input into the modality perception interaction module for feature interaction processing to generate unified features; the unified features and multimodal cue features are input into the context understanding module for bidirectional information propagation processing to generate refined unified features; the multimodal cue features are obtained based on the acquired multimodal cue information; the refined unified features are input into the prediction module for target detection processing to generate target tracking information.

[0209] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the arbitrary modal target tracking method as described in any of the method embodiments.

[0210] Embodiments of the present invention also provide a computer storage medium, which may be disposed in a server to store at least one instruction, at least one program, code set, or instruction set for implementing the method embodiments. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the arbitrary modal target tracking method as described in any of the method embodiments.

[0211] Optionally, in embodiments of the present invention, the storage medium may be located at at least one of a plurality of network servers in a computer network. Optionally, in embodiments of the present invention, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0212] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0213] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more flowcharts and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0214] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more flowcharts and / or boxes Figure 1 The function specified in one or more boxes.

[0215] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more flowcharts and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0216] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0217] Finally, it should be noted that the embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for tracking targets with arbitrary modalities, characterized in that, The method includes: The system acquires the template input image and search input image of the current video frame, as well as a trained arbitrary modality target tracking model. The search input image includes a search image of at least one modality, and the template input image includes a template image of at least one modality. The arbitrary modality target tracking model includes a visual encoding module, a modality-aware interaction module, a context understanding module, and a prediction module. The spatiotemporal token of each modality is input into the modality-shared visual encoding module for feature extraction processing to obtain the visual features of each modality; the spatiotemporal token of each modality is obtained based on the token sequence of the template image of each modality, the token sequence of the search image, and the temporal token; the temporal token is a temporal token generated from the previous video frame; The visual features of each modality are input into the modality perception interaction module for feature interaction processing to generate unified features; The unified feature and the multimodal cue feature are input into the context understanding module for bidirectional information propagation processing to generate a refined unified feature; the multimodal cue feature is obtained based on the acquired multimodal cue information; The refined and unified features are input into the prediction module for target detection processing to generate target tracking information.

2. The arbitrary modal target tracking method according to claim 1, characterized in that, The modality-aware interaction module includes multiple appearance experts and a shared semantic expert. The step of inputting the visual features of each modality into the modality-aware interaction module for feature interaction processing to generate unified features includes: Select a target number of target appearance experts for each modality from the plurality of appearance experts; The visual features of each modality are input to the target number of target appearance experts to perform modality-specific feature extraction processing, thereby generating the target number of specific appearance features for each modality; The visual features of each modality are input into the shared semantic expert for cross-modal shared feature extraction processing to generate cross-modal shared semantic features for each modality. Based on the gating weights corresponding to each target appearance expert in each modality, feature fusion processing is performed on each specific appearance feature, cross-modal shared semantic feature and visual feature of each modality to generate specific enhanced visual features for each modality. Cross-modal temporal aggregation processing is performed on specific enhanced visual features and temporal tokens for each modality to generate the unified feature.

3. The arbitrary modal target tracking method according to claim 2, characterized in that, The modality-aware interaction module also includes a modality-aware router; The step of selecting a target number of target appearance experts for each modality from the plurality of appearance experts includes: The compressed visual features of each modality are input into the modality-aware router, and the routing logic value of each appearance expert corresponding to each modality is output. A noise gating mechanism is adopted to inject learnable noise into the routing logic value of each appearance expert corresponding to each mode, thereby generating the gated routing value of each appearance expert corresponding to each mode. The gated routing values ​​of multiple appearance experts corresponding to each modality are sorted in descending order to generate the gated routing value sorting result for each modality. The appearance experts corresponding to the top number of gated route values ​​in the sorting results of the gated route values ​​for each modality are determined as the target number of target appearance experts for each modality. The gating weights corresponding to each target appearance expert in each modality are determined in the following manner: The gating routing values ​​of each target appearance expert in each modality are normalized to generate the gating weights corresponding to each target appearance expert in each modality.

4. The arbitrary modal target tracking method according to claim 2, characterized in that, The process of cross-modal temporal aggregation of specific enhanced visual features and temporal tokens for each modality to generate the unified feature includes: Through a multi-head cross-attention mechanism, attention interaction processing is performed on specific enhanced visual features and temporal tokens of each modality to generate visually enhanced temporal tokens for each modality. The visual enhancement timing token of each modality is subjected to nonlinear transformation processing to generate a refined timing token for each modality, so as to use the refined timing token of each modality as the timing token of the next video frame. Determine the affinity matrix between the specific enhanced visual features of each modality and the refined temporal token; Based on the affinity matrix of each modality, the specific enhanced visual features of each modality are weighted and fused to generate the unified feature.

5. The arbitrary modal target tracking method according to claim 1, characterized in that, The context understanding module includes Asymmetric bidirectional attention layer; The step of inputting the unified feature and multimodal cue feature into the context understanding module for bidirectional information propagation processing to generate refined unified features includes: The unified feature and the multimodal cue feature are input into the current layer asymmetric bidirectional attention layer. Based on the multi-head self-attention mechanism, the multimodal cue feature is subjected to context aggregation processing to generate cue interaction features. Based on the multi-head cross-attention mechanism, the cue interaction features and the mask features obtained from the mask memory are subjected to cross-context aggregation processing to generate initial memory-enhanced cue features; The initial memory enhancement cue features are subjected to nonlinear transformation processing to generate memory enhancement cue features; Based on a multi-head cross-attention mechanism, cross-context aggregation processing is performed on the unified feature and the memory-enhancing cue feature to generate a cue-enhancing unified feature. The cue enhancement unified feature output from the current asymmetric bidirectional attention layer, along with the multimodal cue feature and the mask feature, are input into the next asymmetric bidirectional attention layer. The steps of performing context aggregation processing on the multimodal cue feature based on a multi-head self-attention mechanism to generate cue interaction features, and then performing cross-context aggregation processing on the unified feature and the memory-enhanced cue feature based on a multi-head cross-attention mechanism to generate cue enhancement unified features, are repeated until the process is complete. Layer-by-layer processing of the asymmetric bidirectional attention layer generates refined and unified features.

6. The arbitrary modal target tracking method according to claim 5, characterized in that, The method further includes: Based on the target tracking information, a target tracking bounding box is determined, wherein the target tracking bounding box includes a bounding box center, a bounding box width, and a bounding box height; The first initial Gaussian variance and the second initial Gaussian variance are determined based on the bounding box width and the bounding box height, respectively. Based on the refined unified features, the variance estimation network predicts the first and second offsets. A Gaussian pseudomask is determined based on the bounding box center, the first initialized Gaussian variance, the second initialized Gaussian variance, the first offset, and the second offset. The Gaussian pseudo-mask is encoded to generate initial mask features; The initial mask features and the refined unified features are subjected to feature fusion processing to generate refined mask features, which are then stored in the mask memory so that the refined mask features can be used as the mask features for the next video frame.

7. The arbitrary modal target tracking method according to claim 1, characterized in that, The trained arbitrary modal target tracking model is obtained based on multi-task loss training, which is based on focus loss. The loss, generalized intersection-union ratio loss, and equilibrium loss are determined as shown in the following formula: ； in, Indicates multi-tasking loss. Indicates focal loss. This indicates the generalized intersection and comparison loss. express loss, Indicates the loss of equilibrium; , , and These represent the first hyperparameter, the second hyperparameter, the third hyperparameter, and the fourth hyperparameter, respectively. The balancing loss is determined based on the importance score and load of each apparent expert for each mode, as shown in the following formula: ； in, Representing modes The The importance score of an expert in appearance, Representing modes The The load of an appearance expert; Representing modes The total number of appearance experts; It represents the ratio of the standard deviation to the mean.

8. A target tracking device for arbitrary modes, characterized in that, The device includes: The data acquisition unit is configured to acquire the template input image and search input image of the current video frame, and to acquire a trained arbitrary modality target tracking model; the search input image includes a search image of at least one modality, and the template input image includes a template image of at least one modality; the arbitrary modality target tracking model includes a visual encoding module, a modality-aware interaction module, a context understanding module, and a prediction module; The target tracking unit is configured to input the spatiotemporal tokens of each modality into a modality-shared visual encoding module for feature extraction processing to obtain visual features for each modality. The spatiotemporal tokens for each modality are obtained based on the token sequence of the template image, the token sequence of the search image, and the temporal tokens. The temporal tokens are generated from the previous video frame. The visual features of each modality are input into the modality perception interaction module for feature interaction processing to generate unified features. The unified features and multimodal cue features are input into the context understanding module for bidirectional information propagation processing to generate refined unified features. The multimodal cue features are obtained based on acquired multimodal cue information. The refined unified features are input into the prediction module for target detection processing to generate target tracking information.

9. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the arbitrary modal target tracking method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction and at least one program, which are loaded and executed by a processor to implement the arbitrary modal target tracking method as described in any one of claims 1 to 7.

Images