Multi-modal joint learning tracking method and system based on attention mechanism

Abstract

The invention discloses a multi-modal joint learning tracking method and system based on an attention mechanism. The method comprises the following steps: acquiring a target template area and a search area of RGB modal and infrared modal images of a fire scene; the target template areas and the search areas of the two modals are subjected to blocking and embedding to obtain corresponding image block embedding, and the image block embedding is combined with empirical vectors to obtain feature representations of the two modals; the features corresponding to the two modals are represented and sent to a feature extraction network, and corresponding feature vectors are extracted; sending the sum of the feature vectors corresponding to the two modals into a multi-modal fusion network to realize mutual fusion of bimodal features, obtaining a robust feature representation after fusion, and updating an empirical vector through the fusion features; and transmitting the fused feature representation to a target positioning head, and generating a classification score graph of the target of the cost frame and a bounding box regression result for personnel search and rescue decision.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a multimodal joint learning tracking method and system based on attention mechanism. Background Technology

[0002] Single-object tracking, a core task in computer vision, aims to accurately and robustly locate a target in subsequent video sequences given its position in an initial frame. It has demonstrated significant application value in various fields such as autonomous driving, intelligent monitoring, and human-computer interaction. Traditional single-object tracking methods primarily rely on RGB image information; however, in challenging scenarios such as drastic lighting changes, target occlusion, complex backgrounds, or insufficient pixel information, the perception capability of a single modality is often severely limited.

[0003] To overcome the inherent limitations of RGB modality, multimodal single-target tracking has emerged. By fusing complementary information from different sensors (such as infrared, depth, and millimeter-wave radar), the robustness and accuracy of tracking systems in complex environments have been significantly enhanced. However, how to efficiently and effectively fuse these heterogeneous modal data, extract their deep semantic associations, and construct a unified and discriminative feature representation remains a key challenge in the field of multimodal target tracking. Existing methods often have shortcomings in terms of intermodal information interaction: either they use simple feature concatenation, leading to information redundancy and insufficient fusion; or they rely too much on the advantages of a single modality, making it difficult to fully utilize the complementarity of multiple modalities; even worse, some methods lack dynamic evaluation of the reliability of the modality itself during multimodal feature fusion, causing low-quality modal information to interfere with high-confidence modalities, ultimately limiting the overall performance and generalization ability of the tracker. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a multimodal joint learning tracking method based on an attention mechanism.

[0005] This invention introduces dynamic unification and reliability-weighted fusion of heterogeneous modal features, as well as empirical labeling, into the existing technical solutions.

[0006] This invention includes the following steps: A multimodal joint learning tracking method based on an attention mechanism significantly improves the performance and robustness of multimodal target tracking in complex scenes by deeply mining and effectively fusing heterogeneous modal information. Its key features include the following steps: 1) Obtain RGB modal images respectively and auxiliary modal images Target template area and search area ; 2) Based on step 1), the target template regions of the two modes are respectively... and search area Image block embeddings are obtained by segmentation and embedding, and then merged with empirical vectors to obtain feature representations for both modalities. and ; 3) Based on step 2), represent the features corresponding to the two modes respectively. and The data is fed into a feature extraction network to extract the corresponding feature vectors. and ; 4) Based on step 3), the feature vectors corresponding to the two modes are... and The data is fed into a multimodal fusion network to achieve the fusion of dual-modal features and obtain a robust feature representation after fusion. And update the experience vector by fusing features; 5) Based on step 4), represent the fused robust features. The data is passed to the target localization head to generate a classification score map and bounding box regression results for the target, thereby accurately predicting the target's location.

[0007] In step 1), the target template regions of the dual-modal images are obtained respectively. and search area This includes the following sub-steps: (11) Based on the manually labeled bounding box information of the tracking target, the tracking template region is cropped from the first frame of the multimodal video sequence. ; (12) Based on the bounding box regression results of the previous frame, crop the search region from the current frame image. ; In step 2), the feature representations of the two modes are obtained respectively. and This includes the following sub-steps: (21) Target template area and search area Divide into a series of sub-blocks, and embed each sub-block into a vector representation; (22) The vector representations of the target template region and the search region are merged with the empirical vectors to obtain the feature representation of the modality; In step 3), the feature vectors corresponding to the two modes are obtained. and The specific method is as follows: The acquired modal features are represented and The data are fed into a neural network-based feature extraction network to obtain feature vectors corresponding to the two modalities. and ; In step 4), the robust feature representation after multimodal fusion is obtained. The empirical vector is updated by fusing features, including the following sub-steps: (41) Convert the RGB modal feature vector Multi-head attention encoding (MA) is performed as follows:

[0008] in Depend on Generated after passing through a linear layer. Indicates the batch quantity. Indicates the number of feature vectors. Representing feature dimension, This indicates the number of heads during the calculation process. , This represents the transpose. Finally, the features encoded by multi-head attention are obtained. Similarly, for auxiliary modal features... The same multi-head attention encoding was also performed to obtain features. .

[0009] (42) Explosion adapter receives RGB features and auxiliary modal features The adapter contains three independent compression paths and one explosion layer: RGB feature compression: compressing RGB features to... Wei indicates that he has received .

[0010] Modal feature compression: compressing modal features to... Wei indicates that he has received .

[0011] Joint feature compression: This involves compressing the concatenated features... , Compression to a low-dimensional representation ∈ .

[0012] Subsequently, these three low-dimensional compressed features were concatenated:

[0013] Get a dimension as The fused features are obtained by passing the fused features through a linear layer. Dimensional explosion back to original feature dimensions Generate residual features ∈ .

[0014] Finally, the output of the explosion adapter is added back to the original RGB and modal features via a residual connection:

[0015]

[0016] (43) The unified projector receives the RGB features processed by the explosion adapter. and auxiliary modal features The following key components are then fed in sequence: RGB and Modal Information Adapter: RGB features and modal features are fed into two separate fully connected layers for further feature refinement, and the original features are enhanced through residual connections.

[0017]

[0018] Modal reliability estimator (Reliable): will and The network is fed into a subnetwork consisting of a linear layer, a PReLU activation function, and a sigmoid function for reliability evaluation. Specifically, for the subnetwork after the normalization layer (NORM)... Features and feature:

[0019]

[0020] These reliability scores are ensured to sum to 1 by the Softmax function, representing the relative weights between modes.

[0021] Joint projection: Based on the calculated reliability score, the unified projector performs weighted fusion of refined RGB features and modal features to generate unified joint projection features. ∈ :

[0022] in This indicates element-wise multiplication.

[0023] Final output layer: A linear layer and a normalization layer map the joint projected features to the final mixed features.

[0024] (44) In addition, , and It is passed to the spatial hybrid token generator to enhance the experience vector. Specifically, the workflow is as follows: Feature selection and recovery: First, It is mapped to a feature space suitable for the module to work by using a simple linear layer (MLP).

[0025]

[0026] Spatial information compression: using one-dimensional convolutional layers right Perform downsampling in the spatial dimension:

[0027] Empirical Token Augmentation: Downsampled Features Divided into two parts: and , respectively corresponding to the beginning and end of the sequence and Information that needs to be enhanced.

[0028]

[0029]

[0030] Finally, these two pieces of information are added to the original RGB features using a residual join. and auxiliary modal features The corresponding empirical token position in the middle.

[0031] Step 5) describes obtaining the target classification score map and bounding box regression results, which includes the following sub-steps: (51) First, the padded search region label sequence is reinterpreted as a two-dimensional spatial feature map, and then input into a fully connected convolutional network, which consists of L stacked convolutional layers, batch normalization layers, and ReLU layers. The output of the fully connected convolutional network contains the object classification score map. Local offset And the normalized bounding box size (i.e., width and height). .in These represent the height and width of the search area, respectively. Indicates the size of the block.

[0032] (52) Local offsets are used to compensate for discretization errors caused by reduced resolution. The position with the highest classification score is considered the target position, i.e. Final target bounding box The calculation formula is:

[0033] A second aspect of the present invention relates to a multimodal joint learning tracking system based on an attention mechanism, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the multimodal joint learning tracking method based on an attention mechanism of the present invention.

[0034] This invention proposes a novel multimodal joint learning framework, with three core innovative modules. First, an explosion adapter lightweightly achieves deep fusion of cross-modal information and extraction of core features within the Transformer encoder. By compressing, fusing, and expanding RGB and auxiliary modal features, it effectively captures the complementarity and redundancy between modalities, generating more discriminative shared representations. Second, a unified projector is designed for the backbone network, dynamically projecting heterogeneous modal features onto a unified semantic space. This module introduces a reliability evaluation mechanism to adaptively weight and fuse different modal information, significantly improving the semantic consistency and cross-modal robustness of features. Finally, to enhance the model's ability to perceive target context, a spatial hybrid token generator is introduced. Utilizing the fused features output by the unified projector, it dynamically enhances empirical markers located at the beginning and end of the input sequence. This enhancement mechanism cleverly injects global contextual information into key locations, thereby refining spatial feature representations and optimizing the tracker's localization accuracy.

[0035] The advantages of this invention are: by deeply mining and effectively fusing heterogeneous modal information, it significantly improves the performance and robustness of multimodal target tracking in complex scenarios, and realizes multi-level, adaptive information interaction and representation learning between modalities. It not only achieves deep information sharing and core feature extraction among modalities within the Transformer, but also realizes dynamic unified projection and reliability-weighted fusion of heterogeneous modal features. Furthermore, by dynamically evaluating modal reliability, it successfully solves the semantic misalignment problem between modalities, effectively enhancing the model's spatial awareness capability. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method used in this application.

[0037] Figure 2 This is a schematic diagram of the multimodal joint learning framework based on the attention mechanism proposed in this application.

[0038] Figure 3This is a schematic diagram of the explosion adapter structure of this application.

[0039] Figure 4 This is a schematic diagram of the unified projector structure of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] Example 1

[0042] This invention provides a method for searching and rescuing people at a fire scene using a multimodal joint learning tracking method based on an attention mechanism, such as... Figure 1 As shown, the steps include: Step S1: Obtain the target template area and search area of ​​the RGB and infrared modal images of the fire scene.

[0043] For real-time acquired video sequences of a fire scene, this embodiment first obtains its tracking template area. and search area Specifically, in this embodiment, a fixed multimodal sensing system (including a visible light camera and an infrared thermal imager) first synchronously acquires RGB modal images and infrared modal images of the fire scene to ensure the alignment of the two modalities in time and space.

[0044] When the system starts up (i.e., in the early stages of a fire), since there is no manually labeled information, this embodiment uses an automatic human candidate detection module to obtain the bounding box information of the human tracking target in the first frame. This detection module can stably detect the outline of a human heat source in complex environments such as dense smoke, insufficient light, or flickering flames, thereby outputting reliable human candidate bounding boxes. When the confidence of the detected human tracking target bounding box is greater than a specified threshold (e.g., 0.7), it indicates that a trapped person has appeared, and this frame is defined as the first frame of the subsequent video sequence.

[0045] Based on the human tracking target bounding box information of this frame The corresponding tracking template regions are cropped from the RGB and infrared images respectively. This region is used to characterize the initial human appearance and thermal features. Meanwhile, to ensure the searchable range of targets in subsequent frames, this embodiment expands the bounding box outwards by a certain proportion (e.g., 1.5 to 2.5 times the target's width and height) to generate the search area. Its center position is the same as the template area, covering the possible movement range of the trapped person. The size of the search area can be dynamically adjusted according to the scene, for example, automatically expanding the range when the flames are intense or the smoke is strong, in order to improve the matching success rate of subsequent frames.

[0046] In subsequent frames, track the template region. Maintain consistency with the first frame to provide a stable visual reference; while the search area Then, based on the bounding box regression results of the previous frame... It is obtained by synchronously cropping from the RGB and infrared images of the current frame. In this way, the system can achieve continuous positioning and tracking of trapped personnel.

[0047] Step S2: Divide and embed the target template region and search region of the two modalities into blocks to obtain the corresponding image block embeddings, and merge them with the empirical vectors.

[0048] Using an RGB image as an example, the template image block and search area blocks Segmented and flattened into block sequences and ,in The resolution for each block, These represent the number of blocks in the template and the search region, respectively. Then, a trainable linear projection layer is used to... and Projected to A 3D latent space, the output of which is an image patch embedding. Simultaneously, the empirical vector is randomly initialized to obtain... Finally, By merging, the feature representations of the RGB modes are obtained. .

[0049] It should be noted that, Indicates the number of channels. Indicates altitude, Indicates width, Indicates the input batch size. ; Similarly, the characteristic representation of the infrared modes can be obtained. ; Step S3: Feed the feature representations corresponding to the two modalities into the feature extraction network respectively to obtain the corresponding feature vectors.

[0050] For example, the original ViT model can be used as a feature extraction network, which can stably output multi-layer feature representations under environments such as smoke, flame, and changing lighting. and After being fed into the feature extraction network, the feature vectors corresponding to the two modes can be obtained. and .

[0051] Step S4: Input the feature vectors corresponding to the two modalities into the multimodal fusion network to achieve mutual fusion of the two modal features.

[0052] To eliminate the differences in feature distribution and semantics between RGB and infrared modes, this embodiment uses RGB mode feature vectors... Multi-head attention encoding (MA) is performed as follows:

[0053] in Depend on Generated after passing through a linear layer. Indicates the batch quantity. Indicates the number of feature vectors. Representing feature dimension, This indicates the number of heads during the calculation process. , This represents the transpose. Finally, the features encoded by multi-head attention are obtained. .

[0054] Similarly, for infrared modal characteristics The same multi-head attention encoding was also performed to obtain features. .

[0055] Subsequently, the explosion adapter receives RGB features. and infrared modal characteristics The adapter contains three independent compression paths and one explosion layer: RGB feature compression: compressing RGB features to... Wei indicates that he has received .

[0056] Infrared modal feature compression: compressing infrared modal features to... Wei indicates that he has received .

[0057] Joint feature compression: This involves compressing the concatenated features... , Compression to a low-dimensional representation ∈ .

[0058] Subsequently, these three low-dimensional compressed features were concatenated:

[0059] Get a dimension as The fused features are obtained by passing the fused features through a linear layer. Dimensional explosion back to original feature dimensions Generate residual features ∈ .

[0060] Finally, the output of the explosion adapter is added back to the original RGB and modal features via a residual connection:

[0061]

[0062] Subsequently, the unified projector receives the RGB features processed by the explosion adapter. and infrared modal characteristics The following key components are then fed in sequence: RGB and Modal Information Receivers (Adapter): RGB features and infrared modal features are fed into two separate fully connected layers for further feature refinement, and the original features are enhanced through residual connections.

[0063]

[0064] Modal reliability estimator (Reliable): will and The network is fed into a subnetwork consisting of a linear layer, a PReLU activation function, and a sigmoid function for reliability evaluation. Specifically, for the subnetwork after the normalization layer (NORM)... Features and feature:

[0065]

[0066] These reliability scores are ensured to sum to 1 by the Softmax function, representing the relative weights between modes.

[0067] Joint projection: Based on the calculated reliability score, the unified projector performs weighted fusion of refined RGB features and infrared modal features to generate unified joint projection features. ∈ :

[0068] in This indicates element-wise multiplication.

[0069] Final output layer: A linear layer and a normalization layer map the joint projected features to the final mixed features.

[0070] also, , and It is passed to the spatial hybrid token generator to enhance the experience vector. Specifically, the workflow is as follows: Feature selection and recovery: First, It is mapped to a feature space suitable for the module to work by using a simple linear layer (MLP).

[0071]

[0072] Spatial information compression: using one-dimensional convolutional layers right Perform downsampling in the spatial dimension:

[0073] Empirical Token Augmentation: Downsampled Features Divided into two parts: and , respectively corresponding to the beginning and end of the sequence and Information that needs to be enhanced.

[0074]

[0075]

[0076] Finally, these two pieces of information are added to the original RGB features using a residual join. and infrared modal characteristics The corresponding empirical token position in the middle.

[0077] Step S5: Pass the fused feature representation to the target localization head to generate the target classification score map and bounding box regression results for personnel search and rescue decision-making.

[0078] First, the padded search region label sequence is reinterpreted as a two-dimensional spatial feature map, and then fed into a fully connected convolutional network. This network consists of L stacked convolutional layers, batch normalization layers, and ReLU layers. The output of the fully connected convolutional network contains the object classification score map. Local offset And the normalized bounding box size (i.e., width and height). .in These represent the height and width of the search area, respectively. Indicates the size of the block.

[0079] Local offsets are used to compensate for discretization errors caused by reduced resolution; the location with the highest classification score is considered the target location. Final target bounding box The calculation formula is:

[0080] If the scores of all candidate positions in the current frame are lower than the preset threshold (e.g., 0.7), the system determines that the target may be occluded or temporarily lost, and enters the re-detection mode (returning to the "automatic human candidate detection module" in step S1 to re-detect the target).

[0081] Predicted bounding box of the current frame The bounding box will serve as the search reference frame for the next frame, and its position and scale information will be used to determine the search area for the next frame. To adapt to target movement or dynamic fire environment, the search area size can be adaptively adjusted according to the target's movement speed or confidence level: if the center offset is large between consecutive frames or the confidence level fluctuates greatly, the search window should be appropriately expanded; if the target is stable and the confidence level is high, the search window can be reduced to minimize invalid calculations.

[0082] Meanwhile, if the tracking confidence level remains above a preset threshold (e.g., 0.7) for multiple consecutive frames, the system triggers a template update mechanism, cropping the high-confidence target region of the current frame into a new template region. The template features are re-extracted by the multimodal fusion module to adapt to changes in the trapped personnel's posture, flame reflection interference, or smoke obstruction.

[0083] If the system is continuous If a high-confidence target is not detected in frames (e.g., frames 3-5), it is determined that the target is temporarily lost; at this time, the multimodal detector is reactivated to search for new candidate targets in the entire frame range.

[0084] When the similarity between a detected new target and a historical template feature exceeds a preset threshold (e.g., 0.7), it is determined to be the same target and the tracking process resumes; if the similarity is insufficient, it is regarded as a new target and a new tracking instance is started.

[0085] Finally, this embodiment outputs the target bounding box coordinates for each frame. and their corresponding confidence levels The system then uses camera extrinsic parameters to perform spatial mapping within a three-dimensional coordinate system at the fire scene to obtain the real-time location coordinates of trapped personnel. The tracking results can be output to a control terminal for personnel location marking and trajectory display in the command center, and for path planning and personnel priority assessment in the backend search and rescue strategy module.

[0086] During training, both classification and regression losses are used. A weighted focus loss is employed for classification. For the predicted bounding boxes, a weighted focus loss is used. Loss and generalization The loss is applied to bounding box regression. Finally, the overall loss function is:

[0087] Example 2

[0088] This embodiment relates to a multimodal joint learning tracking system based on an attention mechanism, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the multimodal joint learning tracking method based on an attention mechanism of Embodiment 1.

[0089] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An attention mechanism-based multi-modal joint learning tracking method, characterized in that, The steps include: 1) obtaining a target template region and a search region of RGB modality images and auxiliary modality images respectively;​​​​ 2) Separate the target template regions of the two modalities. and search area Image block embeddings are obtained by segmentation and embedding, and then merged with empirical vectors to obtain feature representations for both modalities. and ; 3) Represent the features corresponding to the two modes respectively. and The data is fed into a feature extraction network to extract the corresponding feature vectors. and ; 4) Convert the feature vectors corresponding to the two modes. and The data is fed into a multimodal fusion network to achieve the fusion of dual-modal features and obtain a robust feature representation after fusion. And update the experience vector by fusing features; 5) the fused robust feature representation is passed to the target localization head to generate the target classification score map and the bounding box regression results, thus accurately predicting the target location. passed to the target localization head to generate the target classification score map and the bounding box regression results, thus accurately predicting the target location.

2. The attention mechanism based multi-modal joint learning tracking method according to claim 1, characterized in that: Step 1) said target template region of the bimodal image is obtained respectively and search region , comprising the following sub-steps: (11) cropping a tracking template region from the first frame image of the multi-modal video sequence according to the tracking target bounding box information annotated by human ; (12) According to the bounding box regression result of the last frame, the search region is cropped from the current frame . 3.The attention mechanism based multi-modal joint learning tracking method according to claim 1, characterized in that: Step 2) said obtaining feature representations of the two modalities respectively and comprising the following sub-steps: (21) dividing the target template region and the search region into a series of sub-blocks and embedding each sub-block into a vector representation; (22) Merge the vector representation of the target template region and the search region with the experience vector, thereby obtaining the feature representation of the modality. 4.The attention mechanism based multi-modal joint learning tracking method according to claim 1, characterized in that: Step 3) obtaining the feature vectors corresponding to the two modalities and The specific method is: The obtained two modal feature representations and are respectively sent into a neural network-based feature extraction network to obtain feature vectors corresponding to the two modalities and . 5.The attention mechanism based multi-modal joint learning tracking method according to claim 1, characterized in that: Step 4) said obtaining the robust feature representation after multi-modal fusion and updating the experience vector by fusing the features, including the following sub-steps: (41) the RGB modality feature vector Multi-head attention encoding (MA) is performed, in particular as follows: wherein by after linear layer. denotes batch size, denotes feature vector size, denotes feature dimension, denotes the number of heads in the computation, , denotes transposition. Finally, the feature after multi-head attention encoding is obtained. Similarly, the same multi-head attention encoding is done for the auxiliary modality feature to obtain the feature . (42) The explosion adapter receives RGB features and auxiliary modalities features , the adapter contains three independent compression paths and an explosion layer: RGB feature compression: compressing the RGB features to dimensional representation . modal feature compression: compressing modal features to dimensional representation . Joint feature compression: compress the concatenated features [ , ] to a low-dimensional representation ∈ . Subsequently, the three low-dimensional compressed features are concatenated (Concat): A fused feature with dimension is obtained. The fused feature is exploded from dimension back to the original feature dimension by a linear layer to generate a residual feature ∈ . Finally, the output of the explosion adapter is added back to the original RGB and modality features through a residual connection: (43) The unified projector receives the RGB features and auxiliary modalities features from the explosion adapter processed in sequence to the following key components: ​ RGB and modality information receiver adapter: respectively send the RGB features and modality features into two independent fully connected layers for further feature refinement, and strengthen the original features through a residual connection: Reliable: sends and into a subnetwork consisting of a linear layer, a PReLU activation function, and a Sigmoid function for reliability evaluation. Specifically, for normalized features (NORM) and features and features: These reliability scores ensure that their sum is 1 through a Softmax function, representing the relative weight between modalities. Joint projection: based on the calculated reliability scores, the unified projector performs weighted fusion on the refined RGB features and modal features to generate unified joint projection features ∈ : wherein represents an element-wise multiplication. Final output layer: a linear layer and a normalization layer map the joint projection features to the final mixed features (44) Furthermore, , and are passed into the spatially mixed Token Generator for augmenting the experience vector . Specifically, the workflow is as follows: Feature selection and recovery: First we will It is mapped to a feature space suitable for the module to work by a simple linear layer (MLP). Spatial information compression: one-dimensional convolutional layer The spatial dimensions are down-sampled down-sampled in the spatial dimension: Empirical Token Enhancement: Downsampled features are split into two parts: and , corresponding to the beginning and end of the sequence and information that needs to be enhanced. Finally, these two parts of information are added to the corresponding empirical Token positions in the original RGB features and auxiliary modality features respectively via a residual connection. 6.The attention mechanism based multi-modal joint learning tracking method according to claim 1, characterized in that: Step 5) The acquisition of the classification score map and the bounding box regression result of the target includes the following sub-steps: (51)First, the filled search region label sequence is reinterpreted as a two-dimensional spatial feature map, which is then input into a fully connected convolutional network consisting of L stacked convolutional layer-batch normalization layer-RELU layer. The output of the fully connected convolutional network contains the target classification score map , local offset , and normalized bounding box size (i.e., width and height) . Where denotes the height and width of the search region, respectively, denotes the size of the patch. (52) The local offset is used to compensate for the discretization error caused by the resolution reduction, and the position with the highest classification score is regarded as the target position, i.e. the final target bounding box The calculation formula is: 。 7. A multi-modal joint learning tracking system based on attention mechanism, characterized in that, The memory stores executable code, and the one or more processors execute the executable code to implement the attention mechanism-based multi-modal joint learning tracking method of any one of claims 1-6.

Images