Target tracking method based on natural language description and collaborative global-local perspective learning

Abstract

The application discloses a target tracking method based on natural language description and collaborative global-local perspective learning, which comprises the following steps: step 1, feature extraction and preliminary processing of input video frames; step 2, extraction of global and local branch visual features; step 3, multi-modal feature alignment to realize information complementation and consistent expression; step 4, selection of important information through symmetric attention operation; step 5, estimation of the position of the target through a prediction module; and step 6, optimization of the tracking model. Through collaborative modeling of global and local features, multi-modal alignment mechanism and feature enhancement of symmetric attention, the application significantly improves the accuracy and robustness of target tracking, and solves the problems of insufficient fusion of global and local information and insufficient alignment capability of visual and language features in the prior art. The application can accurately locate the target under the condition of a complex scene and natural language description, and exhibits excellent performance on multiple target tracking data sets with natural language annotation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and natural language processing, and relates to a target tracking method, specifically a target tracking method based on natural language description and collaborative global-local perspective learning. Background Technology

[0002] Traditional target tracking methods typically rely on visual information to locate targets. However, with the rise of multimodal learning, more and more research is combining visual information with natural language descriptions to improve tracking accuracy. While existing tracking techniques using natural language descriptions have made some progress in combining local visual and global linguistic information, they often suffer from the loss of crucial information due to information asymmetry and simplified design, affecting tracking accuracy. Therefore, effectively fusing visual and linguistic information from both global and local perspectives to address the problems in existing technologies remains a pressing challenge. Summary of the Invention

[0003] To address the insufficient fusion of visual and linguistic information in existing target tracking technologies, this invention provides a target tracking method based on natural language description and collaborative global-local perspective learning. This method solves the problems of insufficient fusion of global and local information and inadequate alignment capabilities of visual and linguistic features in existing technologies by introducing joint modeling of global and local branches and a multimodal alignment mechanism. Specifically, the global branch captures long-range dependencies, while the local branch focuses on local details. Pixel-level correlation operations achieve efficient fusion of global and local information. The multimodal alignment mechanism deeply integrates visual and linguistic features through cross-attention and self-attention, enabling the model to more accurately understand natural language descriptions. The symmetric attention mechanism, including channel self-attention and spatial window self-attention, not only optimizes global semantic relationships and local spatial details but also further enhances feature representation capabilities through pixel-level fusion.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A target tracking method based on natural language description and collaborative global-local perspective learning includes the following steps:

[0006] Step 1: Feature extraction and preliminary processing of input video frames:

[0007] Given an input frame I∈R H×W×3 The input frame is processed by a visual encoder to extract deep features and generate an initial feature map. Where H and W represent the height and width of the input frame, respectively. and Here, represents the height and width of the initial feature map, respectively, and C is the number of channels;

[0008] Step 2: Extraction of global and local branch visual features:

[0009] Step 2.1, Feature extraction of the global feature branch:

[0010] Step 2.1.1: For the input global features First, it is divided into multiple extended partitions. During the partitioning process, each partition is formed into a non-contiguous region through sparse sampling, and the region size is represented as h. M ×w M , and h M =w M =3, the features after partitioning are represented as: in Let h represent the global token at position (i,j) in the global feature map, and h and w represent the height and width of the global feature map, respectively.

[0011] Step 2.1.2: Each partition calculates self-attention by providing query embedding, key embedding, and value embedding to obtain a global token;

[0012] Step 2.2, Feature extraction of local feature branches:

[0013] The local feature branch passes the value embeddings of the global feature branch through deep convolution, generating query embeddings, key embeddings, and value embeddings, and collects local information f through self-attention. l This yields a local token, and the self-attention computation process is consistent with the self-attention computation process used in the global features.

[0014] Step 3: Multimodal feature alignment achieves information complementarity and consistent representation:

[0015] Step 3.1, Generation and preprocessing of linguistic features:

[0016] To align the natural language description with visual features, a text encoder is used to process the input natural language description to generate language tokens. The specific steps are as follows:

[0017] Step 3.1.1: After the natural language description is converted into a one-hot representation, a preliminary semantic representation is generated through the embedding layer;

[0018] Step 3.1.2: The sentence encoder further converts the preliminary semantic representation into language tokens, denoted as... in This indicates the number of language tokens.

[0019] Step 3.1.3: To ensure that linguistic features and visual features match in the channel dimension, a two-layer fully connected network (FC layer) is introduced to match f. lan Perform dimensional transformation, and append a ReLU activation function to each layer;

[0020] Step 3.2, Multimodal information fusion of the cross-attention layer:

[0021] In the process of multimodal alignment, global and local tokens in visual features are used as key embeddings and value embeddings, respectively, and linguistic features are used as query embeddings. Multimodal information is fused through a cross-attention mechanism.

[0022] Step 3.3, Multimodal Feature Optimization of the Self-Attention Layer:

[0023] The multimodal aligned features generated by the cross-attention mechanism are fed into the self-attention layer to further optimize their expressive power, and finally generate global-text features and local-text features that contain global and local visual information, as well as semantic information of natural language description.

[0024] Step 4: Symmetrical attention operation to select important information:

[0025] Step 4.1, Channel Self-Attention Operation:

[0026] Step 4.1.1: For global-text features f gl Query embedding Q is generated through linear projection. gl , key embedding K g l and value embedding V gl :

[0027] Q gl =f gi W Q ,K gl =f gl W K V gl =f gl W V ,

[0028] Among them, W Q W K W V These represent the projection matrices for query embedding, key embedding, and value embedding, respectively.

[0029] Step 4.1.2: Perform attention calculation along the channel dimension, using the following formula:

[0030]

[0031] Here, α is a learnable temperature parameter used to adjust the range of attention distribution;

[0032] Step 4.2, Spatial Window Self-Attention Operation:

[0033] Step 4.2.1: For local-text features f ll Query embedding Q is generated through linear projection. ll , key embedding K ll Sum embedding V ll :

[0034] Q ll =f ll W Q ,K ll =f ll W K V ll =f ll W V ,

[0035] Step 4.2.2: Divide the local features into multiple non-overlapping windows, each containing m×n pixels. Calculate the attention within each window using the following formula:

[0036]

[0037] Where d is the scaling parameter, and D represents the relative position encoding, which is used to enhance the spatial relationships within the window;

[0038] Step 4.3: Using a pixel-level correlation calculation process, global and local features are fused into a joint feature f. GL ;

[0039] Step 5: Estimate the target's location using the prediction module:

[0040] Steps 5.1 and 2 to 4 are considered as one stage. In order to fully extract global and local features, a total of four stages are performed to obtain the final features for result prediction.

[0041] Step 5.2: The prediction module contains two prediction heads: one for estimating the probability of foreground or background, and the other for regressing the bounding box of the target. These two prediction heads process the joint feature f respectively. GL Target classification is performed by the classification head, and bounding box regression is performed by the regression head.

[0042]

[0043] in, It is a feedforward neural network (FFN) used for classification, which outputs a two-dimensional vector P. cls This corresponds to the foreground and background scores at that location. It is a feedforward neural network used for regression, which outputs a four-dimensional vector P. reg , representing the distance from the current position to the four sides of the bounding box. Finally, the bounding box prediction process is as follows: the position with the highest foreground probability is selected as the target center point using the classification score, and the final bounding box is generated by combining the regression results.

[0044] Step 6: Tracking model optimization:

[0045] The tracking model consists of a four-stage global-local module and a prediction module. The model is optimized by calculating the loss of classification and regression on the global-local enhanced features. The optimization objective includes binary cross-entropy loss. and generalized IoU loss The loss for classification tasks is calculated using binary cross-entropy, while the loss for regression tasks uses generalized IoU loss. The total loss function is... Represented as:

[0046]

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] 1. This invention significantly improves the accuracy and robustness of target tracking through collaborative modeling of global and local features, a multimodal alignment mechanism, and feature enhancement using symmetric attention. Specifically, the global branch captures the long-range dependencies between the target and the scene through a self-attention mechanism; the local branch focuses on fine-grained features of the target through deep convolution and self-attention, achieving efficient fusion of global and local features through pixel-level correlation; the multimodal alignment module deeply integrates visual features with linguistic descriptions through cross-attention and self-attention operations, generating global-textual and local-textual features to provide high-quality multimodal representations for target prediction; and the symmetric attention mechanism captures global semantic relationships through channel self-attention and models the interactions between local features through spatial window self-attention, further optimizing feature fusion and expression.

[0049] 2. This invention can accurately locate targets under complex scene and natural language description conditions, and demonstrates excellent performance on multiple target tracking datasets with natural language annotations. Attached Figure Description

[0050] Figure 1 This is a flowchart of the target tracking method based on natural language description and collaborative global-local perspective learning according to the present invention;

[0051] Figure 2 This is a structural diagram of the multimodal alignment module of the present invention;

[0052] Figure 3This is a structural diagram of the symmetric attention module of the present invention;

[0053] Figure 4 This paper compares the method of the present invention with existing visual target tracking methods on the OTB-lang and TNL2k datasets;

[0054] Figure 5 This paper compares the method of the present invention with existing visual target tracking methods on the LaSOT and LaSOT_ext datasets;

[0055] Figure 6 This paper compares the method of the present invention with existing visual target tracking methods on the WebUAV-3M dataset. Detailed Implementation

[0056] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0057] This invention provides a target tracking method based on natural language description and collaborative global-local perspective learning, such as... Figure 1 As shown, the specific steps include the following:

[0058] Step 1: Feature extraction and preliminary processing of input video frames:

[0059] Given an input frame I∈R H×W×3 Feature extraction is performed, transforming the features into a multi-scale feature representation suitable for deep learning modeling. The input frame is then processed by a visual encoder to generate an initial feature map. in and Here, C represents the height and width of the initial feature map, respectively, and C is the number of channels.

[0060] Step 2: Extraction of global and local branch visual features:

[0061] Step 2.1, Feature extraction of the global feature branch:

[0062] The global feature branch in this invention is constructed using a full Transformer structure, comprising four stages, with the feature map decreasing progressively in each stage. Combined with a "scanning" mechanism, it simulates the rapid capture of overall target information from a global perspective, mimicking human scene observation. This branch allows for modeling long-range dependencies between the target and the background, providing extensive contextual information for subsequent tracking tasks. Specifically, for the input global features... First, it is divided into multiple extended partitions. During the partitioning process, each partition is sparsely sampled to form a non-contiguous region, the size of which is denoted as h.M ×w M , and h M =w M =3. The features after partitioning are represented as follows: in Let represent the global token at position (i,j) in the global feature map, h and w be the height and width of the global feature map, respectively, and C be the number of channels. The partitioning operation is implemented through an unfolding operation. Then, each partition undergoes the original self-attention process by providing query embeddings, key embeddings, and value embeddings.

[0063] Step 2.2, Feature extraction of local feature branches:

[0064] The design of the local feature branch originates from the "stare" mechanism, aiming to compensate for local details that the global branch might overlook when modeling long-range dependencies. The global branch primarily captures overall information across regions, but due to its characteristics, it struggles to effectively handle local interactions between partitions. However, in target tracking tasks, these local details are crucial for understanding the target's edges, textures, and local structures. Therefore, this invention constructs a local feature branch through deep convolution, refining the output of the global feature branch to optimize the joint representation of global and local information. The value embedding of the global branch is passed through deep convolution, with the kernel of the local branch set to k. In this way, global and local features are effectively optimized. Here, query embeddings, key embeddings, and value embeddings are also generated to collect local information f through basic self-attention. l .

[0065] Step 3: Multimodal feature alignment achieves information complementarity and consistent representation:

[0066] Step 3.1, Generation and preprocessing of linguistic features:

[0067] To align natural language descriptions with visual features, such as Figure 2 As shown, this invention first processes the input natural language description using a text encoder to generate language tokens. Specifically, after the natural language description is converted into a one-hot representation, a preliminary semantic representation is generated through an embedding layer. Next, a sentence encoder further converts the preliminary semantic representation into language tokens, denoted as... in The number of language tokens is represented by C, and the number of channels is represented by F. To ensure that language features and visual features match in the channel dimension, this invention introduces two fully connected network layers (FC layers) to match f. lan A dimensionality transformation is performed, with a ReLU activation function appended to each layer. This process ensures the expressive power of language features while providing dimensionally consistent feature representations for multimodal alignment.

[0068] Step 3.2, Multimodal information fusion of the cross-attention layer:

[0069] In the multimodal alignment process, visual features (including global and local tokens) serve as keys (K) and values ​​(V), while linguistic features act as queries (Q). A cross-attention mechanism is used to fuse multimodal information. During this process, the query information in the linguistic description is matched with global or local information in the visual features, achieving deep fusion between the linguistic description and the target visual features. This step ensures that the visual features can understand the semantic information in the linguistic description, such as the target's appearance, location, and behavioral descriptions.

[0070] Step 3.3, Multimodal Feature Optimization of the Self-Attention Layer:

[0071] Multimodal aligned features generated through a cross-attention mechanism are fed into a self-attention layer to further optimize their expressive power. The self-attention mechanism aggregates and enhances the multimodal features, capturing the complex relationship between visual and linguistic features. Through the above steps, this invention ultimately generates global-text features and local-text features that contain global and local visual information, as well as semantic information from natural language descriptions.

[0072] Step 4: Symmetrical attention operation to select important information:

[0073] This invention designs a symmetric attention mechanism to optimize the expressive power of global and local multimodal features. This symmetric attention mechanism processes global and local features respectively through two architectures: channel self-attention and spatial window self-attention, capturing global semantic relationships and local spatial details. Furthermore, pixel-level correlation operations are introduced to search for key information in both global and local perspectives, thereby achieving more efficient feature fusion.

[0074] Step 4.1, Channel self-attention calculation:

[0075] like Figure 3 As shown, channel self-attention focuses on global-text features f gl Channel-dimensional dependency modeling. Its aim is to enhance global contextual semantic information through cross-channel information interaction. Specifically, global-text features f gl First, the query embedding Q is generated through linear projection. gl , key embedding K gl Sum embedding V gl :

[0076] Q gl =f gl W Q ,K gl =f gl WK V gl =f gl W V ,

[0077] Among them, W Q W K W V These represent the projection matrices for query embedding, key embedding, and value embedding, respectively.

[0078] Subsequently, attention is calculated along the channel dimension, using the following formula:

[0079]

[0080] Here, α is a learnable temperature parameter used to adjust the range of attention distribution. Channel self-attention operations can effectively model global semantic information, making feature representations more global and holistic.

[0081] Step 4.2, Spatial Window Self-Attention Operation:

[0082] Spatial window self-attention focuses on local-text features f ll This is used to capture fine-grained spatial relationships within a local area. Local-text features f ll First, the query embedding Q is generated through linear projection. ll , key embedding K ll Sum embedding V ll ::

[0083] Q ll =f ll W Q ,K ll =f ll W K V ll =f ll W V ,

[0084] Among them, W Q W K W V These represent the projection matrices for query embedding, key embedding, and value embedding, respectively.

[0085] Subsequently, the local features are divided into multiple non-overlapping windows, each containing m×n pixels. Attention is calculated within each window, as shown in the following formula:

[0086]

[0087] Here, d is the scaling parameter, and D represents the relative position encoding, used to enhance spatial relationships within the window. Through this windowing process, local features can capture detailed information about the target, such as texture, edges, and shape, thus improving the ability to distinguish local features.

[0088] Finally, to effectively couple global and local features, this invention employs pixel-level correlation to achieve high-quality feature transformation. It can search for similar patterns from both local and global perspectives. Therefore, global and local features can be better fused into f. GL .

[0089] Step 5: Estimate the target's location using the prediction module:

[0090] Steps 2 through 4 are considered as one stage. To fully extract global and local features, a total of four stages are executed to obtain the final features used for result prediction. This invention employs two prediction heads: one for estimating the probability of foreground or background, and the other for regressing the bounding box of the target. These two prediction heads process the joint feature f respectively. GL Target classification is performed by the classification head, and bounding box regression is performed by the regression head.

[0091]

[0092] in, It is a feedforward neural network (FFN) used for classification, which outputs a two-dimensional vector P. cls This corresponds to the foreground and background scores for that location. It is a feedforward neural network used for regression, which outputs a four-dimensional vector P. reg , which represents the distance from the current position to the four sides of the bounding box. Finally, the bounding box prediction process is the same as other methods. The position with the highest foreground probability is selected as the target center point using the classification score, and the final bounding box is generated by combining the regression results.

[0093] Step 6: Tracking model optimization:

[0094] The tracking model consists of a four-stage global-local module and a prediction module. This invention optimizes the model by calculating the loss of classification and regression of the global-local enhanced features. The model optimization objective includes binary cross-entropy loss. and generalized IoU loss The loss for classification tasks is calculated using binary cross-entropy, while the loss for regression tasks uses generalized IoU loss. The total loss function is... Represented as:

[0095]

[0096] Example:

[0097] This embodiment collects video and natural language query pairs from three datasets: VisualGenome, MSCOCO, and Youtube-BoundingBox, and trains the entire model using training sets from LaSOT and OTB-lang. Furthermore, for fairness in comparison, performance testing on the TNL2k dataset uses only the TNL2k training set. This invention is implemented in PyTorch and runs at 31.2 FPS on hardware equipped with an Intel(R) 10700k CPU and an Nvidia 3090 GPU. The model input is uniformly adjusted to 224×224, and the maximum length of the natural language query is set to 40. If the query length is less than 40, an empty token is added after the [SEP] token. If the query length is greater than 40, the query is truncated. The entire model is trained for 86 epochs, each containing 1000 iterations, using the ADAM optimizer with an initial learning rate of 1×10⁻⁶. -4 The learning rate decay factor is 0.2 every 10 epochs.

[0098] In the performance testing phase, this invention was tested on public datasets with natural language query annotations and compared with other methods. The test datasets included OTB-lang, LaSOT, and TNL2k, all using the same evaluation metrics, including Precision (P) and Success (AUC). Table 1 shows the performance comparison between this invention and existing natural language query-based target tracking methods on the OTB-lang, LaSOT, and TNL2k datasets. Specifically, in the OTB-lang dataset, this invention outperformed other methods in terms of success rate, thanks to the proposed framework's ability to effectively promote feature fusion between and within modalities. In the LaSOT dataset, this invention significantly outperformed existing methods under both initialization conditions (NL and NL+Box), particularly achieving a 5.2% improvement in success rate compared to JointNLT. In the TNL2k dataset, this invention achieved a success rate of 0.598 under the condition of using only NL initialization, a significant improvement over JointNLT.

[0099] Table 1 compares the performance on three datasets: OTB-lang, LaSOT, and TNL2k.

[0100]

[0101] Furthermore, this invention also achieved state-of-the-art performance on the larger-scale WebUAV-3M dataset, further validating its robustness and generalization ability in complex scenarios. These experimental results demonstrate that this invention can effectively integrate global and local information, as well as information between visual and linguistic modalities, and exhibits superior performance on multiple target tracking datasets with natural language annotations.

[0102] The performance of this invention was compared with existing state-of-the-art methods on multiple target tracking datasets with natural language annotations, such as OTB-lang, TNL2k, LaSOT, LaSOText, and WebUAV-3M. All of these methods used the "NL+Box" initialization strategy, which verified the superior performance of this invention.

[0103] In the OTB-lang dataset, Figure 4 The results show that the CGLT of this invention improves the success rate by 6.6% compared to TransInMo, and achieves performance parity with or even surpasses other competing methods in several metrics. Experimental results demonstrate that effectively utilizing global information is one of the keys to achieving excellent performance in TNL tasks, and the framework of this invention effectively realizes this.

[0104] like Figure 4 As shown, on the TNL2k dataset, the CGLT method of this invention achieved a state-of-the-art score of 0.621 on the AUC metric, improving the success rate and accuracy by 8.9% and 8.3% respectively compared to the second-best performing SimTrack method. This result further demonstrates the ability of this invention to accurately locate targets in complex scenes.

[0105] like Figure 5 As shown, this invention also demonstrates leading performance on the LaSOT and LaSOT_ext datasets. Specifically, on the LaSOT dataset, this invention improves the AUC metric by 3% and 3.3% compared to the SimTrack and AiATrack methods, respectively; while on the LaSOT_ext dataset, this invention's CGLT achieves the best performance among all methods, fully demonstrating its ability to effectively integrate multimodal information to improve tracking performance.

[0106] In the WebUAV-3M dataset, this invention achieves state-of-the-art success and accuracy on this largest drone tracking dataset. WebUAV-3M contains 4,500 videos spanning 3.3 million frames, with language and audio annotations. Figure 6 As shown, the present invention can still maintain its leading position in such a complex and multimodal environment, further verifying its strong adaptability to multimodal features.

[0107] In summary, experimental results on multiple benchmark datasets demonstrate that this invention outperforms existing technologies in both success rate and accuracy, fully showcasing its superior performance and wide applicability in global and local feature fusion and multimodal alignment.

Claims

1. A target tracking method based on natural language description and collaborative global-local perspective learning, characterized in that... The method includes the following steps: Step 1: Feature extraction and preliminary processing of input video frames: Given an input frame The input frame is processed by a visual encoder to extract deep features and generate an initial feature map. ,in W and W represent the height and width of the input frame, respectively. and These are the height and width of the initial feature map, respectively. Number of channels; Step 2: Extraction of global and local branch visual features: Step 2.1, Feature extraction of the global feature branch: Step 2.1.1: For the input global features First, it is divided into multiple extended partitions. During the partitioning process, each partition is sparsely sampled to form a non-contiguous region, and the region size is represented as... ,and The features after partitioning are represented as follows: ,in This represents the global token at position (i,j) in the global feature map. and These are the height and width of the global feature map, respectively. Step 2.1.2: For each partition, self-attention is calculated by providing query embedding, key embedding, and value embedding to obtain a global token; Step 2.2, Feature extraction of local feature branches: The local feature branch passes the value embeddings of the global feature branch through deep convolution, generating query embeddings, key embeddings, and value embeddings, and collects local information through self-attention. This yields a local token, and the self-attention computation process is consistent with the self-attention computation process used in the global features. Step 3: Multimodal feature alignment achieves information complementarity and consistent representation: Step 3.1, Generation and preprocessing of linguistic features: To align natural language descriptions with visual features, a text encoder is used to process the input natural language descriptions to generate language tokens. Step 3.2, Multimodal information fusion of the cross-attention layer: In the process of multimodal alignment, global and local tokens in visual features are used as key embeddings and value embeddings, respectively, and linguistic features are used as query embeddings. Multimodal information is fused through a cross-attention mechanism. Step 3.3, Multimodal Feature Optimization of the Self-Attention Layer: The multimodal aligned features generated by the cross-attention mechanism are fed into the self-attention layer to further optimize their expressive power, and finally generate global-text features and local-text features that contain global and local visual information, as well as semantic information of natural language description. Step 4: Symmetrical attention operation to select important information: Step 4.1, Channel Self-Attention Operation: Step 4.1.1: For global-text features Generate query embeddings through linear projection Key embedding Sum embedding : ， in, , , These represent the projection matrices for query embedding, key embedding, and value embedding, respectively. Step 4.1.2: Perform attention calculation along the channel dimension, using the following formula: ， in, It is a learnable temperature parameter used to adjust the range of attention distribution; Step 4.2, Spatial Window Self-Attention Operation: Step 4.2.1: For local-text features Generate query embeddings through linear projection Key embedding Sum embedding : ， Step 4.2.2: Divide the local features into multiple non-overlapping windows, each window containing... For each pixel, the attention is calculated within each window using the following formula: ， in, It is a scaling parameter. Represents relative position encoding, used to enhance spatial relationships within a window; Step 4.3: Employ a pixel-level correlation calculation process to fuse global and local features into joint features. ; Step 5: Estimate the target's location using the prediction module: Steps 5.1 and 2 to 4 are considered as one stage. In order to fully extract global and local features, a total of four stages are performed to obtain the final features for result prediction. Step 5.2: The prediction module contains two prediction heads: one for estimating the probability of foreground or background, and the other for regressing the bounding box of the target. These two prediction heads process joint features respectively. Target classification is performed by the classification head, and bounding box regression is performed by the regression head. ， in, It is a feedforward neural network used for classification, which outputs a two-dimensional vector. This corresponds to the foreground and background scores at that location. It is a feedforward neural network used for regression, which outputs a four-dimensional vector. , representing the distance from the current position to the four sides of the bounding box. Finally, the bounding box prediction process is as follows: the position with the highest foreground probability is selected as the target center point using the classification score, and the final bounding box is generated by combining the regression results. Step 6: Tracking model optimization: The tracking model consists of a four-stage global-local module and a prediction module. The model is optimized by calculating the loss from classification and regression of the global-local augmented features. The optimization objective includes binary cross-entropy loss. and generalized IoU loss The loss for classification tasks is calculated using binary cross-entropy, while the loss for regression tasks uses generalized IoU loss. The total loss function is... Represented as: 。 2. The target tracking method based on natural language description and collaborative global-local perspective learning according to claim 1, characterized in that... The global feature branch is composed of a full Transformer structure, which contains four stages, and the feature map of each stage decreases progressively.

3. The target tracking method based on natural language description and collaborative global-local perspective learning according to claim 1, characterized in that... The specific steps of step 3.1 are as follows: Step 3.1.1: After the natural language description is converted into a one-hot representation, a preliminary semantic representation is generated through the embedding layer; Step 3.1.2: The sentence encoder further converts the preliminary semantic representation into language tokens, denoted as... ,in This indicates the number of language tokens. Step 3.1.3: To ensure that linguistic features and visual features match in the channel dimension, a two-layer fully connected network is introduced. Perform dimensional transformation, and append a ReLU activation function to each layer.

Images