Target tracking method based on natural language description and collaborative global-local perspective learning

Abstract

The invention discloses a target tracking method based on natural language description and collaborative global-local perspective learning. The method comprises the following steps: step 1, feature extraction and primary processing of an input video frame; 2, extracting visual features of global and local branches; 3, performing multi-modal feature alignment to realize information complementation and consistency expression; 4, important information is selected through symmetrical attention operation; 5, estimating the position of the target through a prediction module; and step 6, optimizing the tracking model. According to the invention, through collaborative modeling of global and local features, a multi-modal alignment mechanism and feature enhancement of symmetric attention, the precision and robustness of target tracking are significantly improved, and the problems of insufficient fusion of global and local information and insufficient visual and language feature alignment ability in the prior art are solved. According to the method, the target can be accurately positioned under the conditions of complex scenes and natural language description, and excellent performance is shown on a plurality of target tracking data sets with natural language annotations.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and natural language processing, and relates to a target tracking method, and specifically to a target tracking method based on natural language description and collaborative global-local perspective learning. Background Art

[0002] Traditional target tracking methods usually rely on visual information to locate the target. However, with the rise of multimodal learning, more and more studies have begun to combine visual information with natural language descriptions to improve tracking accuracy. Although existing tracking technologies based on natural language descriptions have made some progress in combining local visual information with global language information, they are often prone to losing key information and affecting tracking accuracy due to the asymmetry of information fusion and simplified design. Therefore, how to effectively fuse visual and language information from a global and local perspective to solve problems in existing technologies remains a challenge that needs to be solved. Summary of the invention

[0003] In order to address the problem of insufficient fusion of visual and language information in existing target tracking technologies, the present invention provides a target tracking method based on natural language description and collaborative global-local perspective learning. This method solves the problems of insufficient fusion of global and local information and insufficient alignment of visual and language features in the prior art by introducing joint modeling of global and local branches and a multimodal alignment mechanism. Specifically, the global branch captures long-range dependencies, the local branch focuses on local details, and efficient fusion of global and local information is achieved through pixel-level correlation operations. The multimodal alignment mechanism deeply combines visual and language features through cross-attention and self-attention, so that the model can understand natural language descriptions more accurately. The symmetric attention mechanism includes channel self-attention and spatial window self-attention, which not only optimizes global semantic relationships and local spatial details, but also further improves feature expression capabilities through pixel-level fusion.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A target tracking method based on natural language description and collaborative global-local perspective learning includes the following steps:

[0006] Step 1: Feature extraction and preliminary processing of input video frames:

[0007] Given an input frame I∈R H×W×3 , extract deep features from the input frame through the visual encoder to generate the initial feature map Where H and W represent the height and width of the input frame respectively. and are the height and width of the initial feature map, respectively, and C is the number of channels;

[0008] Step 2: Extraction of global and local branch visual features:

[0009] Step 2.1: Feature extraction of global feature branch:

[0010] Step 2.1.1: Global features of input First, it is divided into multiple extended partitions. During the division process, each partition is formed into a non-continuous area through sparse sampling. The area size is represented by h M × M , and h M =w M =3, the features after division are expressed as: in Represents the global token at position (i, j) in the global feature map, where h and w are the height and width of the global feature map respectively;

[0011] Step 2.1.2, each partition calculates self-attention by providing query embedding (Query), key embedding (Key) and value embedding (Value) to obtain the global token;

[0012] Step 2.2: Feature extraction of local feature branch:

[0013] The local feature branch passes the value embedding of the global feature branch through deep convolution to generate query embedding, key embedding and value embedding, and collects local information f through self-attention. l , get the local token, the self-attention calculation process is consistent with the self-attention calculation process used in the global feature;

[0014] Step 3: Multimodal feature alignment to achieve information complementarity and consistent expression:

[0015] Step 3.1: Generation and preprocessing of language features:

[0016] In order to align the natural language description with the visual features, the input natural language description is processed by the text encoder to generate language tokens. The specific steps are as follows:

[0017] Step 3.1.1: After the natural language description is converted into a one-hot representation, a preliminary semantic representation is generated through the embedding layer;

[0018] Step 3.1.2: The sentence encoder further converts the preliminary semantic representation into language tokens, denoted as in Indicates the number of language tokens;

[0019] Step 3.1.3: In order to ensure that the language features match the visual features in the channel dimension, a two-layer fully connected network (FC layer) is introduced to f lan Perform dimensional transformation, with ReLU activation function after each layer;

[0020] Step 3.2: Multimodal information fusion of cross-attention layer:

[0021] In the multimodal alignment process, the global and local tokens in the visual features are respectively embedded as keys and values, and the language features are embedded as queries. The fusion of multimodal information is completed through the cross-attention mechanism.

[0022] Step 3.3: Multimodal feature optimization of self-attention layer:

[0023] The multimodal alignment features generated by the cross-attention mechanism enter the self-attention layer to further optimize its expressiveness, and finally generate global-text features and local-text features that contain global and local visual information, as well as semantic information described in natural language;

[0024] Step 4: Symmetric attention operation selects important information:

[0025] Step 4.1: Channel self-attention operation:

[0026] Step 4.1.1: For the global-text feature f gl , generate query embedding Q through linear projection gl , key embedding K g l and value embedding V gl :

[0027] Q gl =f gi W Q ,K gl =f gl W K ,V gl =f gl W V ,

[0028] Among them, W Q ,W K ,W V The projection matrices representing query embedding, key embedding, and value embedding respectively;

[0029] Step 4.1.2: Perform attention calculation on the channel dimension. The formula is as follows:

[0030]

[0031] Among them, α is a learnable temperature parameter used to adjust the range of attention distribution;

[0032] Step 4.2: Spatial window self-attention operation:

[0033] Step 4.2.1: For the local-text feature f ll , generate query embedding Q through linear projection ll , key embedding K ll The sum value is embedded in V ll :

[0034] Q ll =f ll W Q ,K ll =f ll W K ,V ll =f ll W V ,

[0035] Step 4.2.2: Divide the local features into multiple non-overlapping windows, each containing m×n pixels, and calculate the attention in each window. The formula is as follows:

[0036]

[0037] Where d is the scaling parameter and D represents the relative position encoding, which is used to enhance the spatial relationship within the window;

[0038] Step 4.3: Use pixel-level correlation calculation process to fuse global and local features into joint features f GL ;

[0039] Step 5: Estimate the target location through the prediction module:

[0040] Step 5.1, step 2 to step 4 are regarded as one stage. In order to fully extract global and local features, a total of four stages are performed to obtain the final features for result prediction;

[0041] Step 5.2: The prediction module contains two prediction heads, one for estimating the probability of foreground or background, and the other for regressing the bounding box of the target. These two prediction heads process the joint feature f respectively. GL , object classification is done by the classification head, and bounding box regression is done by the regression head:

[0042]

[0043] in, It is a feedforward neural network (FFN) for classification, outputting a two-dimensional vector P cls , corresponding to the foreground and background scores at that location, It is a feedforward neural network for regression, outputting a four-dimensional vector P reg , represents the distance from the current position to the four sides of the bounding box. Finally, the prediction process of the bounding box is: using the classification score to select the position with the highest foreground probability as the target center point, and combining the regression results to generate the final bounding box;

[0044] Step 6: Track model optimization:

[0045] The tracking model consists of a four-stage global-local module and a prediction module. The model is optimized by calculating the loss of classification and regression of global-local enhanced features. The model optimization objectives include binary cross entropy loss. and generalized IoU loss The loss of the classification task is calculated using binary cross entropy, while the generalized IoU loss is used for the regression task. The total loss function is It is expressed as:

[0046]

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] 1. The present invention significantly improves the accuracy and robustness of target tracking through the collaborative modeling of global and local features, multimodal alignment mechanism and feature enhancement of symmetric attention. Among them, the global branch captures the long-range dependency between the target and the scene through the self-attention mechanism; the local branch focuses on the fine-grained features of the target through deep convolution and self-attention mechanism, and realizes the efficient fusion of global and local features through pixel-level correlation; the multimodal alignment module deeply combines visual features with language descriptions through cross-attention and self-attention operations, generates global-text features and local-text features, and provides high-quality multimodal representation for target prediction; the symmetric attention mechanism captures global semantic relationships through channel self-attention, and the spatial window self-attention models the interaction between local features, further optimizing the fusion and expression of features.

[0049] 2. The present invention can accurately locate targets in complex scenes and under conditions of natural language description, and demonstrates excellent performance on multiple target tracking datasets with natural language annotations. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the target tracking method based on natural language description and collaborative global-local perspective learning of the present invention;

[0051] Figure 2 This is a structural diagram of the multimodal alignment module of the present invention;

[0052] Figure 3It is a structural diagram of the symmetric attention module of the present invention;

[0053] Figure 4 Comparison between the method of the present invention and the existing visual target tracking methods on the OTB-lang and TNL2k datasets;

[0054] Figure 5 Comparison between the method of the present invention and the existing visual target tracking method on the LaSOT and LaSOT_ext data sets;

[0055] Figure 6 The figure shows a comparison between the proposed method and existing visual target tracking methods on the WebUAV-3M dataset. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0057] The present invention provides a target tracking method based on natural language description and collaborative global-local perspective learning, such as Figure 1 As shown, the specific steps include:

[0058] Step 1: Feature extraction and preliminary processing of input video frames:

[0059] Given an input frame I∈R H×W×3 Extract features and convert them into multi-scale feature representations suitable for deep learning modeling. Process the input frame through the visual encoder to generate the initial feature map in and are the height and width of the initial feature map respectively, and C is the number of channels.

[0060] Step 2: Extraction of global and local branch visual features:

[0061] Step 2.1: Feature extraction of global feature branch:

[0062] The global feature branch in the present invention is composed of a full Transformer structure, which includes 4 stages, and the feature maps of each stage decrease step by step. Combined with the "scanning" mechanism, it simulates the human observation of the scene to quickly capture the overall information of the target from a global perspective. Through this branch, the long-range dependency between the target and the background can be modeled, providing extensive contextual information for subsequent tracking tasks. Specifically, for the input global feature First, it is divided into multiple extended partitions. During the division process, each partition is formed into a non-contiguous area through sparse sampling, and the area size is represented by hM × M , and h M =w M =3. The feature after division is expressed as: in The global token representing the position (i, j) in the global feature map, h and w are the height and width of the global feature map, respectively, and C is the number of channels. The partitioning operation is implemented by the expansion operation. Then, each partition undergoes the vanilla self-attention process by providing query embedding, key embedding, and value embedding.

[0063] Step 2.2: Feature extraction of local feature branch:

[0064] The design of the local feature branch originates from the “gaze” mechanism, and its purpose is to make up for the local details that the global branch may ignore when modeling long-range dependencies. The global branch mainly captures the overall information across regions, but due to its characteristics, it is difficult to effectively handle local interactions between partitions. In the target tracking task, these local details are crucial to understanding the edges, textures, and local structures of the target. Therefore, the present invention constructs a local feature branch through deep convolution, and refines the output of the global feature branch to optimize the joint expression of global and local information. The value embedding of the global branch is passed through deep convolution, where the convolution kernel of the local branch is set to k. In this way, the global and local features are effectively optimized. Here, query embedding, key embedding, and value embedding are also generated for collecting local information f through basic self-attention. l .

[0065] Step 3: Multimodal feature alignment to achieve information complementarity and consistent expression:

[0066] Step 3.1: Generation and preprocessing of language features:

[0067] In order to align natural language descriptions with visual features, such as Figure 2 As shown in FIG. 1 , the present invention first processes the input natural language description through a text encoder to generate language tokens. Specifically, after the natural language description is converted into a one-hot representation, a preliminary semantic representation is generated through an embedding layer. Then, the sentence encoder further converts the preliminary semantic representation into language tokens, which are recorded as in In order to ensure that the language features match the visual features in the channel dimension, the present invention introduces a two-layer fully connected network (FC layer) to f lan The dimension transformation is performed, and the ReLU activation function is attached after each layer. This process ensures the expressiveness of language features and provides a dimensionally consistent feature representation for multimodal alignment.

[0068] Step 3.2: Multimodal information fusion of cross-attention layer:

[0069] In the multimodal alignment process, visual features (including global and local tokens) are used as keys (K) and values ​​(V), respectively, and language features are used as queries (Q), and the fusion of multimodal information is completed through the cross-attention mechanism. In this process, the query information in the language description is matched with the global or local information in the visual features, achieving a deep fusion between the language description and the target visual features. This step ensures that the visual features can understand the semantic information in the language description, such as the appearance, location, and behavior description of the target.

[0070] Step 3.3: Multimodal feature optimization of self-attention layer:

[0071] The multimodal alignment features generated by the cross-attention mechanism enter the self-attention layer to further optimize its expressiveness. The self-attention mechanism aggregates and strengthens the multimodal features to capture the complex relationship between visual features and language features. Through the above steps, the present invention finally generates global-text features and local-text features that contain global and local visual information, as well as semantic information described by natural language.

[0072] Step 4: Symmetric attention operation selects important information:

[0073] The present invention designs a symmetric attention mechanism to optimize the expressiveness of global and local multimodal features. The symmetric attention mechanism processes global and local features respectively through two architectures: channel self-attention and spatial window self-attention, capturing global semantic relationships and local spatial details. In addition, pixel-level correlation operations are introduced to search for key information in global and local perspectives, thereby achieving more efficient feature fusion.

[0074] Step 4.1, channel self-attention calculation:

[0075] like Figure 3 As shown, the channel self-attention focuses on the global-text feature f gl The purpose is to strengthen the global contextual semantic information through cross-channel information interaction. Specifically, the global-text feature f gl First, the query embedding Q is generated by linear projection gl , key embedding K gl The sum value is embedded in V gl :

[0076] Q gl =f gl W Q ,K gl =f gl WK ,V gl =f gl W V ,

[0077] Among them, W Q ,W K ,W V The projection matrices representing query embedding, key embedding, and value embedding respectively;

[0078] Then, the attention calculation is performed on the channel dimension as follows:

[0079]

[0080] Among them, α is a learnable temperature parameter used to adjust the range of attention distribution. The channel self-attention operation can effectively model global semantic information and make the feature expression more global and holistic.

[0081] Step 4.2: Spatial window self-attention operation:

[0082] Spatial window self-attention focuses on local-text features f ll , used to capture fine-grained spatial relationships within a local range. Local-text features f ll First, the query embedding Q is generated by linear projection ll , key embedding K ll The sum value is embedded in V ll ::

[0083] Q ll =f ll W Q ,K ll =f ll W K ,V ll =f ll W V ,

[0084] Among them, W Q ,W K ,W V The projection matrices representing query embedding, key embedding, and value embedding respectively;

[0085] Subsequently, the local features are divided into multiple non-overlapping windows, each containing m×n pixels, and attention is calculated in each window. The formula is as follows:

[0086]

[0087] Among them, d is the scaling parameter, and D represents the relative position encoding, which is used to enhance the spatial relationship within the window. Through this windowing process, local features can capture the detailed information of the target, such as texture, edge, and shape, and improve the ability to distinguish local features.

[0088] Finally, in order to effectively couple global and local features, the present invention adopts pixel-level correlation to achieve high-quality feature conversion. It can search for similar patterns from local and global perspectives. Therefore, global and local features can be better fused into f GL .

[0089] Step 5: Estimate the target location through the prediction module:

[0090] Steps 2 to 4 are considered as one stage. In order to fully extract global and local features, four stages are performed to obtain the final features for result prediction. The present invention uses two prediction heads, one for estimating the probability of foreground or background, and the other for regressing the bounding box of the target. The two prediction heads process the joint feature f respectively. GL , object classification is done by the classification head, and bounding box regression is done by the regression head:

[0091]

[0092] in, It is a feedforward neural network (FFN) for classification, outputting a two-dimensional vector P cls , corresponding to the foreground and background scores at that location. It is a feedforward neural network for regression, outputting a four-dimensional vector P reg , represents the distance from the current position to the four sides of the bounding box. Finally, the prediction process of the bounding box is the same as other methods. The classification score is used to select the position with the highest foreground probability as the target center point, and the regression result is combined to generate the final bounding box.

[0093] Step 6: Track model optimization:

[0094] The tracking model consists of two parts: a global-local module and a prediction module in four stages. The present invention optimizes the global-local enhanced features by calculating the loss of classification and regression. The model optimization objectives include binary cross entropy loss and generalized IoU loss The loss of the classification task is calculated using binary cross entropy, while the generalized IoU loss is used for the regression task. The total loss function is It is expressed as:

[0095]

[0096] Example:

[0097] This embodiment collects video and natural language query pairs from three data sets including VisualGenome, MSCOCO, and Youtube-BoundingBox, and trains the entire model in combination with the training sets of LaSOT and OTB-lang. In addition, for the fairness of comparison, the present invention only uses the training set of TNL2k to train the model when performing performance testing on the TNL2k data set. The present invention is implemented in Pytorch and runs on hardware equipped with Intel (R) 10700k CPU and Nvidia 3090GPU, with a speed of 31.2FPS. The input of the model is uniformly adjusted to 224×224, and the maximum length of natural language queries is set to 40. If the length of the query is less than 40, an empty token is filled after the [SEP] token. If the length of the query is greater than 40, the query is truncated. The entire model was trained for a total of 86 epochs, each epoch containing 1000 iterations, and was trained using the ADAM optimizer with an initial learning rate of 1×10 -4 , the learning rate decay factor is 0.2 every 10 epochs.

[0098] In the performance testing phase, the present invention was tested on a public dataset with natural language query annotations and compared with other methods. The test datasets include OTB-lang, LaSOT and TNL2k. The above datasets use the same evaluation indicators, including accuracy (P) and success rate (AUC). Table 1 shows the performance comparison of the present invention and the existing target tracking methods based on natural language queries on the OTB-lang, LaSOT and TNL2k datasets. Among them, in the OTB-lang dataset, the present invention surpasses other methods in the success rate indicator, thanks to the proposed framework that can effectively promote the fusion of inter-modal and intra-modal features; in the LaSOT dataset, the present invention is significantly better than the existing methods under two initialization conditions (NL and NL+Box), especially in terms of the success rate indicator, which is 5.2% higher than JointNLT; in the TNL2k dataset, the present invention achieves a success rate of 0.598 under the condition of using only NL initialization, which is significantly improved compared to JointNLT.

[0099] Table 1 Performance comparison on three datasets including OTB-lang, LaSOT and TNL2k

[0100]

[0101] In addition, on the larger dataset WebUAV-3M, the present invention also achieved the best performance, further verifying the robustness and generalization ability of the present invention in complex scenes. The above experimental results show that the present invention can effectively integrate global and local information as well as information between visual and language modalities, and demonstrates excellent performance on multiple target tracking datasets with natural language annotations.

[0102] The performance of the present invention is compared with that of existing advanced methods on multiple target tracking datasets with natural language annotations, such as OTB-lang, TNL2k, LaSOT, LaSOText and WebUAV-3M, all of which use the "NL+Box" initialization strategy, verifying the excellent performance of the present invention.

[0103] In the OTB-lang dataset, Figure 4 The results show that the CGLT of the present invention improves the success rate by 6.6% compared with TransInMo, and achieves the same or even better performance than other competing methods in multiple indicators. The experimental results show that the ability to effectively utilize global information is one of the keys to achieve excellent performance in TNL tasks, and the framework of the present invention achieves this well.

[0104] like Figure 4 As shown in the figure, in the TNL2k dataset, the CGLT of the present invention achieved the best score of 0.621 in terms of AUC index, and compared with the second-ranked SimTrack method, the success rate and accuracy indexes were improved by 8.9% and 8.3% respectively. This result further demonstrates the ability of the present invention to accurately locate targets in complex scenes.

[0105] like Figure 5 As shown in the figure, in the LaSOT and LaSOT_ext datasets, the present invention also shows leading performance. Specifically, in the LaSOT dataset, the present invention improves the AUC index by 3% and 3.3% respectively compared with the SimTrack and AiATrack methods; and in the LaSOT_ext dataset, the CGLT of the present invention achieves the best performance among all methods, which fully proves that it can effectively integrate multimodal information to improve tracking performance.

[0106] In the WebUAV-3M dataset, the present invention achieves the best success rate and accuracy on the largest drone tracking dataset. WebUAV-3M contains 4500 videos, spanning 3.3 million frames, with language and audio annotations. Figure 6 As shown, the present invention can still maintain its leading position in such a complex and multimodal environment, further verifying its strong adaptability to multimodal features.

[0107] In summary, the experimental results of the present invention on multiple benchmark datasets show that it is superior to the existing technology in terms of success rate and accuracy, and fully demonstrates the superior performance and wide applicability of global and local feature fusion and multimodal alignment.

Claims

1. A target tracking method based on natural language description and collaborative global-local perspective learning, characterized in that The method comprises the following steps: Step 1: Feature extraction and preliminary processing of input video frames: Given an input frame I∈R H×W×3 , extract deep features from the input frame through the visual encoder to generate the initial feature map Where H and W represent the height and width of the input frame respectively. and are the height and width of the initial feature map, respectively, and C is the number of channels; Step 2: Extraction of global and local branch visual features: Step 2.1: Feature extraction of global feature branch: Step 2.1.1: Global features of input First, it is divided into multiple extended partitions. During the division process, each partition is formed into a non-continuous area through sparse sampling. The area size is represented by h M × M , and h M =w M =3, the features after division are expressed as: in Represents the global token at position (i, j) in the global feature map, where h and w are the height and width of the global feature map respectively; Step 2.1.2, each partition calculates self-attention by providing query embedding, key embedding and value embedding to obtain a global token; Step 2.2: Feature extraction of local feature branch: The local feature branch passes the value embedding of the global feature branch through deep convolution to generate query embedding, key embedding and value embedding, and collects local information f through self-attention. l , get the local token, the self-attention calculation process is consistent with the self-attention calculation process used in the global feature; Step 3: Multimodal feature alignment to achieve information complementarity and consistent expression: Step 3.1: Generation and preprocessing of language features: In order to align the natural language description with the visual features, the input natural language description is processed through a text encoder to generate language tokens; Step 3.2: Multimodal information fusion of cross-attention layer: In the multimodal alignment process, the global and local tokens in the visual features are used as key embedding and value embedding respectively, and the language features are used as query embedding. The fusion of multimodal information is completed through the cross-attention mechanism. Step 3.3: Multimodal feature optimization of self-attention layer: The multimodal alignment features generated by the cross-attention mechanism enter the self-attention layer to further optimize its expressiveness, and finally generate global-text features and local-text features that contain global and local visual information, as well as semantic information described in natural language; Step 4: Symmetric attention operation selects important information: Step 4.1: Channel self-attention operation: Step 4.1.1: For the global-text feature f gl , generate query embedding Q through linear projection gl , key embedding K gl The sum value is embedded in V gl : Q gl =f gl W Q ,K gl =f gl W K ,V gl =f gl W V , Among them, W Q , W K , W V The projection matrices representing query embedding, key embedding, and value embedding respectively; Step 4.1.2: Perform attention calculation on the channel dimension. The formula is as follows: Among them, α is a learnable temperature parameter used to adjust the range of attention distribution; Step 4.2: Spatial window self-attention operation: Step 4.2.1: For the local-text feature f ll , generate query embedding Q through linear projection ll , key embedding K ll The sum value is embedded in V ll : Q ll =f ll W Q ,K ll =f ll W K ,V ll =f ll W V , Step 4.2.2: Divide the local features into multiple non-overlapping windows, each containing m×n pixels, and calculate the attention in each window. The formula is as follows: Where d is the scaling parameter and D represents the relative position encoding, which is used to enhance the spatial relationship within the window; Step 4.3: Use pixel-level correlation calculation process to fuse global and local features into joint features f GL ; Step 5: Estimate the target location through the prediction module: Step 5.1, step 2 to step 4 are regarded as one stage. In order to fully extract global and local features, a total of four stages are performed to obtain the final features for result prediction; Step 5.2: The prediction module contains two prediction heads, one for estimating the probability of foreground or background, and the other for regressing the bounding box of the target. These two prediction heads process the joint feature f respectively. GL , object classification is done by the classification head, and bounding box regression is done by the regression head: in, It is a feedforward neural network for classification, outputting a two-dimensional vector P cls , corresponding to the foreground and background scores at that location, It is a feedforward neural network for regression, outputting a four-dimensional vector P reg , represents the distance from the current position to the four sides of the bounding box. Finally, the prediction process of the bounding box is: using the classification score to select the position with the highest foreground probability as the target center point, and combining the regression results to generate the final bounding box; Step 6: Track model optimization: The tracking model consists of a four-stage global-local module and a prediction module. The model is optimized by calculating the loss of classification and regression of global-local enhanced features. The model optimization objectives include binary cross entropy loss. and generalized IoU loss The loss of the classification task is calculated using binary cross entropy, while the generalized IoU loss is used for the regression task. The total loss function is It is expressed as:

2. The target tracking method based on natural language description and collaborative global-local perspective learning according to claim 1 is characterized in that The global feature branch is composed of a full Transformer structure, including 4 stages, and the feature map of each stage decreases step by step.

3. The target tracking method based on natural language description and collaborative global-local perspective learning according to claim 1 is characterized in that The specific steps of step 3.1 are as follows: Step 3.1.1: After the natural language description is converted into a one-hot representation, a preliminary semantic representation is generated through the embedding layer; Step 3.1.2: The sentence encoder further converts the preliminary semantic representation into language tokens, denoted as in Indicates the number of language tokens; Step 3.1.3: In order to ensure that the language features match the visual features in the channel dimension, a two-layer fully connected network is introduced to f lan Perform dimensionality transformation and add ReLU activation function after each layer.

Images