Task-specific target tracking method and system based on trajectory cue

Abstract

The invention provides a task-specific target tracking method and system based on trajectory prompt, and relates to the field of target tracking. Firstly, a trajectory prompting module encodes position information of historical frames in an autoregressive manner so as to mine a motion trajectory of a target, so that a tracker is helped to process a scene with similar interferents, and the tracking accuracy is improved; then, a tracking result is obtained by adopting a task-aligned header network, and the relation between a classification task and a regression task is established by introducing an IoU-perceived classification score, so that the negative influence of misalignment of the classification task and the regression task on target tracking is relieved; the cascade optimization module restricts the attention range, reduces the interference of the background area on the target, and improves the precision of the target prediction frame. And finally, designing a template updating strategy based on a statistical method to adaptively propagate reliable appearance representation in the tracking process, so that a tracker is helped to adapt to appearance change and deformation of the target, and the tracking robustness is improved.

Description

Technical Field

[0001] The present invention relates to the field of target tracking, and in particular to a target tracking method based on trajectory prompting, which is suitable for target tracking in complex scenes. Background Art

[0002] Object tracking is a key task in computer vision. It aims to track an object in a video sequence based on a given initial state. It is widely used in fields such as human-computer interaction and intelligent surveillance. Currently, most tracking methods model the tracking problem as a joint feature extraction and feature interaction task to facilitate the extraction of target-oriented features. While such methods have achieved high accuracy on multiple benchmarks (such as GOT-10k, LaSOT, and TrackingNet), they still have several limitations.

[0003] First, mainstream tracking methods use a frame-level training strategy, treating the tracking task as a matching problem between a template image and a search image, ignoring the impact of historical frame information on the prediction of the current frame. To address this limitation, recent researchers have focused on sequence-level tracking methods, which take into account the temporal dependencies of videos and enable the model to capture the appearance changes and motion trends of the target. Sequence-level tracking methods are mainly divided into two categories: one uses historical information to capture the appearance changes of the target, but these methods do not consider the object's motion information and are prone to drift in scenes with interference; the other predicts the target's motion trajectory based on the state of historical frames. Although their performance is superior, they are relatively complex to implement.

[0004] Secondly, some trackers use a task-specific framework to divide the tracking problem into multiple specific subtasks, using a classification head and a regression head to locate the target and regress the bounding box. Although these trackers have shown excellent performance, they still suffer from a common problem, namely task misalignment. When ranking candidate boxes based on classification scores, predicted boxes with high regression accuracy may be removed due to low confidence scores, resulting in a decrease in tracking accuracy. In addition, many tracking methods use visual transformers (ViT) to extract global features, but often lack target-centric inductive biases. Studies have shown that this lack of prior knowledge hinders the model's ability to accurately identify the target area, thereby limiting the accuracy of regression. More importantly, some workers believe that background areas may interfere with the target, further affecting tracking performance.

[0005] Finally, considering the deformation and appearance changes of the object during the tracking process, providing the tracker with new appearances in a timely manner is crucial to improving tracking accuracy. Some tracking algorithms use techniques such as spatiotemporal memory networks or feature splicing to incorporate appearance information obtained from historical frames. However, the insufficient accuracy of historical template features can lead to error accumulation. Another widely used method involves formulating an update strategy to update the dynamic template, which usually relies on a confidence score to evaluate the reliability of the dynamic template, and the update only occurs when a preset update interval is reached. The above-mentioned dynamic templates are usually obtained through frame-level sampling, which cannot capture the continuous appearance evolution and lacks the exploration of the contextual information of consecutive frames. Summary of the Invention

[0006] In response to the shortcomings of existing tracking methods, the present invention proposes a task-specific target tracking method based on trajectory hints, constructs a trajectory hint module, and fuses the target position information of historical frames through simple and direct trajectory attention operations to achieve the purpose of trajectory prediction, thereby enabling the tracking method to handle scenes containing interference. In order to adapt to the appearance changes of the target during the tracking process, the present invention designs a dynamic template update strategy based on statistical methods, which can propagate reliable appearance representations between consecutive frames. In addition, to improve the tracking accuracy, the present invention constructs a cascaded bounding box optimization module to constrain the attention range and reduce background interference. In each tracking head of the cascaded bounding box optimization module, an IoU-aware classification score is introduced as the classification score, which can simultaneously reflect the target presence score and the bounding box regression accuracy, solving the misalignment between the classification task and the regression task.

[0007] The technical solution provided by the present invention is a task-specific target tracking method based on trajectory prompting, comprising the following steps:

[0008] Step 1: pre-process the template image and search image respectively, and input them into the feature extraction network for feature extraction to obtain the search image features;

[0009] Step 2: Construct a trajectory hint module, generate a Gaussian mask based on the trajectory hints of the historical N frames, and expand the Gaussian mask along the channel dimension. Then, input the search image features and the expanded Gaussian mask into the trajectory attention layer and normalization layer to obtain features containing trajectory information.

[0010] Step 3: construct a cascaded bounding box optimization module to process the features containing trajectory information to obtain the final prediction result; the cascaded bounding box optimization module includes two head networks and a cascaded attention layer;

[0011] Step 4: Template update based on statistical methods: Based on the attention weights obtained by the feature extraction network in step 1, the weight of the search image relative to the center point of the template image is separated. The dynamic threshold and the confidence score of the current search image area are calculated according to the statistical characteristics. When the confidence score and the dynamic threshold meet certain conditions, the current search image area is passed to the subsequent frame as the new dynamic template; if the conditions are not met, the dynamic template remains unchanged.

[0012] Furthermore, in step 1, the template image and search image sampled from the video sequence are first pre-processed and cropped to obtain the template region t and the search region S. Then, the template region T and the search region S are processed by the progressive image block embedding layer to obtain the template token sequence and the search region token sequence. Subsequently, the template token sequence and the search region token sequence are concatenated and input into the hybrid attention module of several layers for feature extraction. The output features are separated by length to obtain the search image features f s .

[0013] Furthermore, the formula for generating the Gaussian mask M is:

[0014]

[0015] M=Concat(m1,m2,…,m N ).

[0016] Among them, m i Indicates that the i-th trajectory prompts p i The generated Gaussian mask, c i For p i The center point of y represents the search image feature f s For each point on , σ represents the standard deviation; Concat refers to concatenation along the number of channels.

[0017] Furthermore, the trajectory prompts P pre It is composed of the predicted frame coordinates of N historical frames. During training, the real frame coordinates of the first frame search image are copied N times to obtain the initial track prompt; during testing, the real frame coordinates of the template image are used to initialize the track prompt.

[0018] Furthermore, the preprocessing in step 1 is a cropping operation, and the template scale factor T is set during cropping. factor and search image scale factor S factor , take the prediction frame of the previous search image as the center and crop the current search image to the crop size C sz and scaling factor R factor Obtained by the following calculation formula:

[0019]

[0020] R factor =S sz / C sz .

[0021] in, The predicted box of the t-1 frame is in the global coordinate system O global The width and height of the upper factor To search for image scale factor, S sz is the size of the search area;

[0022] The prediction box of the tth frame is transformed from the local coordinate system O t Mapped to the global common coordinate system O global The calculation method on is defined as follows:

[0023]

[0024] in, Is the predicted box of the current frame in the local coordinate system O t The center point coordinates and width and height in is the predicted box of the current frame in the global coordinate system O global The coordinate representation in ; Indicates that the prediction box of the previous frame is at O global The center point coordinates on , Δx and Δy represent the offset of the center point of the prediction box relative to the center of the search area.

[0025] Furthermore, the feature f with trajectory information obtained in step 2 p Input into the first head network Head0 to get a rough prediction box B coarse , based on the rough prediction box to generate a Gaussian mask, and f p Input together into the cascade attention layer to obtain the feature map f c , feature map f c The optimized prediction box B is obtained through the second head network Head1 ref ;

[0026] The structures of the head networks Head0 and Head1 are the same, both including a multi-scale feature module, a classification branch and a regression branch, and Head0 and Head1 do not share weights.

[0027] Furthermore, the multi-scale feature module includes multiple convolutional layers and upsampling layers. After convolution and upsampling operations, three feature maps with different resolutions and channel numbers are obtained. Then, three convolutional layers are used to change the channel numbers of these three feature maps with different resolutions to 256 to obtain the multi-scale feature f j , where j represents the scale;

[0028] The regression branch consists of an initial regression sub-branch and an optimized regression sub-branch. The initial regression sub-branch contains five convolutional layers, and the feature f output by the multi-scale feature module j The initial prediction box is obtained through the initial regression sub-branch The optimized regression sub-branch consists of a star-shaped deformable convolution and a convolution layer, according to the initial prediction box predicted by the initial regression sub-branch The offset of each sampling point can be calculated and input into the star-shaped deformable convolution together with the regression feature to obtain the adjusted regression feature. The regression feature passes through a convolution layer to obtain the offset of the initial prediction box. The offset is consistent with the initial prediction box. Multiplying them together gets the optimized prediction box, which is the final prediction result; the classification branch consists of four convolutional layers and a star-shaped deformable convolution, f j First, the classification features are obtained through three convolutional layers, and then the classification features are combined with the The calculated offsets are input together into the star-shaped deformable convolution to obtain classification features with prediction box information, which are then passed through a convolutional layer to predict the classification score with IoU awareness.

[0029] Furthermore, we use Varifocal Loss as the classification loss and train the classification branch to predict the classification score with IoU perception. For positive samples, the true label is the IoU between the predicted box and the true box; for negative samples, the true label is 0; SIoU loss is used as the regression loss. Both the initial regression loss and the optimized regression loss use SIoU loss. The loss of each head network is By classification loss Initial regression loss and optimize the regression loss It consists of three parts:

[0030]

[0031] Total training loss Composed of the losses of Head0 and Head1:

[0032]

[0033] Where k∈{0, 1}, λ cls ,λ init and λ ref is the weight.

[0034] Furthermore, in step 4, for each frame of the search image, the attention weight W of the search image relative to the center of the template image is obtained based on the attention weight W obtained from the feature extraction network. s , the sum of its mean and variance is the statistical threshold t w ; From Ws Select N f samples as statistical samples, and the mean of the similarity scores of the statistical samples is m f As the confidence score of the current search image area; if the confidence score m f With the statistical threshold t w Meet the conditions: m f ≥α2×t w , α2 is a constant, the current search image area is passed to the subsequent frame as a new dynamic template; if the condition is not met, the dynamic template remains unchanged.

[0035] The present invention also provides a task-specific target tracking system based on trajectory prompting, comprising:

[0036] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the task-specific target tracking method based on trajectory prompting as described in the above technical solution.

[0037] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0038] The task-specific target tracking method based on trajectory cues proposed in the present invention captures the target's motion trajectory by fusing the target position information of historical frames, thereby suppressing similar interferers and improving the tracking robustness of the tracker in complex scenes with interferers.

[0039] Furthermore, to address the misalignment between classification and regression tasks, the present invention designs a task-aligned head network. This connects the classification and regression branches through star-shaped deformable convolutions, enabling the classification branch to predict IoU-aware classification scores, helping to improve tracking accuracy. Furthermore, the cascaded bounding box optimization module constructed by the present invention effectively improves the tracker's ability to distinguish between foreground and background regions by limiting its attention to the foreground region and reducing background interference, further enhancing tracking accuracy.

[0040] Finally, this paper proposes for the first time a template update strategy based on statistical methods, which updates the template through the statistical characteristics of features and provides new templates to the tracker in a timely manner, helping the tracker to deal with challenges such as object deformation and rapid changes in appearance. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall network structure diagram of the present invention;

[0042] Figure 2 This is the structural diagram of the trajectory prompt module.

[0043] Figure 3It is a visualization of coordinate mapping, where (a) shows that the prediction box is mapped from the local coordinate system to the global coordinate system to maintain the motion trajectory, and (b) shows that the coordinate mapping is achieved by the offset between the center point of the prediction box and the center point of the prediction box in the previous frame.

[0044] Figure 4 This is the structural diagram of the head network. DETAILED DESCRIPTION

[0045] In order to make the technical solutions and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings.

[0046] This invention uses a trajectory-cueing-based tracking method to integrate target motion information into search image features, accurately locating the target using its motion trajectory. Furthermore, a cascaded bounding box optimization module is introduced to suppress background features, reducing background interference and improving bounding box regression accuracy. The invention uses star-shaped deformable convolutions to establish a connection between the classification and regression heads, enabling the classification head to output IoU-aware classification scores and mitigating the negative impact of misalignment between the classification and regression tasks on tracking performance. The invention also proposes a statistically-based template update strategy to propagate reliable appearance representations between consecutive frames, facilitating the tracker's handling of challenges such as object appearance changes and deformation. By integrating a trajectory-cueing module, a cascaded optimization structure, a task alignment framework, and a statistically-based template update strategy, the invention addresses existing target tracking methods, including issues such as trajectory prediction, insufficient prediction box accuracy, task misalignment, and template updates. This enables target tracking in challenging scenarios involving interfering objects, appearance changes, and object deformation.

[0047] like Figure 1 As shown in the figure, the target tracking method proposed in this invention mainly consists of three parts: a feature extraction network based on a hybrid attention module, a trajectory prompting module for encoding the target's historical position information, and a cascaded bounding box optimization module for locating the target. The specific implementation method is as follows:

[0048] In step 1, an asymmetric hybrid attention module is used to simultaneously extract features and exchange information between the template image and the search image. Compared with symmetric hybrid attention, asymmetric hybrid attention can reduce the amount of computation.

[0049] Step 2: Build a trajectory prompting module. Generate a Gaussian mask based on the predicted box of the historical frame. Perform attention calculation on the Gaussian mask and the search image features extracted in step 1 to obtain the search image features with motion information.

[0050] In step 3, the search image features with motion information obtained in step 2 are input into the cascade bounding box optimization module to obtain the tracking result. This process is mainly divided into three steps: the first step is to input the features obtained in step 2 into the first cascade head to obtain a rough prediction box; the second step is to generate a Gaussian mask based on the rough prediction box and perform a cascade attention calculation with the search image features with motion information. The target area is weighted and the weight of the background area is reduced to reduce the interference of the background area, so that the model pays more attention to the target area; the third step is to input the weighted search image features into the second cascade head to obtain the optimized prediction box.

[0051] In step 4, based on the attention weights obtained by the feature extraction network in step 1, the weight of the search image relative to the template center point is separated. The dynamic threshold and confidence score are calculated based on their statistical characteristics. If the confidence score is greater than or equal to 1.5 times the dynamic threshold, the current search image is considered reliable and can be used as the new dynamic template for subsequent frames; otherwise, the dynamic template remains unchanged.

[0052] Step 1: Feature extraction network extracts features. The template image and search image sampled from the video sequence are first pre-processed and cropped and scaled to and The size of the preprocessed template region T and search region S are then input into the feature extraction network to extract features. Specifically, the template region T and search region S are processed by the progressive image block embedding layer to obtain a size of The template token sequence and size are The search area token sequence is then concatenated and fed into an 11-layer hybrid attention module for feature extraction. t and W t Represents the height and width of the template area, H s and W s represents the height and width of the search area, and C represents the number of channels.

[0053] In this example, the template scale factor T is set when cropping the image. factor and search image scale factor S factor The template image size is 2.0 and 4.5 respectively. The cropping size is calculated by combining the template scale factor, the search image scale factor, and the size of the object in the image. t ×W t and search image size H s ×W sThey are 128×128 and 288×288 respectively, and the number of channels C is set to 768. The target tracking algorithm proposed in this invention uses ViT-B as the feature extraction network. Specifically, the cropped image is input into the model, and the image is first progressively encoded into blocks through three layers of image block embedding layers to obtain a token sequence. The first image block embedding layer uses a convolution kernel of size 4×4 with a step size of 4. The second and third image block embedding layers both use a convolution kernel of size 2×2 with a step size of 2. Subsequently, the spliced token sequence is input into the 11-layer ViT to extract features, and an asymmetric mixed attention operation is used at this time. Finally, the search image features can be separated by length from the output features.

[0054] Step 2: Build the track prompt module, such as Figure 2 As shown. Track prompt based on historical N frames Generate Gaussian mask And expand the Gaussian mask along the channel dimension into (Track Tips P pre It is composed of the predicted frame coordinates of N historical frames. During training, the real frame coordinates of the first frame search image are copied N times to obtain the initial track prompt; during testing, the real frame coordinates of the template are used to initialize the track prompt. In the subsequent tracking process, the "first in, first out" strategy is adopted to update the track prompt using the predicted frame coordinates of each frame search image. (N s =H s W s / 16 2 ) and Gaussian mask M are input into the trajectory attention layer (TA) and normalization layer (Norm) to obtain features containing trajectory information The process can be formulated as follows:

[0055]

[0056] Among them, Linear1 and Linear2 represent two linear layers used to obtain query Q and key K, Represents element-wise multiplication.

[0057] The formula for generating the Gaussian mask M is:

[0058]

[0059] Among them, m i Indicates that the i-th trajectory prompts p i The generated Gaussian mask, c i For p i The center point of y represents the feature map f sFor each point on , σ represents the standard deviation. Concat refers to concatenation along the number of channels.

[0060] The preprocessing operations on the template image and the search image in step 1 usually include shaking, cropping, scaling, translation, and flipping. These operations will destroy the motion continuity of the target in the video sequence, resulting in the output of each search image relying only on the local coordinate system of the current frame after preprocessing. In order to preserve the motion trajectory of the target, the present invention proposes a new cropping method, which takes the prediction box of the previous search image as the center and crops the current search image, eliminating operations such as translation and flipping. In addition, the present invention also proposes a new global common coordinate system O global , the motion trajectory of the target is captured by mapping the prediction boxes of different frames from different local coordinate systems to the global common coordinate system, such as Figure 3 shown.

[0061] Taking the search image of frame t as an example, its cropping size C sz and scaling factor R factor It can be obtained by the following calculation formula:

[0062]

[0063] in, The predicted box of the t-1 frame is in the global coordinate system O global The width and height of the upper factor is the scale factor of the search image, S sz is the size of the search area.

[0064] The prediction box of the tth frame is transformed from the local coordinate system O t Mapped to the global common coordinate system O global The calculation method on is defined as follows:

[0065]

[0066] in, Is the predicted box of the current frame in the local coordinate system O t The center point coordinates and width and height in is the predicted box of the current frame in the global coordinate system O global The coordinate representation in . Indicates that the prediction box of the previous frame is at O global The coordinates of the center point on . Δx and Δy represent the offset of the center point of the prediction box relative to the center of the search area.

[0067] In this example, the number of historical track cues N is set to 8. During training, the track cues P are initialized using the ground-truth box of the first search image. pre ; During testing, use the real box of the template image to initialize the trajectory prompt Ppre .

[0068] Step 3: Cascade the bounding box optimization module to get the predicted box. Figure 1 As shown in Figure 2, the cascaded bounding box optimization module consists of two heads (Head0, Head1) and a cascaded attention layer (CA). The feature f with trajectory information obtained in step 2 p Input into Head0 to get a rough prediction box B coarse , based on the rough prediction box to generate a Gaussian mask, and f p Together, we limit the range of attention through cascaded attention calculation, reduce the weight of the background area, and obtain the feature map f c The feature map f that suppresses the interference of background area c Get the optimized prediction box B through Head1 ref .

[0069] It is worth noting that in the cascade bounding box optimization module, the present invention uses a task-aligned network as the head network (Head0 and Head1 do not share weights). The task alignment network consists of a multi-scale feature module, a classification branch, and a regression branch. The regression branch consists of an initial regression sub-branch and an optimized regression sub-branch. In order to adapt to the scale change of the target and improve the robustness of tracking, the single-scale feature map is first input into a multi-scale feature module composed of multiple convolutional layers and upsampling layers. After convolution and upsampling operations, three feature maps with different resolutions and channel numbers are obtained. Subsequently, three convolutional layers are used to change the number of channels of these three feature maps with different resolutions to 256 to obtain a multi-scale feature. C′ represents the number of channels of the multi-scale feature, which is set to 256. Then, the multi-scale feature f j It is input into the classification branch and regression branch composed of multiple convolutional layers to obtain the prediction results. In order to alleviate the misalignment problem between the classification task and the regression task, the present invention uses star-shaped deformable convolution to establish the connection between the classification branch and the regression branch, so that the classification branch prediction has an IoU-aware classification score. Specifically, the regression branch consists of two sub-branches. The initial regression sub-branch contains five convolutional layers, f j The initial prediction box is obtained through the initial regression sub-branch The optimized regression sub-branch consists of a star-shaped deformable convolution and a convolution layer, according to the initial bounding box predicted by the initial regression sub-branch The offset of each sampling point can be calculated and input into the star-shaped deformable convolution together with the regression feature to obtain the adjusted regression feature. The feature passes through a convolution layer to obtain the offset of the initial prediction box. The offset is consistent with the initial prediction box. Multiply to get the optimized prediction box This improves the accuracy of bounding box regression. The classification branch consists of four convolutional layers and a star-shaped deformable convolution, f j First, the classification features are obtained through three convolutional layers, and then the classification features are combined with the above-mentioned The calculated offsets are input together into the star-shaped deformable convolution to obtain the classification feature with the prediction box information, which is predicted by a convolution layer to predict the classification score with IoU perception. This paper uses Varifocal Loss as the classification loss, and trains the classification branch to predict the classification score with IoU perception. For positive samples, the true label is the IoU between the predicted box and the true box; for negative samples, the true label is 0. SIoU loss is used as the regression loss. The loss of each head network is By classification loss Initial regression loss and optimize the regression loss It consists of three parts:

[0070]

[0071] Where k∈{0, 1}, λ cls ,λ init and λ ref are set to 1, 1, and 2 respectively. The total training loss Composed of the losses of Head0 and Head1:

[0072]

[0073] In this example, the multi-scale feature module outputs three features of different resolutions, f1, f2, and f3, whose sizes are 18×18, 36×36, and 72×72, respectively, and the number of channels C′ is 256. These three features of different resolutions are input into the head network to obtain the corresponding classification scores. and And the prediction box and For Head0, and The first 768 prediction boxes with the highest classification scores are selected as B coarse ; For Head1, non-maximum suppression (NMS) is used to and The final prediction result B is obtained ref .

[0074] Step 4: Template update based on statistical methods. In order to deal with the deformation and appearance changes of objects during tracking, this paper designs a template update strategy based on statistical methods to propagate reliable appearance representations between consecutive frames. For each search image, the attention weights obtained from the last hybrid attention layer of the feature extraction network are used. (N s =H s W s / 16 2 , N t =H t W t / 16 2 ), we can get the attention weight of the search image relative to the center of the template image The sum of its mean and variance is the statistical threshold t w From W s Select the N with the highest similarity score f samples as statistical samples, and the mean of the similarity scores of these statistical samples is m f As the confidence score of the current search image area, the confidence score of the search area reflects whether the prediction result of the current search area is reliable, that is, the accuracy of tracking, which can also be understood as B ref If the confidence score m f With the statistical threshold t w Meet the conditions: m f ≥α2×t w , the current search image area can be passed as a new dynamic template to the subsequent frames; if the condition is not met, the dynamic template remains unchanged.

[0075] In this example, N f by searching the number of region tokens N s And the proportional coefficient α1 is calculated to get: N f =α1×N f In this example, α1 and α2 are set to 0.7 and 1.5 respectively.

[0076] The present invention designs a trajectory prompt module to encode the historical position information of the target, thereby mining the target's motion trajectory, allowing the tracker to handle complex scenes with interference. In addition, the present invention also constructs a cascade bounding box optimization module, which limits the scope of attention by generating a Gaussian mask, reduces background interference, and enables the tracker to more accurately identify the target area, thereby improving the accuracy of the tracker prediction. The present invention uses a task-aligned prediction head and introduces IoU-aware classification scores to establish a connection between the classification task and the regression task, alleviating the negative impact of task misalignment and further improving the accuracy and robustness of tracking. Finally, a template update strategy based on statistical characteristics is proposed for the first time, which enables the tracker to capture reliable appearance representations in a timely manner and adapt to changes in the appearance and deformation of objects.

[0077] On the other hand, an embodiment of the present invention further provides a task-specific target tracking system based on trajectory prompting, including:

[0078] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the task-specific target tracking method based on trajectory prompting as described in the above technical solution.

[0079] It will be understood that the present invention is described by way of some embodiments, and those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and embodiments. They can be fully applied to various fields suitable for the present invention, and further modifications can be easily implemented by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A task-specific target tracking method based on trajectory cues, characterized in that: The steps include: Step 1: pre-process the template image and search image respectively, and input them into the feature extraction network for feature extraction to obtain the search image features; Step 2: Construct a trajectory hint module, generate a Gaussian mask based on the trajectory hints of the historical N frames, and expand the Gaussian mask along the channel dimension. Then, input the search image features and the expanded Gaussian mask into the trajectory attention layer and normalization layer to obtain features containing trajectory information. Step 3: construct a cascaded bounding box optimization module to process the features containing trajectory information to obtain the final prediction result; the cascaded bounding box optimization module includes two head networks and a cascaded attention layer; Step 4: Template update based on statistical methods: Based on the attention weights obtained by the feature extraction network in step 1, the weight of the search image relative to the center point of the template image is separated. The dynamic threshold and the confidence score of the current search image area are calculated according to the statistical characteristics. When the confidence score and the dynamic threshold meet certain conditions, the current search image area is passed to the subsequent frame as the new dynamic template; if the conditions are not met, the dynamic template remains unchanged.

2. The task-specific target tracking method based on trajectory prompting according to claim 1, wherein: In step 1, the template image and search image sampled from the video sequence are first preprocessed and cropped to obtain the template region T and the search region S. Then, the template region T and the search region S are processed by the progressive image block embedding layer to obtain the template token sequence and the search region token sequence. Subsequently, the template token sequence and the search region token sequence are concatenated and input into the hybrid attention module of several layers for feature extraction. The output features are separated by length to obtain the search image features f s .

3. The task-specific target tracking method based on trajectory prompting according to claim 1, wherein: The formula for generating the Gaussian mask M is: M=Concat(m1,m2,…,m N ). Among them, m i Indicates that the i-th trajectory prompts p i The generated Gaussian mask, c i For p i The center point of y represents the search image feature f s For each point on , σ represents the standard deviation; Concat refers to concatenation along the number of channels.

4. The task-specific target tracking method based on trajectory prompting according to claim 1, wherein: Track Tips P pre It is composed of the predicted frame coordinates of N historical frames. During training, the real frame coordinates of the first frame search image are copied N times to obtain the initial track prompt; during testing, the real frame coordinates of the template image are used to initialize the track prompt.

5. The task-specific target tracking method based on trajectory prompting according to claim 1, wherein: The preprocessing in step 1 is a cropping operation, and the template scale factor T is set during cropping. factor and search image scale factor S factor , take the prediction frame of the previous search image as the center and crop the current search image to the crop size C sz and scaling factor R factor Obtained by the following calculation formula: R factor =S sz / C sz · in, The predicted box of the t-1 frame is in the global coordinate system O global The width and height of the upper factor To search for image scale factor, S sz is the size of the search area; The prediction box of the tth frame is transformed from the local coordinate system O t Mapped to the global common coordinate system O global The calculation method on is defined as follows: in, Is the predicted box of the current frame in the local coordinate system O t The center point coordinates and width and height in, is the predicted box of the current frame in the global coordinate system O global The coordinate representation in ; Indicates that the prediction box of the previous frame is at O global The center coordinates on , Δx and Δy represent the offset of the center point of the prediction box relative to the center of the search area.

6. The task-specific target tracking method based on trajectory prompting according to claim 1, wherein: The feature f with trajectory information obtained in step 2 p Input into the first head network Head0 to get a rough prediction box B coarse , based on the rough prediction box to generate a Gaussian mask, and f p Input together into the cascade attention layer to obtain the feature map f c , feature map f c The optimized prediction box B is obtained through the second head network Head1 ref ; The structures of the head networks Head0 and Head1 are the same, both including a multi-scale feature module, a classification branch and a regression branch, and Head0 and Head1 do not share weights.

7. The task-specific target tracking method based on trajectory prompting according to claim 6, characterized in that: The multi-scale feature module includes multiple convolutional layers and upsampling layers. After convolution and upsampling operations, three feature maps with different resolutions and channel numbers are obtained. Then, three convolutional layers are used to change the channel numbers of these three feature maps with different resolutions to 256 to obtain the multi-scale feature f j , where j represents the scale; The regression branch consists of an initial regression sub-branch and an optimized regression sub-branch. The initial regression sub-branch contains five convolutional layers, and the feature f output by the multi-scale feature module j The initial prediction box is obtained through the initial regression sub-branch The optimized regression sub-branch consists of a star-shaped deformable convolution and a convolution layer, according to the initial prediction box predicted by the initial regression sub-branch The offset of each sampling point can be calculated and input into the star-shaped deformable convolution together with the regression feature to obtain the adjusted regression feature. The regression feature passes through a convolution layer to obtain the offset of the initial prediction box. The offset is consistent with the initial prediction box. Multiplying them together gets the optimized prediction box, which is the final prediction result; the classification branch consists of four convolutional layers and a star-shaped deformable convolution, f j First, the classification features are obtained through three convolutional layers, and then the classification features are combined with the The calculated offsets are input together into the star-shaped deformable convolution to obtain classification features with prediction box information, which are then passed through a convolutional layer to predict the classification score with IoU awareness.

8. The task-specific target tracking method based on trajectory prompting according to claim 6, wherein: Using Varifocal Loss as the classification loss, the classification branch is trained to predict the classification score with IoU perception. For positive samples, the true label is the IoU between the predicted box and the true box; for negative samples, the true label is 0; using SIoU loss as the regression loss, both the initial regression loss and the optimized regression loss use SIoU loss, and the loss of each head network is By classification loss Initial regression loss and optimize the regression loss It consists of three parts: Total training loss Composed of the losses of Head0 and Head1: Where k∈{0, 1}, λ cls ,λ init and λ ref is the weight.

9. The task-specific target tracking method based on trajectory prompting according to claim 1, wherein: In step 4, for each frame of the search image, the attention weight W of the search image relative to the center of the template image is obtained based on the attention weight W obtained from the feature extraction network. s , the sum of its mean and variance is the statistical threshold t w ; From W s Select N f samples as statistical samples, and the mean of the similarity scores of the statistical samples is m f As the confidence score of the current search image area; if the confidence score m f With the statistical threshold t w Meet the conditions: m f ≥α2×t w , α2 is a constant, the current search image area is passed to the subsequent frame as a new dynamic template; if the condition is not met, the dynamic template remains unchanged.

10. A task-specific target tracking system based on trajectory cues, characterized in that: include: A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the task-specific target tracking method based on trajectory prompting as described in any one of claims 1 to 9.

Images