A target tracking method and system based on color cue embedding and position constraint

Abstract

The application discloses a target tracking method and system based on color prompt embedding and position constraint, relates to the technical field of computer vision, and comprises the following steps: extracting the depth feature of a target search image according to a preset deep learning network; extracting a color histogram of the target search image through a color prompt encoder and converting the color histogram into a color target probability map, and obtaining a mapping feature through feature mapping after preprocessing; obtaining a color prompt target perception feature map according to the mapping feature and the depth feature, inputting a model predictor, and obtaining an appearance model; processing a test set image through a feature extractor, obtaining a target response map through the appearance model, determining the position of a next frame of the target according to the target peak position of the target response map, and updating the model predictor through an HNM algorithm combined with a target constraint mechanism to realize real-time updating of the appearance model. The color prompt encoder and the position constraint mechanism are added to the deep learning network, so that the model updated in real time is more accurate.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a target tracking method and system based on color cues embedding and positional constraints. Background Technology

[0002] Visual object tracking, as an important direction in computer vision, aims to predict the state of a target in a video sequence, especially when only the initial target state is provided. With the rapid development of deep learning in this field, the two currently popular trackers based on discriminant correlation filters (DCF) and Siamese networks have made significant progress. However, visual tracking tasks have always been plagued by some complex challenges, such as cluttered backgrounds, scale variations, and similar interfering objects.

[0003] Currently, DCF and Siamese network frameworks are widely favored by researchers. Both core ideologies utilize trainable deep networks to learn appearance models of arbitrary target objects online or offline. Siamese network-based trackers (SN) typically consist of two sub-branches, each taking a template and a search sample as input. Specifically, the template is twice the size of the search region. SN can be viewed as a template matching process, matching the most similar template from the search region. SN achieves commendable tracking accuracy and speed on such a concise network structure, greatly advancing the application of deep learning in visual tracking. The entire tracking architecture of SN can be subdivided into three parts: feature extraction, feature fusion, and prediction head. Regarding feature extraction, SN initially built upon AlexNet, subsequently employing backbone networks ranging from CNNs (e.g., VGG, ResNet) to visual Transformers (e.g., ViT, SwinT) to extract more robust feature representations. Furthermore, by employing different feature fusion techniques and anchor-based or anchorless prediction heads, SN continuously sets new state-of-the-art records on multiple benchmarks. Following the successful application of Transformers in computer vision tasks, neural networks using CNN feature extraction and Transformer feature fusion have been proposed, and the feasibility of a single-branch architecture using ViT to jointly perform feature extraction and fusion has been demonstrated. However, SN utilizes limited background information to learn the appearance model, and this approach typically only uses the initial target as a template without updating it. Compared to DCF-based trackers, SN is less prone to accumulated errors during the inference phase, but is relatively susceptible to drastic appearance changes and the influence of similar objects.

[0004] In comparison, DCF can draw support from more background information for discriminative model prediction, especially since recent methods like SuperDiMP and ToMP can achieve end-to-end training like SN. Furthermore, DCF requires an online update mechanism to determine whether the prediction results dynamically update the training sample set. This invention divides DCF into four parts: feature extraction, target localization, target scale estimation, and online update. Although DCF-based trackers have achieved impressive performance on mainstream benchmarks, this invention recognizes two areas for improvement in these tracking methods: 1) Feature extraction has always been a primary task before tracking, and excellent feature representation is beneficial for training appearance models. ResNet, as a popular CNN backbone network, can be considered the preferred feature extractor, rather than handcrafted methods (e.g., HOG, color features). How to transform pre-trained feature attributes from image classification into more suitable feature attributes for visual tracking has always been a research topic worth exploring. 2) Another challenging issue in DCF is how to effectively update the model online. Since the target changes dynamically during tracking, it is reasonable to update the tracking model appropriately to adapt to its appearance changes. In this regard, linear interpolation or stored sample optimization schemes are commonly used. Regardless of the method used, blindly updating the model with predicted targets is unreasonable and leads to model drift. Therefore, recent DCF methods (such as SuperDiMP and ToMP) employ a hard negative sample mining (HNM) strategy to control model updates. The HNM method evaluates the target tracking state through target location and response map, but does not consider the motion inertia of the tracked target in the video sequence. Summary of the Invention

[0005] In view of this, the present invention provides a target tracking method and system based on color cues embedding and position constraints to solve the problems existing in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A target tracking method based on color cue embedding and position constraints includes: acquiring a target search image and extracting a color cue target perception feature map using a feature extractor;

[0008] The feature extractor includes: extracting depth features of the target search image according to a preset deep learning network, and processing the depth features using a CBR module to obtain processed features; dividing the target search image into foreground and background regions using a color cue encoder, extracting and processing the color histograms of the foreground and background regions to obtain a color target probability map; smoothing the color target probability map using a Hamming window to obtain a color cue feature map, scaling the color cue feature map using bilinear interpolation and performing feature mapping to obtain mapped features; and element-wise multiplying the mapped features with the processed features and performing normalization to obtain a color cue target perception feature map.

[0009] The color-coded target perception feature map is input into the model predictor to obtain the appearance model.

[0010] A test set is established by acquiring the target image to be searched. The test set images are processed by the feature extractor, and the target response map is obtained by the appearance model. The position of the target in the next frame is determined based on the target peak position of the target response map. Then, the scale is estimated, and the model predictor is updated by combining the HNM algorithm with the target constraint mechanism to realize the real-time update of the appearance model.

[0011] Preferably, extracting the color histograms of the foreground and background regions specifically includes dividing the foreground and background regions according to the target state s in the target search image, i.e. and in, This indicates that only the color histogram inside the rectangle is counted, and This indicates that only the color histogram outside the rectangle is counted, and the color histograms of the foreground and background areas are extracted separately. The formula is:

[0012]

[0013] Where β is the histogram, x im Indicates an image patch, x im ∈[0,255], This represents the RGB image region of the foreground or background area. CalHis(·) counts the identical values ​​of each pixel in the image, and sum(·) represents the summation function.

[0014] Preferably, obtaining the color target probability map specifically includes processing the foreground histogram. and background histogram After processing, a color target probability map is obtained, using the following formula:

[0015]

[0016] in, Let λ represent the probability of a colored target, and λ be a hyperparameter.

[0017] Preferably, obtaining the mapped features specifically includes performing bilinear interpolation to adjust the shape of the color cue feature map to the shape of the depth features, and then using three fully convolutional layers and the sigmoid function for mapping, as shown in the formula:

[0018]

[0019] in, Here, γ represents the color-coded feature map, FC represents the resizing operation, and FC is a fully convolutional layer.

[0020] Preferably, the acquisition of the color cue target perception feature map specifically includes:

[0021]

[0022] in, This indicates CBR. To make the feature hints smoother, Conv1×1 processing is used, x im Indicates an image patch. In this diagram, R represents the backbone network, and the subscript c4 indicates the 4th mapping convolutional layer at the network output. Finally, InstanceNorm is used to normalize the features to obtain the color-coded target-aware feature map. The formula is as follows:

[0023] x f =N(c 1 (f prompt ));

[0024] Among them, c 1 This represents Conv1×1, and N represents the InstanceNorm layer.

[0025] Preferably, the model predictor The objective function is formulated as follows:

[0026]

[0027] Where r(·) is the residual function, β is a scalar, f(·) is the regularization term, and r(·) is used to calculate the response plot. And Gaussian label y g The error between them, * indicates convolution operation, x f Training samples are used as target features. This is the appearance model.

[0028] Preferably, the target constraint mechanism specifically includes:

[0029]

[0030] Where d is the displacement ratio, (w,h) is the initial target size, and π is the constraint radius.

[0031] A target tracking system based on color cues embedding and position constraints includes:

[0032] The feature map acquisition module acquires the target search image and extracts the color cue target perception feature map based on the feature extractor.

[0033] The feature extractor includes: extracting depth features of the target search image according to a preset deep learning network, and processing the depth features using a CBR module to obtain processed features; dividing the target search image into foreground and background regions using a color cue encoder, extracting and processing the color histograms of the foreground and background regions to obtain a color target probability map; smoothing the color target probability map using a Hamming window to obtain a color cue feature map, scaling the color cue feature map using bilinear interpolation and performing feature mapping to obtain mapped features; and element-wise multiplying the mapped features with the processed features and performing normalization to obtain a color cue target perception feature map.

[0034] The appearance model acquisition module inputs the color cue target perception feature map into the model predictor to acquire the appearance model.

[0035] The update module acquires the target image to be searched to establish a test set, processes the test set images through the feature extractor, obtains the target response map through the appearance model, determines the target's position in the next frame based on the target peak position of the target response map, then performs scale estimation, and updates the model predictor through the HNM algorithm combined with the target constraint mechanism to achieve real-time updating of the appearance model.

[0036] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a target tracking method and system based on color cue embedding and position constraints. It uses histogram statistics of RGB images to construct color target probability features, realizing the effective fusion of shallow color features and deep CNN features. Inspired by cue learning, the present invention regards the color target probability as a cue and encodes the cue as a residual feature to guide the acquisition of target perception features. In order to reduce the impact of inaccurate tracking results on the online model update process, a position offset constraint method is proposed to force the tracked target to satisfy the traditional motion inertia. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 A schematic diagram of the color-coded encoder structure provided by the present invention;

[0039] Figure 2 An unfolded diagram of the color-coded encoder provided by this invention;

[0040] Figure 3 A schematic diagram of the probability of a visual color target provided by the present invention;

[0041] Figure 4 A visual schematic diagram of the position constraint mechanism provided by the present invention;

[0042] Figure 5 The diagram shows an improved SuperDiMP-based tracker provided by this invention.

[0043] Figure 6 The challenge sequence visualization results provided by this invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] This invention discloses a target tracking method based on color cue embedding and position constraints, comprising: acquiring a target search image, and extracting a color cue target perception feature map using a feature extractor;

[0046] The feature extractor includes: extracting depth features of the target search image based on a preset deep learning network, and processing the depth features using a CBR module to obtain processed features; dividing the target search image into foreground and background regions using a color cue encoder, extracting and processing the color histograms of the foreground and background regions to obtain a color target probability map; smoothing the color target probability map using a Hamming window to obtain a color cue feature map, scaling the color cue feature map using bilinear interpolation and performing feature mapping to obtain mapped features; and element-wise multiplying the mapped features with the processed features and performing normalization to obtain a color cue target perception feature map.

[0047] Input the color-coded target perception feature map into the model predictor to obtain the appearance model;

[0048] A test set is established by acquiring the target image to be searched. The test set images are processed by a feature extractor, and the target response map is obtained by the appearance model. The position of the target in the next frame is determined based on the target peak position in the target response map. Then, the scale is estimated, and the model predictor is updated by combining the HNM algorithm with the target constraint mechanism to achieve real-time updating of the appearance model.

[0049] In one specific embodiment, after determining the target's position in the next frame and obtaining the target position and response map, the reliability of the prediction can be determined based on the HNM (Hidden Number Map).

[0050] In one specific embodiment, to correctly integrate color cues into the deep network, a color cue encoder (CPE) is used to encode the color cues into the ResNet, such as... Figure 1-2 As shown, this guides the generation of target-aware feature maps. CTP is obtained through foreground and background probability maps. First, this invention divides the foreground and background regions according to the target state s ( Figure 2 (The red rectangle in the middle) and here, This indicates that only the color histogram inside the rectangle is counted, and This indicates that only the color histogram outside the bounding box is counted. Extracting the color histograms of the foreground and background regions specifically involves dividing the foreground and background regions based on the target state s in the target search image. and in, This indicates that only the color histogram inside the rectangle is counted, and This indicates that only the color histogram outside the rectangle is counted, and the color histograms of the foreground and background areas are extracted separately. The formula is:

[0051]

[0052] Where β is the histogram, with a default size of 16, and x... im Indicates an image patch, x im ∈[0,255], This represents the RGB image region of the foreground or background area. CalHis(·) counts the identical values ​​of each pixel in the image, and sum(·) represents the summation function.

[0053] In one specific embodiment, obtaining the color target probability map specifically includes processing the foreground histogram. and background histogram After processing, the Color Target Probability Map (CTP) is obtained, using the following formula:

[0054]

[0055] in, This represents the probability of a colored target, where λ is a hyperparameter (default value is 0.01).

[0056] To eliminate interference from background probability information, this invention uses a Hamming window for smoothing. Obtain color cue feature map

[0057] In one specific embodiment, obtaining the mapping features specifically includes: obtaining the color cue feature map. Afterwards, as Figure 3 The image shown consists of foreground and background probability maps. (a) Image patch, (b) Average foreground probability, (c) Average background probability, (d) Average color target probability. The invention uses a color histogram method to convert the RCB image (a) into (b)-(d) maps. (e) Color target probability, and also visualizes the R, G, and B channels of CTP; the upper right corner represents the color values ​​of the visualized feature maps in R, G, B of the color target probability maps bd and e. Larger values ​​indicate better feature attention. Since the shape is the same as the original RGB image, the width and height of the depth features are downsampled with a stride of 16. A bilinear interpolation method is used to adjust the shape of the color cue feature map to the shape of the depth features. Then, three fully convolutional layers and a Sigmoid function are used for mapping. The formula is:

[0058]

[0059] in, This is a color cue feature map, γ represents the resizing operation, FC is a fully convolutional layer, and the channels of the FC layer are [3, 16, 16, 1], where 3 is the input channel, 16 is the hidden channel, and 1 is the output channel. For example... Figure 1 As shown, the FC layer consists of two CBRs (i.e., Conv3×3+BN+ReLU) and a 1×1 convolution (i.e., Conv1×1).

[0060] In one specific embodiment, obtaining the color cue target perception feature map specifically includes:

[0061]

[0062] in, This indicates CBR. To make the feature hints smoother, Conv1×1 processing is used, x im Indicates an image patch. In this diagram, R represents the backbone network, and the subscript c4 indicates the 4th mapping convolutional layer at the network output. Finally, InstanceNorm is used to normalize the features to obtain the color-coded target-aware feature map. The formula is as follows:

[0063] x f =N(c 1 (f prompt ));

[0064] Among them, c 1 This represents Conv1×1, and N represents the InstanceNorm layer.

[0065] In one specific embodiment, the model predictor The objective function is formulated as follows:

[0066]

[0067] Where r(·) is the residual function, β is a scalar, f(·) is the regularization term, and r(·) is used to calculate the response plot. And Gaussian label y g The error between them, * indicates convolution operation, x f Training samples are used as target features. This is the appearance model.

[0068] In one specific implementation, most DCF-based trackers find feasible target locations by generating robust response maps. However, the appearance models of these trackers are updated online based on initial annotation frames and prediction results, making the filtering of prediction results crucial. Although SuperDiMP achieves considerable performance using the HNM method, the results obtained from the response map are not necessarily reliable in complex scenes. Furthermore, the HNM method outputs the final position offset and target state from the response map, but suffers from insufficient utilization of the position offset. Figure 4 As can be seen, PCM is mainly based on the target position in the previous frame. The target's motion range is determined by the preset constraint radius π (i.e., the red circle). If the target position in the next frame... If the target is outside the circle, its movement is considered abnormal, as shown in the formula:

[0069]

[0070] Here, o p This represents the position offset. To better measure the effectiveness of the position offset, a constraint radius is set based on the initial target scale (w, h). The target constraint mechanism (PCM) specifically includes:

[0071]

[0072] Where d is the displacement ratio, which defaults to 0.8, (w,h) is the initial target size, and π is the constraint radius. In this PCM, o p >p indicates that it is not recommended to use the prediction results to update the sample set S. train .

[0073] Identifying tracking anomalies based on the positional offset between the previous target location and the current predicted location, such as... Figure 4 As shown, the search region (blue rectangle), target state (green rectangle), and constraint radius (red circle) in image frame #0010 of the Basketball sequence for OTB100 provide an intuitive explanation. The red circle is drawn with the constraint radius determined based on the target position in the previous frame as the center. The concept of PCM proposed in this invention is that the target in the next frame is located outside the circle to indicate a tracking anomaly, rather than being considered as a model update sample. PCM is intended to provide a complementary clue to the HNM method, which modulates the update of sample memory and model learning rate (default 0.01). The sample memory is used to store initial and predicted samples, and its capacity is set to a maximum of 50. Furthermore, memory is added or replaced in normal and hard negative cases, and the hard negative state will double the model learning rate (i.e., 0.02). If the phase response graph... If the peak value is below 0.25, the "not_found" state considers the target lost. For uncertain states, the target state will still be output normally, but no update operation will be performed. Due to the complexity of tracking conditions, HNM cannot cover all aspects. Therefore, by following the position offset constraint method based on motion inertia, the reliability of updated samples can be ensured as high as possible. The purpose of HNM is to control the storage and replacement of training frame sample sets and the update of the appearance model's learning rate. Sample storage includes annotated initial frames and their data augmentations, as well as later predicted frames with pseudo-labels. Furthermore, in normal and hard-negative sample states, stored samples may be stored or replaced; the hard-negative state will double the learning rate. If the peak value is below 0.25, the "not_found" state considers the target lost. The "uncertain" state will output the predicted target state normally, but no update control will be performed.

[0074] The above approach addresses the case of a single frame. In a specific embodiment, for multi-frame position offsets, the previous position storage space is first initialized and updated using a first-in, first-out (FIFO) method. Subsequently, the present invention calculates the current frame... With multiple location sets stored in the next frame The positional offsets between the values ​​are used to obtain a set of positional offsets, and finally the average value is compared with the constraint radius π.

[0075] In one specific embodiment, the improved tracker employs state-of-the-art SuperDiMP as the baseline tracker, which includes a DiMP target localization module and a PrDiMP probabilistic bounding box regression module. Specifically, SuperDiMP can be decomposed into four stages: feature extraction, target localization, target scale estimation, and online model update. First, image frames are preprocessed and fed into a backbone network to extract depth feature maps. The backbone network uses a pre-trained ResNet50 to acquire conv4_x feature maps as image features. Furthermore, the network weights before the conv4_x layers are frozen. To make the features obtained by the image classification network more suitable for visual tracking, SuperDiMP adds a convolutional block to obtain the learned feature expression. In this section, the invention proposes a Color Cue Encoder (CPE) designed to encode color information as cues to generate target-aware feature maps. CPE can provide timely guidance and target attention capabilities. The feature extraction stage, consisting of the backbone network and the CPE module, is formulated as follows:

[0076]

[0077] Here, x im Indicates image patch (e.g.) Figure 5 The test frames in the image. R represents the backbone network (e.g., ResNet50), and the subscript indicates which layer of features the network output maps to (e.g., c4 represents conv4_x). This indicates the CPE module.

[0078] in, Figure 5 An illustration summarizing the color prompt encoder (CPE) and position constraint mechanism (PCM) of the present invention, based on an improvement of the SuperDiMP tracker.

[0079] The feature extraction component of CPE proposed in this invention can obtain more diverse and specific target features x. f These features are used by the target localization module to obtain the appearance model (i.e., the convolutional kernel). (From the model predictor) The appearance model α can output the expected object score. (e.g., Gaussian labels). Training samples Input into the model predictor To obtain the appearance model n represents the total number of training samples. Model predictor The objective function formula is

[0080]

[0081] Here, the objective function includes the residual function r(·) and the regularization term f(·) weighted by the scalar β. r(·) is used to calculate the response plot. And Gaussian label y g The error between them, * indicates the convolution operation.

[0082] In fact, the convolution kernel is similar to the correlation filter of the original DCF, which affects the feature map of the test frame. Perform convolution to obtain the response map Then The coordinates of the peak (x) l ,y l The target location is determined as the position of the target in the next frame. After target localization, ATOM IoUnet uses the coordinates (x, y) to determine the target position. l ,y l Perform target scale estimation. Unlike target localization, target scale estimation uses conv3_x and conv4_x features to estimate the final target state, i.e. This represents the target scale estimation module. Since DCF-based trackers allow storing target samples to update the appearance model online, further target identification and selection of better samples for storage and control of online model updates are crucial. Therefore, SuperDiMP employs the HNM method to handle this. While the HNM method achieves significant improvements, it is often insufficient in complex scenes. As a supplement, this invention designs a position constraint mechanism (PCM) to constrain the positional offset of the tracked target relative to conventional motion inertia.

[0083] After training the appearance model, when using it, first obtain the target search map based on the target state of the previous frame. Then, extract the color cue target perception feature map from the prediction image frame using the target search map feature extractor. Convolve this feature map with the appearance model to obtain the response map. The peak value of the response map is the position of the target in the next frame. Then, the target scale estimation method is used to estimate the scale. After obtaining the new target state, HNM and the PCM of this invention are used to judge the accuracy of the predicted target and control the online update of the appearance model.

[0084] A target tracking system based on color cues embedding and position constraints includes:

[0085] The feature map acquisition module acquires the target search image and extracts the color cue target perception feature map based on the feature extractor.

[0086] The feature extractor includes: extracting depth features of the target search image based on a preset deep learning network, and processing the depth features using a CBR module to obtain processed features; dividing the target search image into foreground and background regions using a color cue encoder, extracting and processing the color histograms of the foreground and background regions to obtain a color target probability map; smoothing the color target probability map using a Hamming window to obtain a color cue feature map, scaling the color cue feature map using bilinear interpolation and performing feature mapping to obtain mapped features; and element-wise multiplying the mapped features with the processed features and performing normalization to obtain a color cue target perception feature map.

[0087] The appearance model acquisition module inputs the color cue target perception feature map into the model predictor to obtain the appearance model;

[0088] The update module acquires the target image to be searched to establish a test set, processes the test set images through a feature extractor, obtains the target response map through the appearance model, determines the target's position in the next frame based on the target peak position in the target response map, then performs scale estimation, and updates the model predictor through the HNM algorithm combined with the target constraint mechanism to achieve real-time updating of the appearance model.

[0089] The CPE and PCM modules of this invention can be easily integrated with recent DCF-based frameworks such as SuperDiMP and ToMP. This invention uses the CPE module instead of the original ClsFeat for end-to-end training. The target classification features obtained through the method of this invention are fed into the model predictor to obtain a stronger appearance model. Except for the CPE module, the training setup of this invention is consistent with the SuperDiMP and ToMP trackers, but the training device of this invention is two NVIDIA GeForce RTX 3090 GPUs. Furthermore, the PCM method uses the predicted position from the model predictor and has no training requirements, which can be considered a screening method to enhance the stability of model updates.

[0090] In one specific embodiment, to verify the practicality of the CPE and PCM modules designed in this invention, they are integrated into recent DCF-based trackers (i.e., SuperDiMP and ToMP). The improved tracker of this invention demonstrates comparative performance on six challenging benchmarks, including OTB100, NFS, UAV123, LaSOT, TrackingNet, and VOT2020. The method of this invention is implemented in Python using PyTorch, and all experiments were run on a GeForce RTX 4070 GPU processor.

[0091] Performance on different datasets:

[0092] LaSOT: Table 1 shows the comparison results of the Precision and AUC scores of various trackers. The LaSOT dataset contains mostly long video sequences, with an average of 2500 frames per video sequence, making it more challenging than other datasets. Therefore, the color cue encoder and position constraint mechanism proposed in this invention are designed to enhance and maintain the discriminative power of the target classifier. This is crucial for long video sequences, as the longer the video sequence, the more likely the model is to be contaminated by erroneous background information. This invention selects the most recent DiMP, PrDiMP, SuperDiMP, TrDiMP, TransT, STARK-ST50, Sim-B / 32, CSWinTT, and ToMP50 for comparison. The CPE and PCM methods proposed in this invention are integrated into SuperDiMP and ToMP50, achieving competitive performance, outperforming SuperDiMP by 2.1% and 1.4% (%), respectively. Furthermore, ProToMP demonstrates competitive performance compared to Transformer-based trackers such as TrDiMP, TransT, STARK-ST50, and ToMP50. The results show significant improvements on purely CNN-based trackers (such as SuperDiMP), while Transformer-based trackers do not show significant improvements on this dataset. However, ProToMP still achieves good results. In conclusion, the proposed method is significant in improving tracking performance.

[0093] Table 1 Comparison Results of Trackers

[0094]

[0095]

[0096] The ProDiMP and ProToMP trackers of this invention are compared with state-of-the-art trackers on the LaSOT, TrackingNet, OTB100, NFS, and UAV123 datasets in terms of accuracy (Prec.), normalized accuracy (Norm.Prec.), and AUC score (%).

[0097] The TrackingNet dataset test set was also used to test the method of this invention, which consists of 511 video sequences. Experimental results are shown in Table 1. The ProToMP tracker of this invention achieves an AUC score of 81.8%, outperforming state-of-the-art trackers such as TrDiMP, TransT, STARK-ST50, Sim-B / 32, and ToMP50. Furthermore, compared to SuperDiMP, the ProDiMP of this invention improves upon SuperDiMP, with a relative AUC gain of 0.4%, and increases in accuracy and normalized accuracy by 0.7 and 0.4%, respectively. Moreover, compared to ToMP50, the ProToMP of this invention also achieves competitive performance, contributing relative increases in accuracy, normalized accuracy, and AUC of ToMP50 by 1.0, 0.5, and 0.6%, respectively. These competitive results on this dataset further demonstrate the effectiveness of the proposed method.

[0098] OTB100: OTB100 consists of 98 video sequences and 100 tracked targets. Table 1 provides the AUC scores of ProDiMP and ProToMP from this invention compared to state-of-the-art trackers. As can be seen from the table, the AUC scores of ProDiMP and ProToMP from this invention are 70.4 and 70.1%, respectively, exceeding those of SuperDiMP and ToMP50, with AUC gains of 0.3 and 0.1%, respectively. Furthermore, ProDiMP outperforms recent state-of-the-art trackers (e.g., STARK-ST50, TransT). Additionally, DiMP, PrDiMP, STARK, and ToMP use ResNet-50 as the backbone by default for fair comparison. Experimental results further demonstrate the effectiveness of the proposed method.

[0099] NFS: This invention evaluates the tracker on a 30 frames / second version of the dataset. This dataset consists of 100 video sequences with fast-moving targets. As shown in Table 1, the ProToMP method of this invention achieves state-of-the-art performance with an AUC score of 67.5%, exceeding ToMP50 by 0.6%. Furthermore, the method of this invention also outperforms recent methods like TransT and STARK.

[0100] UAV123: The UAV dataset consists of 123 aerial videos and is designed to test trackers suitable for UAVs. Table 1 shows that the AUC scores of the ProDiMP and ProToMP trackers of this invention are 69.2 and 68.9%, respectively, achieving competitive performance compared to other state-of-the-art methods (e, g, STARK). Although ProToMP's tracking performance is inferior to CSWinTT and ToMP50, the ProDiMP of this invention outperforms SuperDiMP by 1.5%.

[0101] VOT2020: This invention evaluated its ProDiMP and ProToMP trackers in the 2020 version of the Visual Object Tracking (VOT) Short-Term Challenge. Although the dataset contains 60 videos with segment mask annotations, this invention only compares them with trackers that predict bounding boxes, as this invention's tracker will predict the bounding boxes. Trackers were evaluated in a multi-start protocol and ranked based on the EAO metric, which considers tracking accuracy (average overlap of successful frames) and robustness (failure rate). The results in Table 2 show that ProDiMP achieves first-class accuracy and EAO. ProToMP achieves first-class robustness. In particular, ProDiMP achieves robustness of 0.798 and EAO of 0.311, outperforming SuperDiMP in both robustness (+7%) and EAO (+0.6%). ProToMP achieves robustness of 0.815 and EAO of 0.311, outperforming ToMP50 in both robustness (+2.6%) and EAO (+0.8%).

[0102] Table 2 Comparison Results

[0103]

[0104]

[0105] Recent comparison of bounding box-only methods on VOT2020ST in terms of accuracy (A), robustness (R), and expected average overlap (EAO).

[0106] Qualitative analysis:

[0107] The above experiments primarily demonstrate the effectiveness of the proposed method from a quantitative perspective, and show that it can effectively improve tracking results. Furthermore, this invention also qualitatively demonstrates… Figure 5Tracking performance of representative video sequences. This invention visualizes the tracking results of SuperDiMP, ToMP50, and the trackers of this invention (i.e., ProDiMP and ProToMP) on four challenging sequences from OTB100 and LaSOT. This invention selects two video sequences (Basketball and Skating2_1) from the short-term OTB100 dataset and two image sequences (dog-15 and zebra-17) from the long-term LaSOT dataset for qualitative analysis. Figure 6 As shown, the number of video frames in the four video sequences are 725, 473, 5009, and 2964, respectively. This invention displays four representative frames, with the first frame showing the bounding box of the tracked target. This can be seen from... Figure 6 As observed, the ProDiMP and ProToMP trackers can accurately predict target objects despite occlusion, deformation, and background clutter. In frame 0470 of the Basketball sequence, the tracker of this invention misidentifies similar distracting objects as targets, but SuperDiMP and ToMP50 may also obtain incorrect bounding boxes. In the Skating2_1 sequence, the tracker of this invention can generally track female players even with distracting objects, but the SuperDiMP and ToMP50 methods of this invention are affected by background clutter. In frame 0110 of the Dog-15 sequence, SuperDiMP loses tracked targets and misidentifies a dog in a mirror as a target. In the Zebra-17 sequence, SuperDiMP and ToMP50 are not robust enough and may be affected by similar objects and their own deformation. Furthermore, in frame #1258, SuperDiMP gradually tracks background information, while ToMP50 only tracks a portion of the target. However, although the tracker of this invention is closer to the Ground Truth tracking area, updating the model as usual at this point can also lead to contamination of the tracking model. As can be seen from frame #2934, the compared tracker still maintains correct tracking of the target. In complex scenes, the tracking bounding box is not accurate enough, and surrounding objects are easily misidentified as the target. Although the tracker of this invention has excellent robustness to complex situations such as deformation, interference, and background clutter, there is still a gap compared to the marked bounding box.

[0108] in, Figure 6 Visualizations of the proposed trackers (i.e., ProDiMP and ProToMP) and baselines (i.e., SuperDiMP and ToMP50) on several challenging sequences in the OTB100 and LaSOT datasets. The trackers of this invention can achieve more accurate tracking results.

[0109] Although the technology of this invention is based on the DCF tracker for instance verification, it is also applicable to deep learning trackers such as Siamese network trackers and Transformer trackers. The target tracking technology that uses color attention encoding and position offset constraint rules is within the scope of protection of this invention.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0111] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method based on color cues embedding and positional constraints, characterized in that, include: Acquire the target search image and extract the color cue target perception feature map using the feature extractor; The feature extractor includes: extracting depth features of the target search image according to a preset deep learning network, and processing the depth features using a CBR module to obtain processed features; dividing the target search image into foreground and background regions using a color cue encoder, extracting and processing the color histograms of the foreground and background regions to obtain a color target probability map; smoothing the color target probability map using a Hamming window to obtain a color cue feature map, scaling the color cue feature map using bilinear interpolation, and performing feature mapping using three fully convolutional layers and a Sigmoid function to obtain mapped features; and performing element-wise multiplication of the mapped features and processed features, followed by normalization, to obtain a color cue target perception feature map. The color-coded target perception feature map is input into the model predictor to obtain the appearance model; The model predictor The objective function formula is: ； in, For the residual function, As a scalar, For regularization terms, Calculate the response graph and Gaussian tags The error between them This represents the convolution operation. Training samples are used as target features. , For appearance model; A test set is established by acquiring the target image to be searched. The test set image is processed by the feature extractor, and the target response map is obtained by the appearance model. The position of the target in the next frame is determined according to the target peak position of the target response map. Then, the scale is estimated, and the model predictor is updated by combining the HNM algorithm with the target constraint mechanism to realize the real-time update of the appearance model. The target constraint mechanism specifically includes: ； in, It is the displacement ratio. The initial target scale, The constraint radius.

2. The target tracking method based on color cues embedding and position constraints according to claim 1, characterized in that, The extraction of the color histograms of the foreground and background regions specifically includes searching for the target state in the target image. Divide the foreground and background areas, that is... and ,in, This indicates that only the color histogram inside the rectangle is counted, and This indicates that only the color histogram outside the rectangle is counted, and the color histograms of the foreground and background areas are extracted separately. The formula is: ； in, For histograms, Indicates an image patch. , RGB image regions representing the foreground or background areas. Count the identical values ​​of each pixel in the image. This represents the summation function.

3. The target tracking method based on color cues embedding and position constraints according to claim 2, characterized in that, The acquisition of the color target probability map specifically includes processing the foreground histogram. and background histogram After processing, a color target probability map is obtained, using the following formula: ； in, This represents the probability of a colored target. This is a hyperparameter.

4. The target tracking method based on color cues embedding and position constraints according to claim 1, characterized in that, The obtained mapped features specifically include performing bilinear interpolation to adjust the shape of the color cue feature map to the shape of the depth features, and then using three fully convolutional layers and the sigmoid function for mapping, as shown in the formula: ； in, For color-coded feature maps, This indicates a resizing operation. It is a fully convolutional layer.

5. The target tracking method based on color cues embedding and position constraints according to claim 4, characterized in that, The acquisition of the color cue target perception feature map specifically includes: ； in, This indicates CBR. To make the feature hints smoother, Conv1×1 processing is used. Indicates an image patch. middle Indicates backbone network, subscript This indicates that the network outputs the 4th mapping convolutional layer, and finally uses InstanceNorm to normalize the features to obtain the color cue target perception feature map. The formula is as follows: ； in, Represents Conv1×1, This indicates the InstanceNorm layer.

6. A target tracking system based on color cues embedding and position constraints, employing the target tracking method based on color cues embedding and position constraints as described in any one of claims 1-5, characterized in that, include: The feature map acquisition module acquires the target search image and extracts the color cue target perception feature map based on the feature extractor. The feature extractor includes: extracting depth features of the target search image according to a preset deep learning network, and processing the depth features using a CBR module to obtain processed features; dividing the target search image into foreground and background regions using a color cue encoder, extracting and processing the color histograms of the foreground and background regions to obtain a color target probability map; smoothing the color target probability map using a Hamming window to obtain a color cue feature map, scaling the color cue feature map using bilinear interpolation and performing feature mapping to obtain mapped features; and element-wise multiplying the mapped features with the processed features and performing normalization to obtain a color cue target perception feature map. The appearance model acquisition module inputs the color cue target perception feature map into the model predictor to acquire the appearance model. The update module acquires the target image to be searched to establish a test set, processes the test set images through the feature extractor, obtains the target response map through the appearance model, determines the target's position in the next frame based on the target peak position of the target response map, then performs scale estimation, and updates the model predictor through the HNM algorithm combined with the target constraint mechanism to achieve real-time updating of the appearance model.

Images