Target tracking method and system based on color prompt embedding and position constraint
Through the method based on color prompt embedding and position constraints, the shortcomings of appearance model update and feature extraction in visual target tracking are solved, and more accurate and stable target tracking is achieved to adapt to the dynamically changing target appearance.
Patent Information
- Application Number
- CN202411059201.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-03
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-03
AI Technical Summary
Existing visual target tracking methods perform poorly in the face of background messy, scale changes and similar distractors, especially the trackers based on DCF and Siamese networks have shortcomings in appearance model updates and feature extraction, making it difficult to adapt to dynamic changes. Target.
The target tracking method based on color prompt embedding and position constraints is adopted. The color histogram is processed through the color prompt encoder and the Hanming window to generate the color target probability map, and combined with bilinear interpolation and InstanceNorm, the color prompt target perception feature map is obtained. Used to update the appearance model and control real-time update of the model through position constraint mechanism.
The effective fusion of shallow color features and deep CNN features is achieved, which improves the accuracy and robustness of target tracking, reduces the impact of inaccurate tracking results on model updates, and ensures the adaptability and stability of the model.
Smart Images

Figure CN118941593A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and more particularly to a target tracking method and system based on color cue embedding and position constraint. Background Art
[0002] Visual object tracking, as one of the important directions in computer vision, aims to predict the state of an object in a video sequence, especially when only the initial state of the object is provided. With the booming development of deep learning in this field, the two popular trackers based on discriminative correlation filters (DCF) and Siamese networks have made significant progress, but the visual tracking task has been plagued by some complex and challenging factors, such as background clutter, scale changes, and similar distractors.
[0003] At present, the DCF and Siamese network frameworks are widely sought after by researchers. Both core ideologies use trainable deep networks to learn the appearance model of arbitrary target objects online or offline. Siamese network trackers (SN) usually include two sub-branches, each of which takes a template and a search sample as input. Specifically, the template is twice as small as the search area. SN can be regarded as a template matching process, that is, matching the most similar template from the search area. SN has achieved commendable tracking accuracy and speed on such a concise network structure, which has greatly promoted the application of deep learning in visual tracking. The entire tracking architecture of SN can be divided into three parts: feature extraction, feature fusion, and prediction head. In terms of feature extraction, SN was initially built on AlexNet, and then adopted a backbone network from CNN (such as VGG, ResNet) to visual Transformer (such as ViT, SwinT) to extract more powerful feature representations. In addition, by adopting different feature fusion techniques and anchor-based or anchor-free prediction heads, SN continues to refresh the most advanced new records on multiple benchmarks. With the successful application of Transformer in computer vision tasks, neural networks using CNN feature extraction and Transformer feature fusion were proposed, and it was proved that a single-branch architecture using ViT to jointly complete feature extraction and fusion is feasible. However, SN uses limited background information to learn the appearance model, and this method usually only uses the initial target as a template and does not update it. Compared with DCF-based trackers, SN is less prone to cumulative errors in the reasoning stage, but is relatively susceptible to drastic appearance changes and similar objects.
[0004] Relatively speaking, DCF can gain support from more background information for discriminative model prediction, especially the recent SuperDiMP and ToMP can also achieve end-to-end training like SN. In addition, DCF requires an online update mechanism to determine whether the prediction result dynamically updates the training sample set. The present invention divides DCF into four parts: feature extraction, target localization, target scale estimation and online update. Although DCF-based trackers have achieved impressive performance on mainstream benchmarks, the present invention recognizes that there are two points in these tracking methods that need to be further improved. 1) Feature extraction has always been the primary task before tracking, and excellent feature representation is positive for training appearance models. ResNet, as a popular CNN backbone network, can be confirmed as the preferred feature extractor instead of hand-crafted methods (such as HOG, color features). How to convert pre-trained feature attributes in image classification into feature attributes that are more suitable for visual tracking has always been a research topic worth exploring. 2) Another thorny issue worth discussing in DCF is how to effectively update the model online. Since the target in the tracking process changes dynamically, it is reasonable to appropriately update the tracking model to adapt to its appearance changes. In this regard, linear interpolation or storage sample optimization schemes are usually adopted. Regardless of the method used, blindly updating the model with the predicted target is unreasonable, which will lead to model drift. Therefore, recent DCFs (such as SuperDiMP and ToMP) adopt hard negative sample mining (HNM) strategies to control model updates. HNM methods evaluate the target tracking status through the target position and response map, but do not consider the motion inertia of the tracked target in the video sequence. Summary of the invention
[0005] In view of this, the present invention provides a target tracking method and system based on color cue embedding and position constraint to solve the problems existing in the background technology.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A target tracking method based on color cue embedding and position constraint, comprising: acquiring a target search image, extracting a color cue target perception feature map according to a feature extractor;
[0008] The feature extractor comprises: extracting the deep features of the target search image according to a preset deep learning network, and processing the deep features with a CBR module to obtain processing features; dividing the target search image into a foreground area and a background area through a color cue encoder, extracting and processing the color histograms of the foreground area and the background area, and obtaining a color target probability map; smoothing the color target probability map with a Hamming window to obtain a color cue feature map, scaling the color cue feature map with a bilinear interpolation method and performing feature mapping to obtain mapping features; element-wise multiplying the mapping features with the processing features, and performing normalization processing to obtain a color cue target perception feature map;
[0009] Inputting the color cue target perception feature map into a model predictor to obtain an appearance model;
[0010] A target image to be searched is obtained to establish a test set, the test set image is processed by the feature extractor, and a target response map is obtained by the appearance model. The position of the target in the next frame is determined according to the target peak position of the target response map, and then the scale size is estimated. The model predictor is updated by the HNM algorithm combined with the target constraint mechanism to realize the real-time update of the appearance model.
[0011] Preferably, the extracting the color histogram of the foreground area and the background area specifically includes dividing the foreground area and the background area according to the target state s in the target search image, that is, and in, means that only the color histogram inside the rectangular box is counted, and It means that only the color histogram outside the rectangular box is counted, and the color histograms of the foreground and background areas are extracted respectively. The formula is:
[0012]
[0013] Among them, b is the histogram, x im represents an image patch, x im ∈[0,255], The RGB image region represents the foreground area or the background area, CalHis(·) counts the same value of each pixel in the image, and sum(·) represents the summation function.
[0014] Preferably, the acquisition of the color target probability map specifically includes the foreground histogram and background histogram After processing, the color target probability map is obtained, and the formula is:
[0015]
[0016] in, represents the color target probability, and λ is a hyperparameter.
[0017] Preferably, obtaining the mapping feature specifically includes executing a bilinear interpolation method to adjust the shape of the color cue feature map to the shape of the depth feature, and then using three full convolutional layers and a Sigmoid function for mapping, and the formula is:
[0018]
[0019] in, is the color cue feature map, γ represents the resizing operation, and FC is the fully convolutional layer.
[0020] Preferably, the step of obtaining a color prompt target perception feature map specifically includes:
[0021]
[0022] in, Represents CBR. In order to make the feature hint smoother, Conv1×1 is used for processing. im represents an image patch, R represents the backbone network, the subscript c4 represents the 4th layer of network output mapping convolution layer, and finally the features are normalized using InstanceNorm to obtain the color cue target perception feature map, the formula is:
[0023] x f =N(c 1 (f prompt ));
[0024] Among them, c 1 represents Conv1×1, and N represents the InstanceNorm layer.
[0025] Preferably, the model predictor The objective function formula is:
[0026]
[0027] Among them, r(·) is the residual function, b is a scalar, f(·) is the regularization term, and r(·) calculates the response map and Gaussian labels y g The error between them, * indicates the convolution operation, x f is the target feature, training sample For the appearance model.
[0028] Preferably, the target constraint mechanism specifically includes:
[0029]
[0030] Where d is the displacement ratio, (w,h) is the initial target scale, and π is the constraint radius.
[0031] A target tracking system based on color cue embedding and position constraint, comprising:
[0032] The feature map acquisition module acquires the target search image and extracts the color prompt target perception feature map according to the feature extractor;
[0033] The feature extractor comprises: extracting the deep features of the target search image according to a preset deep learning network, and processing the deep features with a CBR module to obtain processing features; dividing the target search image into a foreground area and a background area through a color cue encoder, extracting and processing the color histograms of the foreground area and the background area, and obtaining a color target probability map; smoothing the color target probability map with a Hamming window to obtain a color cue feature map, scaling the color cue feature map with a bilinear interpolation method and performing feature mapping to obtain mapping features; element-wise multiplying the mapping features with the processing features, and performing normalization processing to obtain a color cue target perception feature map;
[0034] An appearance model acquisition module inputs the color cue target perception feature map into a model predictor to acquire an appearance model;
[0035] The updating module obtains the target image to be searched to establish a test set, processes the test set image through the feature extractor, obtains the target response map through the appearance model, determines the position of the target in the next frame according to the target peak position of the target response map, and then estimates the scale size, and updates the model predictor through the HNM algorithm combined with the target constraint mechanism to achieve real-time update of the appearance model.
[0036] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a target tracking method and system based on color cue embedding and position constraint, which adopts histogram statistics of RGB images to construct color target probability features, and realizes the effective fusion of shallow color features and deep CNN features; inspired by cue learning, the present invention regards color target probability as a cue, and encodes the cue as a residual feature to guide the acquisition of target perception features; in order to reduce the impact of inaccurate tracking results on the online model updating process, a position offset constraint method is proposed to force the tracked target to satisfy the traditional motion inertia. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0038] Figure 1 A schematic diagram of the color prompt encoder structure provided by the present invention;
[0039] Figure 2 An expanded diagram of the color prompt encoder provided by the present invention;
[0040] Figure 3 A schematic diagram of the visualized color target probability provided by the present invention;
[0041] Figure 4 A visual schematic diagram of the position constraint mechanism provided by the present invention;
[0042] Figure 5 An improved diagram based on the SuperDiMP tracker provided by the present invention;
[0043] Figure 6 This is a visualization result diagram of the challenge sequence provided by the present invention. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] The embodiment of the present invention discloses a target tracking method based on color cue embedding and position constraint, comprising: acquiring a target search image, extracting a color cue target perception feature map according to a feature extractor;
[0046] The feature extractor includes: extracting the deep features of the target search image according to a preset deep learning network, and processing the deep features with a CBR module to obtain processing features; dividing the target search image into a foreground area and a background area through a color cue encoder, extracting and processing the color histograms of the foreground area and the background area, and obtaining a color target probability map; smoothing the color target probability map with a Hamming window to obtain a color cue feature map, scaling the color cue feature map with a bilinear interpolation method and performing feature mapping to obtain mapping features; element-wise multiplying the mapping features with the processing features, and performing normalization processing to obtain a color cue target perception feature map;
[0047] Input the color cue target perception feature map into the model predictor to obtain the appearance model;
[0048] The target images to be searched are obtained to establish a test set. The test set images are processed by the feature extractor, and the target response map is obtained by the appearance model. The position of the target in the next frame is determined according to the target peak position of the target response map, and then the scale size is estimated. The model predictor is updated through the HNM algorithm combined with the target constraint mechanism to achieve real-time update of the appearance model.
[0049] In a specific embodiment, after the position of the target in the next frame is determined to obtain the target position and the response map, the reliability of the prediction can be determined based on the HNM for the positioning result.
[0050] In a specific embodiment, in order to correctly integrate color cues into the deep network, color cues are encoded into ResNet through a color cues encoder (CPE), such as Figure 1-2 As shown in Figure 2, the generation of the target perception feature map is guided. The CTP is obtained through the foreground and background probability maps. First, the present invention divides the foreground and background regions according to the target state s ( Figure 2 The red rectangle in the middle), and here, means that only the color histogram inside the rectangular box is counted, and It means that only the color histogram outside the rectangular box is counted, and the color histogram of the foreground area and the background area is extracted. Specifically, the foreground area and the background area are divided according to the target state s in the target search image, that is, and in, means that only the color histogram inside the rectangular box is counted, and It means that only the color histogram outside the rectangular box is counted, and the color histograms of the foreground and background areas are extracted respectively. The formula is:
[0051]
[0052] Among them, b is the histogram, the default size is 16, x im represents an image patch, x im ∈[0,255], The RGB image region represents the foreground area or the background area, CalHis(·) counts the same value of each pixel in the image, and sum(·) represents the summation function.
[0053] In a specific embodiment, obtaining the color target probability map specifically includes: and background histogram After processing, the color target probability map (CTP) is obtained, and the formula is:
[0054]
[0055] in, represents the color target probability, and λ is a hyperparameter (the default value is 0.01).
[0056] In order to eliminate the interference of background probability information, the present invention uses a Hamming window for smoothing To obtain the color hint feature map
[0057] In a specific embodiment, obtaining the mapping feature specifically includes: obtaining the color prompt feature map Afterwards, Figure 3 As shown (consisting of foreground and background probability maps. (a) Image patch, (b) Average foreground probability, (c) Average background probability, (d) Average color target probability. Among them, the present invention uses the color histogram method to convert the RCB image (a) into (b)-(d) images. (e) Color target probability, and also visualizes the R, G and B channels of CTP; the upper right corner represents the color value of the visualized feature map in R, G, B of the color target probability map bd and e. The larger the value, the better the feature attention effect). Since the shape is the same as that of the original RGB image, the width and height of the depth feature are downsampled by a step size of 16, and the bilinear interpolation method is performed to adjust the shape of the color cue feature map to the shape of the depth feature, and then three full convolution layers and Sigmoid functions are used for mapping. The formula is:
[0058]
[0059] in, is the color cue feature map, γ represents the resizing operation, FC is the full convolution layer, the channels of the FC layer are [3, 16, 16, 1], 3 is the input channel, 16 is the hidden channel, and 1 is the output channel. Figure 1 As shown, the FC layer consists of two CBRs (i.e., Conv3×3+BN+ReLU) and one 1×1 convolution (i.e., Conv1×1).
[0060] In a specific embodiment, obtaining a color cue target perception feature map specifically includes:
[0061]
[0062] in, Represents CBR. In order to make the feature hint smoother, Conv1×1 is used for processing. im represents an image patch, R represents the backbone network, the subscript c4 represents the 4th layer of network output mapping convolution layer, and finally the features are normalized using InstanceNorm to obtain the color cue target perception feature map, the formula is:
[0063] x f =N(c 1 (f prompt ));
[0064] Among them, c 1 represents Conv1×1, and N represents the InstanceNorm layer.
[0065] In one embodiment, the model predictor The objective function formula is:
[0066]
[0067] Among them, r(·) is the residual function, b is a scalar, f(·) is the regularization term, and r(·) calculates the response map and Gaussian labels y g The error between them, * indicates the convolution operation, x f is the target feature, training sample For the appearance model.
[0068] In a specific embodiment, most DCF-based trackers generate robust response maps to find feasible target locations. However, the appearance models of these trackers will be updated online based on the initial annotation frames and prediction results, so the screening of prediction results is crucial. Although SuperDiMP achieves impressive performance using the HNM method, the results obtained from the response map are not necessarily reliable in complex scenes. In addition, the HNM method outputs the final position offset and target state from the response map, but has the defect of insufficient utilization of the position offset, which Figure 4 As can be seen, PCM is mainly based on the target position in the previous frame and the preset constraint radius π (i.e., the red circle) to determine the target motion range. If the target position in the next frame Outside the circle, the target movement is considered abnormal, and the formula is:
[0069]
[0070] Here, p In order to better measure the effectiveness of the position offset, the constraint radius is set according to the size of the initial target scale (w, h). The target constraint mechanism (PCM) specifically includes:
[0071]
[0072] Where d is the displacement ratio, the default value is 0.8, (w,h) is the initial target scale, and π is the constraint radius. In this PCM, o p >p indicates that it is not recommended to use the prediction results to update the sample set S train .
[0073] Tracking anomalies are identified based on the position offset between the previous target position and the current predicted position, such as Figure 4 As shown in the figure, an intuitive explanation is provided for the search area (blue rectangular box), target state (green rectangular box), and constraint radius (red circle) in image frame #0010 of the Basketball sequence of OTB100. The red circle is drawn based on the constraint radius determined based on the target position of the previous frame as the center. The concept of PCM proposed in the present invention is that the target in the next frame is positioned outside the circle to indicate a tracking anomaly and is not considered as a model update sample. The intention of PCM provides a complementary clue to the HNM method, which regulates the update of sample memory and model learning rate (default is 0.01). The sample memory is used to store initial samples and predicted samples, and its capacity is set to a maximum of 50. In addition, the memory is added or replaced in the case of normal and hard negative, and the hard negative state will double the model learning rate (i.e., 0.02). If the corresponding response graph If the peak is lower than 0.25, the not_found state determines the target as lost. For uncertain states, the target state will also be output normally, but no update operation will be performed. Due to the complexity of tracking conditions, HNM cannot cover all aspects. Therefore, by following the position offset constraint method of motion inertia, the credibility of the updated samples can be ensured as much as possible. The purpose of HNM is to control the storage and replacement of the training frame sample set and the update of the appearance model learning rate. The sample storage includes the annotated initial frame and its data enhancement, as well as the later predicted frame with pseudo labels. In addition, in the normal and hard negative sample states, the stored samples may be stored or replaced, and the hard negative state will double the learning rate. If the peak is lower than 0.25, the not_found state considers the target lost. The uncertain state will output the predicted target state normally, but no update control will be performed.
[0074] The above method is for the case of a single frame. In a specific embodiment, for the condition of multi-frame position offset, the previous position storage space is initialized and updated using the first-in-first-out method. Then, the present invention calculates the current frame Multiple positions stored in the next frame The position offsets between are calculated to obtain a position offset set, and finally the average value is compared with the constraint radius π.
[0075] In a specific embodiment, the improved tracker adopts the state-of-the-art SuperDiMP as the baseline tracker, and SuperDiMP includes the target localization module of DiMP and the probabilistic rectangular box regression of PrDiMP. In particular, SuperDiMP can be decomposed into four stages: feature extraction, target localization and target scale estimation, and online model update. First, the image frame is preprocessed and input into the backbone network to extract the deep feature map. The backbone network adopts the pre-trained ResNet50 to obtain the conv4_x feature map as the image feature. In addition, the network weights before the conv4_x layer will be frozen. In order to make the features obtained by the image classification network more suitable for visual tracking, SuperDiMP adds a convolution block to obtain the learned feature expression. In this part, the present invention proposes a color prompt encoder (CPE) which aims to encode color information as prompts to generate target-aware feature maps. CPE can provide the functions of timely guidance and target attention. The feature extraction stage composed of the backbone network and the CPE module is formulated as follows
[0076]
[0077] Here, x im Represents an image patch (e.g. Figure 5 R represents the backbone network (e.g. ResNet50), and the subscript indicates which layer of features the network output is mapped to (e.g. c4 represents conv4_x). Indicates a CPE module.
[0078] in, Figure 5 SUMMARY The Color Prompt Encoder (CPE) and Position Constraint Mechanism (PCM) of the present invention are based on an improved illustration of the SuperDiMP tracker.
[0079] The feature extraction component of the CPE proposed in the present invention can obtain more diverse and specific target features x f , these features are used to obtain the appearance model (i.e., convolution kernel) through the target localization module. From the model predictor The appearance model α can output the expected object score (e.g. Gaussian labels). Training samples is input to the model predictor Get the appearance model n represents the total number of training samples. Model predictor The objective function formula is
[0080]
[0081] Here, the objective function consists of the residual function r(·) and a regularization term f(·) weighted by a scalar b. r(·) computes the response map and Gaussian labels y g The error between , * indicates the convolution operation.
[0082] In fact, the convolution kernel is similar to the correlation filter of the original DCF, which performs Perform convolution to obtain the response map Then The coordinates of the peak (x l ,y l ) is determined as the target position of the next frame. After the target is located, ATOM IoUnet is based on the coordinates (x l ,y l ) performs target scale estimation. Unlike target localization, target scale estimation uses conv3_x and conv4_x features to estimate the final target state, i.e. Represents the target scale estimation module. Since the DCF-based tracker allows the storage of target samples to update the appearance model online, it is crucial to further identify the target and screen better samples to store and control the online model update. Therefore, SuperDiMP adopts the HNM method to deal with this matter. Although the HNM method has achieved obvious improvement, it is often not enough in complex scenes. As a supplement, the present invention designs a position constraint mechanism (PCM) to constrain the position offset of the tracked target relative to the traditional motion inertia.
[0083] After the appearance model is trained, the target search map is first obtained based on the target state of the previous frame, and the color prompt target perception feature map is obtained from the target search map feature extractor in the predicted image frame. The feature map is convolved with the appearance model to obtain the response map, that is, The peak value of the response graph is the position of the target in the next frame. The target scale estimation method is then used to perform scale estimation. After obtaining a new target state, HNM and the PCM of the present invention are used to determine the accuracy of the predicted target and control the online update of the appearance model.
[0084] A target tracking system based on color cue embedding and position constraint, comprising:
[0085] The feature map acquisition module acquires the target search image and extracts the color prompt target perception feature map according to the feature extractor;
[0086] The feature extractor includes: extracting the deep features of the target search image according to a preset deep learning network, and processing the deep features with a CBR module to obtain processing features; dividing the target search image into a foreground area and a background area through a color cue encoder, extracting and processing the color histograms of the foreground area and the background area, and obtaining a color target probability map; smoothing the color target probability map with a Hamming window to obtain a color cue feature map, scaling the color cue feature map with a bilinear interpolation method and performing feature mapping to obtain mapping features; element-wise multiplying the mapping features with the processing features, and performing normalization processing to obtain a color cue target perception feature map;
[0087] The appearance model acquisition module inputs the color cue target perception feature map into the model predictor to obtain the appearance model;
[0088] The update module obtains the target image to be searched to establish a test set, processes the test set image through the feature extractor, and obtains the target response map through the appearance model. The position of the target in the next frame is determined according to the target peak position of the target response map, and then the scale size is estimated. The model predictor is updated through the HNM algorithm combined with the target constraint mechanism to realize real-time update of the appearance model.
[0089] The CPE and PCM modules of the present invention can be easily combined with recent DCF-based frameworks such as SuperDiMP and ToMP. The present invention replaces the original ClsFeat with the CPE module for end-to-end training. The target classification features obtained by the method of the present invention are fed into the model predictor to obtain a stronger appearance model. Except for the CPE module, the training setting of the present invention is consistent with the SuperDiMP and ToMP trackers, but the training device of the present invention is two NVIDIA GeForce RTX 3090 GPUs. In addition, the PCM method adopts the predicted positions from the model predictor and has no training requirements, which can be regarded as a screening method to enhance the stability of model updates.
[0090] In a specific embodiment, in order to verify the practicality of the CPE and PCM modules designed by the present invention, the present invention integrates them into the recent DCF-based trackers (i.e., SuperDiMP and ToMP). The improved tracker of the present invention shows comparative performance on six challenging benchmarks, including OTB100, NFS, UAV123, LaSOT, TrackingNet, and VOT2020. The method of the present invention is implemented in Python using PyTorch, and all experiments are run in the GPU processor of GeForceRTX4070.
[0091] Performance on different datasets:
[0092] LaSOT: Table 1 shows the comparison results of Precision and AUC scores of various trackers. The LaSOT dataset contains mostly longer video sequences, with an average of 2500 frames per video sequence, which makes this dataset more challenging than other datasets. Therefore, the color cue encoder and position constraint mechanism proposed in this paper are to enhance and maintain the discriminative ability of the target classifier. This is crucial for long video sequences, because the longer the video sequence, the more likely the model is to be contaminated by erroneous background information. The recent DiMP, PrDiMP, SuperDiMP, TrDiMP, TransT, STARK-ST50, Sim-B / 32, CSWinTT and ToMP50 are selected for comparison. The CPE and PCM methods proposed in this paper are integrated into SuperDiMP and ToMP50, achieving competitive performance, outperforming SuperDiMP with relative gains of 2.1% and 1.4% (%), respectively. In addition, ProToMP shows competitive performance compared with Transformer-based trackers such as TrDiMP, TransT, STARK-ST50 and ToMP50. From the results, we can see that there is a significant improvement over pure CNN-based trackers such as SuperDiMP, while Transformer-based trackers do not show significant improvement on this dataset. However, ProToMP can still achieve good results. In summary, it can be seen that the proposed method is of some significance in improving the tracking performance.
[0093] Table 1 Tracker comparison results
[0094]
[0095]
[0096] Comparison results of the proposed ProDiMP and ProToMP trackers with the state-of-the-art trackers on LaSOT, TrackingNet, OTB100, NFS and UAV123 datasets in terms of precision (Prec.), normalized precision (Norm.Prec.) and AUC scores (%).
[0097] TrackingNet: The test set of the TrackingNet dataset is also used to test the method of the present invention, which consists of 511 video sequences. The experimental results are shown in Table 1. The ProToMP tracker of the present invention has an AUC score of 81.8%, which is better than the most advanced trackers such as TrDiMP, TransT, STARK-ST50, Sim-B / 32 and ToMP50. In addition, compared with SuperDiMP, it can be seen that the ProDiMP of the present invention promotes SuperDiMP, with a relative AUC gain of 0.4%, and the accuracy and normalized accuracy increased by 0.7 and 0.4 (%), respectively. In addition, compared with ToMP50, the ProToMP of the present invention also achieved competitive performance, which promoted the accuracy, normalized accuracy and AUC of ToMP50 to increase by 1.0, 0.5 and 0.6 (%), respectively. The competitive results on this dataset further demonstrate the effectiveness of the method proposed in the present invention.
[0098] OTB100: OTB100 consists of 98 video sequences and 100 tracking targets. Table 1 provides the AUC scores of ProDiMP and ProToMP of the present invention compared with the state-of-the-art trackers. As can be seen from the table, the AUC scores of ProDiMP and ProToMP of the present invention are 70.4 and 70.1 (%), respectively, which exceed SuperDiMP and ToMP50, with AUC gains of 0.3 and 0.1 (%), respectively. In addition, the performance of ProDiMP is better than that of recent SOTA trackers (e.g., STARK-ST50, TransT). In addition, DiMP, PrDiMP, STARK and ToMP use ResNet-50 as the backbone by default for fair comparison. The experimental results further demonstrate the effectiveness of the proposed method.
[0099] NFS: We evaluate our tracker on the 30 frames / second version of the dataset. The dataset consists of 100 video sequences with fast-moving target objects. As shown in Table 1, our ProToMP achieves SOTA performance with an AUC score of 67.5%, surpassing ToMP50 by 0.6%. In addition, our method also outperforms the recent TransT and STARK.
[0100] UAV123: The drone dataset consists of 123 aerial videos and is designed to test trackers suitable for drones. Among the compared methods, Table 1 shows that the AUC scores of the proposed ProDiMP and ProToMP trackers are 69.2 and 68.9 (%), respectively, which achieves competitive performance compared to other state-of-the-art methods (e, g, STARK). Although the tracking performance of ProToMP is not as good as CSWinTT and ToMP50, the proposed ProDiMP exceeds SuperDiMP with a gain of 1.5%.
[0101] VOT2020: Our ProDiMP and ProToMP trackers were evaluated in the 2020 edition of the Visual Object Tracking (VOT) short challenge. Although the dataset contains 60 videos annotated with segment masks, we only compare them with trackers that predict bounding boxes because our tracker predicts bounding boxes. The trackers are evaluated in a multi-start protocol and ranked based on the EAO metric, which takes into account tracking accuracy (average overlap of successful frames) and robustness (failure rate). The results in Table 2 show that ProDiMP achieves the first accuracy and EAO. Our ProToMP achieves the first robustness. In particular, ProDiMP achieves a robustness of 0.798 and an EAO of 0.311, surpassing SuperDiMP in terms of robustness (+7%) and EAO (+0.6%). ProToMP achieves a robustness of 0.815 and an EAO of 0.311, outperforming ToMP50 in terms of robustness (+2.6%) and EAO (+0.8%).
[0102] Table 2 Effect comparison results
[0103]
[0104]
[0105] State-of-the-art comparison of bounding box-only methods on VOT2020ST in terms of accuracy (A), robustness (R), and expected average overlap (EAO).
[0106] Qualitative analysis:
[0107] The above experiments mainly prove the effectiveness of the proposed method from a quantitative perspective and can effectively improve the tracking results. In addition, the present invention also qualitatively demonstrates Figure 5The present invention visualizes the tracking results of SuperDiMP, ToMP50 and the trackers of the present invention (i.e., ProDiMP and ProToMP) on four challenging sequences of OTB100 and LaSOT. The present invention selects two video sequences (i.e., Basketball and Skating2_1) from the OTB100 short-term dataset and two image sequences (i.e., dog-15 and zebra-17) from the LaSOT long-term dataset for qualitative analysis. Figure 6 As shown in FIG. 1 , the video frame numbers of the four video sequences are 725, 473, 5009 and 2964 respectively. The present invention shows four representative frames, wherein the first frame shows the bounding box of the tracked target. Figure 6 As can be seen in the figure, the ProDiMP and ProToMP trackers can accurately predict the target object despite occlusion, deformation and background clutter. In frame #0470 of the Basketball sequence, the tracker of the present invention mistakes a similar interferer as the target, but SuperDiMP and ToMP50 may also obtain incorrect bounding boxes. In the Skating2_1 sequence, the tracker of the present invention can basically track the female player in the presence of interferers, but the SuperDiMP and ToMP50 methods of the present invention are disturbed by background clutter. In frame #0110 of the dog-15 sequence, SuperDiMP will lose the tracked target and mistake the dog in the mirror as the target. In the zebra-17 sequence, SuperDiMP and ToMP50 are not robust enough and may be affected by similar objects and their own deformations. In addition, in frame #1258, SuperDiMP gradually tracks the background information, while ToMP50 only tracks part of the target. However, although the tracker of the present invention is closer to the tracking area of GroundTruth, updating the model as usual at this time will also cause the tracking model to be contaminated. As can be seen from frame #2934, the compared tracker still keeps tracking the target correctly. In complex scenes, the tracking bounding box is not accurate enough and surrounding objects are easily misidentified as targets. Although the tracker of the present invention has excellent robustness to complex situations such as deformation, interference and background clutter, there is still a gap with the marked bounding box.
[0108] in, Figure 6 Visualization results of the proposed trackers (i.e., ProDiMP and ProToMP) and baselines (i.e., SuperDiMP and ToMP50) on several challenge sequences of OTB100 and LaSOT datasets. The tracker of the present invention can achieve more accurate tracking effect.
[0109] Although the technology of the present invention is verified based on the DCF tracker, it is also applicable to deep learning trackers such as Siamese network trackers and Transformer trackers. The target tracking technology using color attention coding and rule design of position offset constraints are all within the protection scope of the present invention.
[0110] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0111] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A target tracking method based on color cue embedding and position constraint, characterized in that: include: Obtain a target search image, and extract a color prompt target perception feature map according to a feature extractor; The feature extractor comprises: extracting the deep features of the target search image according to a preset deep learning network, and processing the deep features with a CBR module to obtain processing features; dividing the target search image into a foreground area and a background area through a color cue encoder, extracting and processing the color histograms of the foreground area and the background area, and obtaining a color target probability map; smoothing the color target probability map with a Hamming window to obtain a color cue feature map, scaling the color cue feature map with a bilinear interpolation method and performing feature mapping to obtain mapping features; element-wise multiplying the mapping features with the processing features, and performing normalization processing to obtain a color cue target perception feature map; Inputting the color cue target perception feature map into a model predictor to obtain an appearance model; A target image to be searched is obtained to establish a test set, the test set image is processed by the feature extractor, and a target response map is obtained by the appearance model. The position of the target in the next frame is determined according to the target peak position of the target response map, and then the scale size is estimated. The model predictor is updated by the HNM algorithm combined with the target constraint mechanism to realize the real-time update of the appearance model.
2. The target tracking method based on color cue embedding and position constraint according to claim 1, characterized in that: The step of extracting the color histogram of the foreground area and the background area specifically includes dividing the foreground area and the background area according to the target state s in the target search image, that is, and in, Indicates that only the color histogram inside the rectangular box is counted, and It means that only the color histogram outside the rectangular box is counted, and the color histograms of the foreground and background areas are extracted respectively. The formula is: Among them, b is the histogram, x im represents an image patch, x im ∈[0,255], ○ represents the RGB image region of the foreground area or the background area, CalHis(·) counts the same value of each pixel in the image, and sum(·) represents the summation function.
3. The target tracking method based on color hint embedding and position constraint according to claim 2, characterized in that: The method of obtaining the color target probability map specifically includes: obtaining the foreground histogram and background histogram After processing, the color target probability map is obtained, and the formula is: in, represents the color target probability, and λ is a hyperparameter.
4. The target tracking method based on color cue embedding and position constraint according to claim 1, characterized in that: The mapping feature is obtained by performing a bilinear interpolation method to adjust the shape of the color cue feature map to the shape of the depth feature, and then using three full convolution layers and a Sigmoid function for mapping, and the formula is: in, is the color cue feature map, γ represents the resizing operation, and FC is the fully convolutional layer.
5. The target tracking method based on color cue embedding and position constraint according to claim 4, characterized in that: The method of obtaining the color prompt target perception feature map specifically includes: in, Represents CBR. In order to make the feature hint smoother, Conv1×1 is used for processing. im represents an image patch, middle represents the backbone network, the subscript c4 represents the 4th layer of network output mapping convolution layer, and finally the features are normalized using InstanceNorm to obtain the color cue target perception feature map, the formula is: x f =N(c 1 (f prompt )); Among them, c 1 represents Conv1×1, and N represents the InstanceNorm layer.
6. The target tracking method based on color cue embedding and position constraint according to claim 1, characterized in that: The objective function formula of the model predictor M is: Where r(·) is the residual function, b is a scalar, f(·) is the regularization term, and r(·) calculates the response map and Gaussian labels y g The error between them, * indicates the convolution operation, x f is the target feature, training sample For the appearance model.
7. The target tracking method based on color cue embedding and position constraint according to claim 1, characterized in that: The target constraint mechanism specifically includes: Where d is the displacement ratio, (w,h) is the initial target scale, and π is the constraint radius.
8. A target tracking system based on color cue embedding and position constraint, applying a target tracking method based on color cue embedding and position constraint as described in any one of claims 1 to 7, characterized in that: include: The feature map acquisition module acquires the target search image and extracts the color prompt target perception feature map according to the feature extractor; The feature extractor comprises: extracting the deep features of the target search image according to a preset deep learning network, and processing the deep features with a CBR module to obtain processing features; dividing the target search image into a foreground area and a background area through a color cue encoder, extracting and processing the color histograms of the foreground area and the background area, and obtaining a color target probability map; smoothing the color target probability map with a Hamming window to obtain a color cue feature map, scaling the color cue feature map with a bilinear interpolation method and performing feature mapping to obtain mapping features; element-wise multiplying the mapping features with the processing features, and performing normalization processing to obtain a color cue target perception feature map; An appearance model acquisition module inputs the color cue target perception feature map into a model predictor to acquire an appearance model; The updating module obtains the target image to be searched to establish a test set, processes the test set image through the feature extractor, obtains the target response map through the appearance model, determines the position of the target in the next frame according to the target peak position of the target response map, and then estimates the scale size, and updates the model predictor through the HNM algorithm combined with the target constraint mechanism to achieve real-time update of the appearance model.
Citation Information
Patent Citations
Target tracking method based on structured pixel-by-pixel target attention mechanism
CN111951297A
Hyperspectral target tracking method based on double-flow visual prompt
CN117994282A
Method and apparatus with object tracking
US20220164961A1
Cited By
Anti-unmanned aerial vehicle observing and aiming tracking system
CN121661359A
Anti-drone observation tracking system
CN121661359B
