Multi-scale Occlusion-resistant Object Tracking Method Based on Handcrafted Feature Fusion

Abstract

The present invention provides a multi-scale anti-occlusion target tracking method based on manual feature fusion, which is highly robust and has low computational complexity, and can run on most tracking platforms. The present invention designs a scale pool for multi-resolution sampling, which can quickly process the scale changes of the target during movement; the proposed method of fusing the response results of two feature filters based on the correlation peak average energy index realizes the integration of the advantages of HOG features and CN features, and improves the recognition ability of the tracker; by performing occlusion judgment evaluation on the tracking results of each frame, an SVM re-detector is designed to re-detect the nearby area before the target is lost after it is lost, thereby improving the anti-occlusion ability of the tracker. The tracking results of the two trackers are evaluated independently, and the learning rate is adjusted according to the APCE index to achieve their respective adaptive updates.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a multi-scale anti-occlusion target tracking method based on manual feature fusion. Background Art

[0002] In the daily production and life applications of computer vision technology, it is often necessary to stably track specific targets in video images and obtain information such as the position and size of the specific targets in the picture in real time.

[0003] Current mainstream object tracking algorithms can be divided into two categories. The first category is based on correlation filtering, which offers high computational speed and excellent tracking performance. Their basic principle is to train a filter template in the initial frame, calculate the similarity between candidate samples and the filter template in subsequent frames, output a response map, and select the location with the maximum response as the target location. The second category is based on deep learning. Although relatively new, these methods have developed rapidly. Their fundamental idea is to leverage the powerful deep features extracted by convolutional neural networks to separate the target from the background. Early deep learning tracking algorithms used pre-trained convolutional neural networks to extract deep features, which were then combined with correlation filtering to train the tracker. Because deep features contain a wealth of abstract semantic information, the tracker's performance in distinguishing between target and background was outstanding. However, the use of high-dimensional deep features incurs a significant computational burden, making it difficult for the algorithm to meet real-time requirements. The Siamese network tracking algorithm, a recently emerging deep learning tracking method, primarily performs similarity matching between the target and candidate boxes in subsequent frames, selecting the candidate box with the highest similarity as the target location. It demonstrates strong robustness and real-time performance. However, deep learning-based tracking methods often require GPU support, which places high demands on hardware performance and is difficult to run on platforms with limited power consumption and cost, thus limiting their scope of application.

[0004] In common tracking tasks, deformation, scale changes, occlusion, and other situations often occur, which requires the tracking algorithm to have strong robustness. At the same time, in order to obtain the tracking target information in real time, the complexity of the tracking algorithm cannot be too high and must meet real-time requirements. Although the traditional correlation filter tracking algorithm has a significant advantage in speed, its performance in common complex scenarios still needs to be improved. To improve the robustness of the algorithm, many improvements have been proposed in recent years. However, these often come at the expense of the algorithm's real-time performance, causing the correlation filter tracking method to lose its most fundamental advantage and limiting its scope of application. Summary of the Invention

[0005] In view of this, the present invention provides a multi-scale anti-occlusion target tracking method based on manual feature fusion, which can achieve strong robustness and low computational complexity and can run on most tracking platforms.

[0006] To achieve the above-mentioned purpose, the present invention provides a multi-scale anti-occlusion target tracking method based on manual feature fusion. Based on the correlation filter model, two independent filters are trained using directional gradient histogram features and color naming features, the target scale is determined by multi-resolution sampling, the two response results are evaluated and adaptively fused, and the final fusion result is judged for occlusion. If it is determined that occlusion occurs, the re-detector is started for re-detection. If it is determined that no occlusion occurs, the position of the target is determined according to the fused response result, the respective learning rates are adjusted according to the APCE values of the two filter response results, the model is updated, and then it is determined whether it is the last frame. If it is the last frame, the method ends; otherwise, it returns to re-multi-resolution sampling to determine the target scale; when re-detecting, it enters the next frame. If the target is not detected, it enters the next frame again until the target is detected, the target position is updated, and then it is determined whether it is the last frame. If it is the last frame, the method ends; otherwise, it returns to re-multi-resolution sampling to determine the target scale.

[0007] The steps include:

[0008] Step 1: Based on the initial information and the kernel correlation filtering principle, the HOG feature filter and the CN feature filter are initialized.

[0009] Step 2: Multi-resolution sampling, extracting HOG features, and using the HOG feature filter to calculate the response to determine the current scale;

[0010] Step 3: Extract CN features based on the size determined in step 2 and calculate the response of the CN filter;

[0011] Step 4: Adaptively fuse the response results of step 2 and step 3;

[0012] Step 5: Use the APCE indicator to evaluate the fused response results. When the indicator is lower than the set threshold, the target is determined to be occluded, and the re-detector is started for re-detection, and the next frame is entered. If the target is not detected, the next frame is entered again until the target is detected. The target position is updated and it is determined whether it is the last frame. If it is the last frame, the process ends, otherwise it returns to step 2. When the indicator is higher than or equal to the threshold, the target is determined to be not occluded, and step 6 is executed.

[0013] Step 6: Determine the target position based on the fused response results, adjust the learning rates of the two filters based on the APCE values of the response results, and update the model; determine whether it is the last frame. If it is the last frame, end; otherwise, return to step 2.

[0014] Among them, when adjusting the learning rate according to the APCE index of each response, threshold a and threshold b are set, a>b. When the evaluation index is higher than threshold a, the learning rate is fixed at 0.01. When it is between threshold a and threshold b, the learning rate is dynamically adjusted according to the index, varying between 0-0.01. When it is lower than threshold b, no update is performed.

[0015] Among them, the target scale is determined by multi-resolution sampling as follows:

[0016] Using the HOG feature filter trained in step 1, based on the original scale, a scale pool is designed to be [1.05, 1, 0.95], corresponding to 1.05 times the original size, the original size, and 0.95 times the original size, respectively. HOG features are extracted at three resolutions, and the response is calculated. The scale corresponding to the maximum response value is the current scale.

[0017] Among them, in step 4, the response results calculated in steps 2 and 3 are evaluated according to the correlation peak average energy index, and the two response results are adaptively fused based on the evaluation index. When the difference between the two indicators is large, the more reliable response result is selected as the final response result. When the difference is small, linear weighted fusion is performed.

[0018] The re-detector is designed based on the principle of linear soft margin support vector machine.

[0019] Beneficial effects

[0020] This invention uses a scale pool for multi-resolution sampling, enabling rapid processing of scale changes during target motion. The proposed method of fusing the responses of two feature filters based on the average correlation peak energy (APCE) combines the advantages of both HOG and CN features, enhancing the tracker's discrimination capabilities. Furthermore, the tracking results of each frame are evaluated for occlusion, and an SVM re-detector is designed to re-detect the nearby area after the target is lost, improving the tracker's anti-occlusion capabilities. The tracking results of the two trackers are independently evaluated, and the learning rates are adjusted based on the APCE metric to achieve adaptive updates for each tracker. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the process of a multi-scale anti-occlusion target tracking algorithm based on manual feature fusion in an embodiment of the present invention.

[0022] FIG2( a ) is a frame of input image in an embodiment of the present invention.

[0023] FIG2(b)(c)(d) are schematic diagrams of multi-resolution sampling in an embodiment of the present invention.

[0024] FIG3(a) is a frame of a picture under normal tracking conditions in an embodiment of the present invention.

[0025] FIG3( b ) is a HOG feature filter response result in an embodiment of the present invention.

[0026] FIG3( c ) is a CN feature filter response result in an embodiment of the present invention.

[0027] FIG3( d ) is the response result after fusion in an embodiment of the present invention.

[0028] FIG4( a ) is a frame of a picture when occlusion occurs in an embodiment of the present invention.

[0029] FIG4( b ) shows the response result of the HOG feature filter when occlusion occurs in an embodiment of the present invention.

[0030] FIG4( c ) is a CN feature filter response result when occlusion occurs in an embodiment of the present invention.

[0031] FIG4( d ) is a fused response result when occlusion occurs in an embodiment of the present invention.

[0032] Figure 4(e) is a schematic diagram of re-detection by SVM re-detector. DETAILED DESCRIPTION

[0033] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0034] The present invention is based on the correlation filtering method and adopts an anti-occlusion target tracking algorithm based on manual feature fusion, which combines the advantages of morphological features and color features. It adopts a concise and efficient scale processing method to achieve accurate prediction of scale changes. The tracking results of each frame are evaluated, and after determining that the target is lost, re-detection is initiated to re-search the target. The two filters are independently updated according to their respective response results, and the learning rate is adaptively adjusted to improve the anti-interference ability of the tracking algorithm. It is suitable for solving target tracking problems in computer vision applications and can be widely used in intelligent monitoring systems, human-computer interaction, autonomous driving and other fields.

[0035] The multi-scale anti-occlusion target tracking method based on manual feature fusion of the present invention initializes the directional gradient histogram (HOG) feature filter and the color naming (CN) feature filter based on the principle of the kernel correlation filtering method according to the manually selected tracking target information in the initial frame. In each subsequent frame, the method first uses the directional gradient histogram feature filter to perform multi-resolution sampling to determine the optimal scale, then obtains the color naming feature filter response, uses the correlation peak average energy (APCE) indicator to evaluate the responses of the two filters, and performs adaptive fusion. The final fusion result is evaluated again, and if it is determined that occlusion occurs, the support vector machine is started for re-detection. The directional gradient histogram feature filter and the color naming feature filter are updated independently, and the learning rate is adjusted according to the APCE indicator of each response. Two thresholds are designed based on experience, a high threshold a and a low threshold b. When the evaluation index is higher than the threshold a, the learning rate is fixed at 0.01. When it is between the threshold a and the threshold b, the learning rate is dynamically adjusted according to the index, varying between 0-0.01. When it is lower than the threshold b, the model is not updated. The implementation flow chart of this embodiment is as follows Figure 1 As shown, the specific steps include:

[0036] Step 1, filter initialization:

[0037] In the first frame of the video sequence, the Histogram of Oriented Gradients (HOG) feature filter and the Color Naming feature filter are initialized. In this initial frame, the target is manually selected to determine its initial position and size. Based on this initial frame target information, the two filters are trained independently using the KCF tracker principle, extracting HOG features and CN features.

[0038] The filter optimization problem is: Where x represents the sample, y represents the label, w is the weight vector, and λ is the regularization parameter used to control the system complexity and prevent overfitting. The above formula is written in matrix form to obtain where X = [x1, x2, ... x n ] T , y is a column vector composed of sample labels, and the expression of the above solution in the complex domain is w=(X H X+λI)X H y. Use the Gaussian kernel function to project the sample into high-dimensional space. The expression of the Gaussian kernel function is The solution to the filter optimization problem is α=(K+λI) -1 y. Using the diagonal property of the circulant matrix, the filter parameters are expressed in the Fourier domain as is the Fourier transform of the sample label, where k xx The kernel matrix K = C(k xx ) first line.

[0039] Step 2: Multi-resolution sampling, extracting HOG features, and using the HOG feature filter to calculate the response to determine the current scale:

[0040] This paper designs a scale pool to address the scaling issue. The scale pool is set to [0.95, 1, 1.05], corresponding to slightly smaller, unchanged, and slightly larger scales, respectively. To reduce computational complexity, during multi-resolution sampling, only the directional gradient histogram features are extracted. The directional gradient histogram filter is used to calculate the response, and the scale corresponding to the maximum response value is used as the current scale.

[0041] FIG2(a) is an input image of a certain frame in the tracking process, and FIG2(b) is a schematic diagram of multi-resolution sampling. The present invention samples at the position determined in the previous frame, with the search size determined in the previous frame as the base size, at three resolutions, namely 0.95 times the base size, 1 times the base size, and 1.05 times the base size. After extracting the HOG features, the response result of the HOG feature filter is calculated, and the size corresponding to the maximum response value is used as the current size of the target. It should be noted here that the search area of the correlation filter includes the target itself and the background area, so in FIG2(b), the size of the search area is larger than the size of the tracked target.

[0042] Step 3: Extract CN features based on the size determined in step 2 and calculate the response of the CN filter.

[0043] The Histogram of Oriented Gradients (HOG) feature primarily characterizes the target's outline, while the Color Name (CN) feature maps the three-channel color features of an RGB image to 11 color channels: black, blue, brown, gray, green, orange, pink, purple, red, white, and yellow, effectively representing the target's color. Based on the principle of kernel correlation filters, the present invention independently trains two filters using the HOG and CN features. During the tracking process, the responses of the two filters are obtained.

[0044] Step 4: Adaptively fuse the response results of steps 2 and 3.

[0045] The present invention uses the average peak correlation energy (APCE) indicator to evaluate the response results of the directional gradient histogram feature filter and the color naming feature filter. The definition of the APCE indicator is as follows:

[0046]

[0047] where y max Refers to the maximum response value, y min Refers to the minimum response value, y w,hRefers to the response value at (w,h). The APCE index reflects the reliability of the tracking result. The more reliable the tracking result, the closer the response graph is to the ideal two-dimensional Gaussian distribution, and the larger the APCE index. When the APCE index of the two filter response results differs significantly, the most reliable response result is selected as the final response result. When the difference between the two is not large, linear weighted fusion is performed. The fusion formula is as follows:

[0048]

[0049] Among them, APCE_HOG refers to the APCE value of the HOG feature filter response result, and APCE_CN refers to the APCE value of the CN feature filter response result. Figure 3(a) is a frame picture under normal tracking conditions in an embodiment of the present invention. Figure 3(b) is the HOG feature filter response result of the current frame in an embodiment of the present invention, and the corresponding APCE index is 45.30. Figure 3(c) is the CN feature filter response result of the current frame in an embodiment of the present invention, and the corresponding APCE index is 54.12. According to the fusion formula, the response results of the two feature filters are fused. Figure 3(d) is the fused response result in an embodiment of the present invention, and the corresponding APCE index is 64.76.

[0050] Step 5: Perform occlusion judgment on the final response result to determine whether to perform re-detection.

[0051] During the movement of the target, the appearance of the target will change. The re-detector needs to be able to adapt to the changes in the target appearance. The strategy adopted is: use the APCE indicator to evaluate the fused response results, set the threshold to 20 based on experience, and when the indicator is lower than the threshold, determine that the target is occluded, start the support vector machine for re-detection, and enter the next frame. If the target is not detected, enter the next frame again until the target is detected, update the target position, and determine whether it is the last frame. If it is the last frame, end, otherwise return to step 2; when the indicator is higher than or equal to the threshold, determine that the target is not occluded, and execute step 6.

[0052] Figure 4(a) is a frame of the picture when occlusion occurs in an embodiment of the present invention. At this time, the tracking target is completely occluded. Figure 4(b) is the response result of the HOG feature filter when occlusion occurs in an embodiment of the present invention. At this time, the maximum response value is 0.21, and the APCE value is 12.54. Figure 4(c) is the response result of the CN feature filter when occlusion occurs in an embodiment of the present invention. At this time, the maximum value is 0.26, and the APCE value is 9.92. Figure 4(d) is the fused response result when occlusion occurs in an embodiment of the present invention. The corresponding maximum value is 0.21, and the APCE value is 17.14.

[0053] Figure 4(e) shows the re-detection process of the SVM re-detector. Based on experience, the APCE threshold is set to 20. When the APCE value of the final response falls below the empirical threshold, the target is considered occluded. At this point, the SVM detector is activated and searches the area immediately before the target is lost until the target is detected.

[0054] Step 6: Determine the target position based on the fused response results, adjust the learning rates of the two filters based on the APCE values of the response results, and update the model.

[0055] The update of the directional gradient histogram feature filter and the color naming feature filter is performed independently. The learning rate is adjusted according to the average energy index of the correlation peak of each response. Two thresholds are designed based on experience: the high threshold a is 33 and the low threshold b is 25. When the evaluation index is higher than threshold a, the learning rate is fixed at 0.01. When it is between threshold a and threshold b, the learning rate is dynamically adjusted according to the index, varying between 0 and 0.01. When it is lower than threshold b, no model update is performed. The learning rate η adjustment rule is:

[0056]

[0057] The present invention also trains a soft margin support vector machine for target re-detection, using HOG features to characterize the target. The optimization problem of the support vector machine is:

[0058]

[0059] sty i (w T x i +b)≥1-ξ i , i=1,2,…,m

[0060] ξ i ≥0,i=1,2,…,m

[0061] Where ξ is the slack variable, w is the hyperplane normal vector, b is the displacement term, C is the penalty parameter, and the Lagrangian function of the above formula is obtained by the Lagrangian multiplier method. i ≥0, β i ≥0 is the Lagrange multiplier.

[0062]

[0063] Let the partial derivative of the above formula with respect to w, b, and ξ be 0, and we get the following formula.

[0064]

[0065]

[0066]

[0067] Get the dual problem.

[0068]

[0069]

[0070] 0≤α i ≤C,i=1,2,...,m

[0071] Solving the above equation to obtain α, we get the soft margin SVM model as shown below.

[0072]

[0073] The appearance of the target may change during the movement, and the SVM re-detector needs to adapt to this change. When the evaluation index is high, the SVM re-detector is updated. The update formula is as follows.

[0074] w t+1 =w t +τ i y i x i

[0075] τ t =l t (||x|| 2 +1 / 2C)

[0076] Among them, w t+1 represents the updated weight vector, (x i ,y i ) represents samples and labels, l t represents the loss function and C is the aggressiveness parameter.

[0077] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-scale anti-occlusion target tracking method based on manual feature fusion, characterized in that: Based on the correlation filter model, two independent filters are trained using the directional gradient histogram feature and the color naming feature. The target scale is determined by multi-resolution sampling, the two response results are evaluated and adaptively fused, and the final fusion result is judged for occlusion. If occlusion is determined to occur, the re-detector is started for re-detection. If occlusion is determined to be not occurred, the position of the target is determined according to the fused response result. The respective learning rates are adjusted according to the APCE values of the two filter response results, and the model is updated. Then, it is determined whether it is the last frame. If it is the last frame, it ends. Otherwise, it returns to re-multi-resolution sampling to determine the target scale. When re-detecting, it enters the next frame. If the target is not detected, it enters the next frame again until the target is detected, and the target position is updated. Then it is determined whether it is the last frame. If it is the last frame, it ends. Otherwise, it returns to re-multi-resolution sampling to determine the target scale. The method for evaluating and adaptively fusing the two response results is: Among them, result is the final fusion result, result_HOG is the response result of the HOG feature filter, result_CN is the response result of the CN feature filter, APCE_HOG is the APCE value of the response result of the HOG feature filter, and APCE_CN is the APCE value of the response result of the CN feature filter; The method for adjusting the learning rate based on the APCE value of any filter response result is: in, η is the learning rate of the HOG feature filter or the CN feature filter, and APCE is the APCE value of the response result of the HOG feature filter or the response result of the CN feature filter.

2. The method according to claim 1, wherein The following steps are involved: Step 1: Based on the initial information and the kernel correlation filtering principle, the HOG feature filter and the CN feature filter are initialized. Step 2: Multi-resolution sampling, extracting HOG features, and using the HOG feature filter to calculate the response to determine the current scale; Step 3: Extract CN features based on the size determined in step 2 and calculate the response of the CN filter; Step 4: Adaptively fuse the response results of step 2 and step 3; Step 5: Use the APCE indicator to evaluate the fused response results. When the indicator is lower than the set threshold, the target is determined to be occluded, and the re-detector is started for re-detection, and the next frame is entered. If the target is not detected, the next frame is entered again until the target is detected. The target position is updated and it is determined whether it is the last frame. If it is the last frame, the process ends, otherwise it returns to step 2. When the indicator is higher than or equal to the threshold, the target is determined to be not occluded, and step 6 is executed. Step 6: Determine the target position based on the fused response results, adjust the learning rates of the two filters based on the APCE values of the response results, and update the model; determine whether it is the last frame. If it is the last frame, end; otherwise, return to step 2.

3. According to the method of claim 1 or 2, when adjusting the learning rate according to the APCE indicator of each response, threshold a and threshold b are set, a>b. When the evaluation indicator is higher than threshold a, the learning rate is fixed at 0.

01. When it is between threshold a and threshold b, the learning rate is dynamically adjusted according to the indicator, varying between 0-0.

01. When it is lower than threshold b, no update is performed.

4. The method according to claim 1, wherein The target scale is determined by multi-resolution sampling as follows: Using the HOG feature filter trained in step 1, based on the original scale, a scale pool is designed to be [1.05, 1, 0.95], corresponding to 1.05 times the original size, the original size, and 0.95 times the original size, respectively. HOG features are extracted at three resolutions, and the response is calculated. The scale corresponding to the maximum response value is the current scale.

5. The method according to claim 2, characterized in that In step 4, the response results calculated in steps 2 and 3 are evaluated based on the correlation peak average energy index, and the two response results are adaptively fused based on the evaluation index. When the difference between the two indicators is large, the more reliable response result is selected as the final response result. When the difference is small, linear weighted fusion is performed.

6. The method according to claim 1, characterized in that The re-detector is designed based on the principle of linear soft-margin support vector machine.

Images