Occlusion tracking method for mobile robot based on multi-core correlation filter fusion

Abstract

The present invention discloses a mobile robot target occlusion tracking method based on multi-core correlation filter fusion. First, various types of complementary characteristics of the target area image block in the video frame are extracted, and the corresponding filters are independently learned according to the characteristics of multiple types. Then, the response map obtained from the filter is used to achieve fusion to determine the current frame position of the moving target. When occlusion occurs during the tracking process, a depth map is introduced to solve the occlusion problem during the tracking process. The real-time target tracking result is obtained through the tracking algorithm, and the position signal is provided for the subsequent mobile robot during tracking, so as to finally complete the target and follow-up tracking tasks of the mobile robot. While ensuring the real-time performance of the system, the algorithm greatly improves the robustness of the tracking algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a mobile robot target occlusion tracking method based on multi-core correlation filtering fusion. Background Art

[0002] With the development of society and the advancement of science and technology, more and more mobile robots are being applied to our life scenes. Significant advances in robotics technology have made it possible for robots to operate in complex environments while sharing workspaces with humans to perform tasks such as healthcare, home automation, and construction. As a specific aspect of robotics technology, automatic tracking is an essential basic function for these robots to interact effectively with humans. Automatic tracking technology allows mobile robots to continuously maintain a certain distance from the target person, allowing robots to work safely and effectively with humans. However, in existing mobile robot tracking, when the target moves quickly or the target size continues to change or is blocked, the target is prone to drift, resulting in tracking failure and low tracking reliability. Summary of the invention

[0003] In view of the above technical problems, the present invention provides a mobile robot target occlusion tracking method based on multi-core correlation filter fusion.

[0004] The technical solution adopted by the present invention to solve the technical problem is:

[0005] A mobile robot target occlusion tracking method based on multi-core correlation filter fusion, the method comprises the following steps:

[0006] Step S100: obtaining a current frame in a video sequence shot by a mobile robot, selecting an area where a tracking target is located from the current frame as a target area image block, preprocessing the target area image block and extracting multiple types of features, inputting the multiple types of features into corresponding feature-related filters for training, and obtaining a response map of each type of feature;

[0007] Step S200: Obtain the pixel depth value of the depth histogram in the target bounding box of the current frame and the pixel depth value of the depth histogram in the target bounding box of the previous frame, and determine whether the target is blocked according to the pixel depth value of the depth histogram in the target bounding box of the current frame, the pixel depth value of the depth histogram in the target bounding box of the previous frame and a preset depth change threshold. If the target is not blocked, fuse the response map of each type of feature to obtain a final response map, update the preset multi-core correlation filter fusion model according to the final response map and obtain the tracking result of the current frame;

[0008] Step S300: obtaining the next frame in the video sequence shot by the mobile robot, repeating step S100, and obtaining the pixel depth value of the depth histogram in the target bounding box of the next frame and the pixel depth value of the tracking result of the current frame, judging whether the target is blocked according to the pixel depth value of the depth histogram in the target bounding box of the next frame, the pixel depth value of the tracking result of the current frame and a preset depth change threshold, and if the target is blocked, the discriminator tracks and obtains the center point of the target position;

[0009] Step S400: Obtain the depth map corresponding to the next frame, obtain the target depth binary image according to the depth map and the center point of the target position, set the context weight, obtain the spatial weighted map according to the target depth binary image and the context weight, and obtain the tracking result of the next frame according to the spatial weighted map.

[0010] Preferably, in step S100, the target area image block is preprocessed and multiple types of features are extracted, and the multiple types of features are input into corresponding feature-related filters for training, and the response graph of each type of feature is obtained, including:

[0011] Step S110: binarizing the target area image block to obtain a grayscale image;

[0012] Step S120: extracting features of the target HOG, HOGC, likelihood map and likelihood map direction histogram from the grayscale image to obtain a first feature, a second feature, a third feature and a fourth feature;

[0013] Step S130: input the first feature, the second feature, the third feature and the fourth feature into corresponding feature correlation filters for training respectively, and obtain a response map of each type of feature.

[0014] Preferably, judging whether the target is blocked in step S200 according to the pixel depth value of the depth histogram in the target bounding box of the current frame, the pixel depth value of the depth histogram in the target bounding box of the previous frame and a preset depth change threshold value includes:

[0015] Step S210: obtaining a depth value variation characteristic according to a pixel depth value of a depth histogram in a target bounding box of a current frame and a pixel depth value of a depth histogram in a target bounding box of a previous frame;

[0016] Step S220: When the depth value variation characteristic is greater than or equal to the preset depth variation threshold, target occlusion occurs in the current frame; when the depth value variation characteristic is less than the preset depth variation threshold, target occlusion does not occur in the current frame.

[0017] Preferably, step S210 is specifically as follows:

[0018]

[0019] Where D represents the depth value variation characteristic, Represents the pixel depth value at the (x, y) position in the tth frame, Represents the pixel depth value at the (x, y) position in the t-1th frame, represents the depth difference, μ t -σ t is the threshold value.

[0020] Preferably, if the target is not blocked in step S200, the final response map obtained by fusing the response map of each type of feature is specifically:

[0021] P=P 1 ⊙P 2 ⊙P 3 ⊙P 4

[0022] Among them, P is the final response graph, P 1 is the response diagram of the first feature, P 2 is the response diagram of the second feature, P 3 is the response diagram of the third feature, P 4 is the response graph of the fourth feature, and ⊙ represents the element-wise product.

[0023] Preferably, in step S400, a spatial weighted graph is obtained according to the target depth binary image and the context weight, specifically:

[0024]

[0025] Among them, σ is the scale parameter, and the context weight is set by adjusting the scale parameter. (x, y) represents the center position of the current tracked target, K is the binary mapping of the depth image obtained by optimizing the k-means method, (x t ',y t ') represents the target center position in the tth frame.

[0026] Preferably, in step S400, the tracking result of the next frame is obtained according to the spatial weighted graph, specifically:

[0027]

[0028] Where F represents the FFT function, F -1 represents the inverse FFT function, ⊙ represents the element-wise product, W t represents a spatial weighted graph, H t represents the spatiotemporal context model, (x′ t+1 ,y′ t+1 ) represents the center point position coordinates of the target in the t+1th frame.

[0029] Preferably, after step S400, the method further includes:

[0030] Step S500: judging the distance between the mobile robot and the obstacle according to the tracking result of each frame, and controlling the mobile robot to track according to a preset distance-speed-movement mode principle according to the distance between the mobile robot and the obstacle.

[0031] Preferably, the preset distance-speed-movement mode principle includes:

[0032] When the distance is within the first range, the mobile robot stops moving forward and rotates on the spot;

[0033] When the distance is within the second range, the mobile robot moves straight while rotating;

[0034] When the distance is within a third range, the mobile robot moves straight without rotating; wherein the third range is larger than the second range, and the second range is larger than the first range.

[0035] The mobile robot target tracking algorithm based on multi-core correlation filter fusion first extracts various types of complementary characteristics of the target area image blocks in the video frame, independently learns the corresponding filters according to the characteristics of multiple types, and then uses the response map obtained from the filter to achieve fusion to determine the current frame position of the moving target. When occlusion occurs during the tracking process, the depth map is introduced to solve the occlusion problem during the tracking process. The real-time target tracking results are obtained through the tracking algorithm, and the position signal is provided for the subsequent mobile robot during tracking, so as to finally complete the target and follow-up tracking tasks of the mobile robot. While ensuring the real-time performance of the system, the algorithm greatly improves the robustness of the tracking algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flow chart of a mobile robot target occlusion tracking method based on multi-core correlation filter fusion provided by an embodiment of the present invention;

[0037] Figure 2 A schematic flow chart of a method for tracking a mobile robot target with occlusion based on multi-core correlation filtering fusion provided in accordance with another embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0039] In one embodiment, Figure 1 As shown, a mobile robot target occlusion tracking method based on multi-core correlation filter fusion includes the following steps:

[0040] Step S100: Obtain the current frame in the video sequence shot by the mobile robot, select the area where the tracking target is located from the current frame as the target area image block, preprocess the target area image block and extract multiple types of features, input the multiple types of features into the corresponding feature-related filters for training, and obtain a response map for each type of feature.

[0041] In one embodiment, in step S100, the target area image block is preprocessed and multiple types of features are extracted, and the multiple types of features are input into corresponding feature-related filters for training, and the response graph of each type of feature is obtained, including:

[0042] Step S110: binarizing the target area image block to obtain a grayscale image;

[0043] Step S120: extracting target HOG, HOGC, likelihood map and likelihood map direction histogram from the grayscale image as the first feature, the second feature, the third feature and the fourth feature;

[0044] Step S130: input the first feature, the second feature, the third feature and the fourth feature into corresponding feature correlation filters for training respectively, and obtain a response map of each type of feature.

[0045] Specifically, four types of features are used to learn the correlation filter: 1) local intensity histogram features (HOGC); 2) directional gradient histograms (HOG) extracted from grayscale images; 3) features of the target likelihood map that represents the likelihood that each pixel in the search area belongs to the target; 4) features of the directional gradient histogram extracted from the target likelihood map. Since the correlation filters trained using the four features are complementary, the four correlation filters are fused to obtain a multi-feature fusion filter.

[0046] Target local intensity features: In actual target tracking scenarios, there will be a large amount of interfering background. When the color of the tracked object is similar to that of the surrounding background pixels, the background features are mapped to the sample by extracting the color histogram as the target feature, resulting in the pollution of the established model and poor tracking effect. In order to make up for this defect, the present invention further improves the tracking accuracy by extracting the intensity features of the slight difference between the target object and the background. Because the grayscale image is robust to motion blur, the local intensity features can be constructed in the RGB image of the obtained regional block. The grayscale image directional gradient histogram is the directional histogram corresponding to the tracked moving target area. The directional gradient histogram (HOG) is used to calculate the gradient information after image acquisition, and has good performance for some units affected by illumination changes and slower motion blur. The HOG feature is obtained from the RGB image, and the horizontal and vertical gradients of the pixels are calculated from the obtained image, and then the gradient information of the pixel point at the i-th frame is calculated; the target likelihood map In the present invention, a Bayesian classifier based on the color histogram is used on the search domain I, and bx represents the color component bin assigned to I(x). Therefore, we can obtain the target possibility of the position according to the Bayesian law.

[0047] Step S200: Obtain the pixel depth value of the depth histogram in the target bounding box of the current frame and the pixel depth value of the depth histogram in the target bounding box of the previous frame, and determine whether the target is blocked according to the pixel depth value of the depth histogram in the target bounding box of the current frame, the pixel depth value of the depth histogram in the target bounding box of the previous frame and a preset depth change threshold. If the target is not blocked, fuse the response map of each type of feature to obtain a final response map, update the preset multi-core correlation filter fusion model according to the final response map and obtain the tracking result of the current frame;

[0048] In one embodiment, judging whether the target is blocked in step S200 according to the pixel depth value of the depth histogram in the target bounding box of the current frame, the pixel depth value of the depth histogram in the target bounding box of the previous frame, and a preset depth change threshold includes:

[0049] Step S210: obtaining a depth value variation characteristic according to a pixel depth value of a depth histogram in a target bounding box of a current frame and a pixel depth value of a depth histogram in a target bounding box of a previous frame;

[0050] Step S220: When the depth value variation characteristic is greater than or equal to the preset depth variation threshold, target occlusion occurs in the current frame; when the depth value variation characteristic is less than the preset depth variation threshold, target occlusion does not occur in the current frame.

[0051] In one embodiment, step S210 is specifically as follows:

[0052]

[0053] Where D represents the depth value variation characteristic, Represents the pixel depth value at the (x, y) position in the tth frame, Represents the pixel depth value at the (x, y) position in the t-1th frame, represents the depth difference, μ t -σ t is the threshold value.

[0054] In one embodiment, if the target is not blocked in step S200, the final response map obtained by fusing the response map of each type of feature is specifically:

[0055] P=P 1 ⊙P 2 ⊙P 3 ⊙P 4

[0056] Among them, P is the final response graph, P 1 is the response diagram of the first feature, P 2 is the response diagram of the second feature, P 3 is the response diagram of the third feature, P 4 is the response graph of the fourth feature, and ⊙ represents the element-wise product.

[0057] Specifically, given the current search region I and the kernelized correlation filter W, we obtain the response map P for each type of feature 1 =W 1 (I) P 2 =W 2 (I) P 3 =W 3 (I) and P 4 =W 4 (I), and then taking the product of these response maps to get the final response map.

[0058] Step S300: Obtain the next frame in the video sequence shot by the mobile robot, repeat step S100, obtain the pixel depth value of the depth histogram in the target bounding box of the next frame and the pixel depth value of the tracking result of the current frame, and determine whether the target is blocked according to the pixel depth value of the depth histogram in the target bounding box of the next frame, the pixel depth value of the tracking result of the current frame and a preset depth change threshold. If the target is blocked, the discriminator tracks and obtains the center point of the target position.

[0059] Specifically, when the target is not occluded, the target is closest to the target bounding box. When the target is occluded, a new object will appear in the target box, which indicates the beginning of the occlusion state. At this time, the depth histogram in the bounding box will have a new rising peak, and its pixel depth value is less than the target value, and the number of boxes in the histogram around the target depth is reduced, and the change in pixel depth value is approximately Gaussian distribution. Therefore, it can be used as a basis for detecting whether occlusion occurs. If the pixel depth value of the current frame t changes greatly compared to the pixel depth value of the previous frame, it is considered that the target is occluded in the current frame. If the change is not large, it is considered that no occlusion occurs in the current frame.

[0060] When the target is in a dynamic scene where the target is occluded, the tracker determines that the target is occluded. At this time, if multi-core correlation filtering is used for tracking, errors will occur in target feature extraction, causing tracking drift. Although multi-core correlation filtering uses color histogram and direction histogram features for fusion and extraction, this method is only applicable to simple scenes. When the mobile robot tracks a scene with a cluttered background and long-term severe occlusion, this method cannot track the target robustly in real time.

[0061] Step S400: Obtain the depth map corresponding to the next frame, obtain the target depth binary image according to the depth map and the center point of the target position, set the context weight, obtain the spatial weighted map according to the target depth binary image and the context weight, and obtain the tracking result of the next frame according to the spatial weighted map.

[0062] Specifically, in target tracking of mobile robots, the target position in the current frame is usually close to the target position in the next frame. The closer the context position is to the center position of the currently tracked target, the more important the context information is for predicting the target position in the next frame. Therefore, the depth map is used and a larger weight is set for the context to suppress the target background, thereby effectively solving the target occlusion tracking problem.

[0063] In one embodiment, in step S400, a spatial weighted graph is obtained according to the target depth binary image and the context weight, specifically:

[0064]

[0065] Among them, σ is the scale parameter, and the context weight is set by adjusting the scale parameter. (x, y) represents the center position of the current tracked target, K is the binary mapping of the depth image obtained by optimizing the k-means method, (x t ',y t ') represents the target center position in the tth frame.

[0066] Specifically, the context weight belongs to (0,1), generating a spatial weighted map of the weighted correlation between the target and its surrounding background in the tth frame.

[0067] In one embodiment, the tracking result of the next frame is obtained according to the spatial weighted graph in step S400, specifically:

[0068]

[0069] Where F represents the FFT function, F -1 represents the inverse FFT function, ⊙ represents the element-wise product, W t represents a spatial weighted graph, H t represents the spatiotemporal context model, (x′ t+1 ,y′ t+1 ) represents the center point position coordinates of the target in frame t+1.

[0070] In one embodiment, after step S400, the method further includes:

[0071] Step S500: judging the distance between the mobile robot and the obstacle according to the tracking result of each frame, and controlling the mobile robot to track according to a preset distance-speed-movement mode principle according to the distance between the mobile robot and the obstacle.

[0072] In one embodiment, the preset distance-speed-movement mode principle includes:

[0073] When the distance is within the first range, the mobile robot stops moving forward and rotates on the spot;

[0074] When the distance is within the second range, the mobile robot moves straight while rotating;

[0075] When the distance is within a third range, the mobile robot moves straight without rotating; wherein the third range is larger than the second range, and the second range is larger than the first range.

[0076] Specifically, the obstacle distance and the robot movement speed output by the fuzzy control algorithm are inferred to control the robot's movement. The fuzzy inference rules are as follows: when the robot is close to the obstacle, the robot moves forward while rotating until it is far away from the obstacle; when the robot is very close to the obstacle and cannot move forward and rotate at the same time, the robot stops immediately and rotates in place until there is no obstacle in front of the robot; when the robot is far away from the obstacle, the robot moves straight without rotating.

[0077] The present invention proposes a mobile robot target occlusion tracking method based on multi-core correlation filter fusion. First, four types of complementary characteristics of the moving target are extracted, and the corresponding filters are independently learned according to the characteristics of the four types. Then, the response map obtained from the filter is used to achieve fusion to determine the current frame position of the moving target. A depth map is introduced to solve the occlusion problem in the tracking process. When using a visual sensor, the tracking algorithm of the present invention is used to obtain real-time target tracking results, and a position signal is provided to the control module of the mobile robot to finally complete the target and follow-up tracking tasks of the mobile robot. While ensuring the real-time performance of the system, the algorithm greatly improves the robustness of the tracking algorithm.

[0078] The above is a detailed introduction to the mobile robot target occlusion tracking method based on multi-core correlation filter fusion provided by the present invention. In the present invention, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A mobile robot target occlusion tracking method based on multi-core correlation filter fusion, characterized in that: The method comprises the following steps: Step S100: obtaining a current frame in a video sequence shot by a mobile robot, selecting an area where a tracking target is located from the current frame as a target area image block, preprocessing the target area image block and extracting a plurality of type features, inputting the plurality of type features into a corresponding feature-related filter for training, and obtaining a response map of each type feature; Step S200: obtaining pixel depth values ​​of a depth histogram in a target bounding box of a current frame and pixel depth values ​​of a depth histogram in a target bounding box of a previous frame, judging whether a target is blocked according to the pixel depth values ​​of the depth histogram in the target bounding box of the current frame, the pixel depth values ​​of the depth histogram in the target bounding box of the previous frame and a preset depth change threshold, if the target is not blocked, fusing the response graphs of each type of feature to obtain a final response graph, updating a preset multi-core correlation filter fusion model according to the final response graph and obtaining a tracking result of the current frame; Step S300: obtaining the next frame in the video sequence shot by the mobile robot, repeating step S100, and obtaining the pixel depth value of the depth histogram in the target bounding box of the next frame and the pixel depth value of the tracking result of the current frame, judging whether the target is blocked according to the pixel depth value of the depth histogram in the target bounding box of the next frame, the pixel depth value of the tracking result of the current frame and a preset depth change threshold, and if the target is blocked, the discriminator tracks and obtains the center point of the target position; Step S400: Obtain the depth map corresponding to the next frame, obtain a target depth binary image based on the depth map and the center point of the target position, set the context weight, obtain a spatial weighted map based on the target depth binary image and the context weight, and obtain the tracking result of the next frame based on the spatial weighted map.

2. The method according to claim 1, characterized in that In step S100, the target area image block is preprocessed and multiple types of features are extracted, and the multiple types of features are input into corresponding feature-related filters for training, and the response graph of each type of feature is obtained, including: Step S110: binarizing the target area image block to obtain a grayscale image; Step S120: extracting features of the target HOG, HOGC, likelihood map and likelihood map direction histogram from the grayscale image to obtain a first feature, a second feature, a third feature and a fourth feature; Step S130: inputting the first feature, the second feature, the third feature and the fourth feature into corresponding feature correlation filters for training respectively, and obtaining a response map of each type of feature.

3. The method according to claim 2, characterized in that In step S200, judging whether the target is blocked according to the pixel depth value of the depth histogram in the target bounding box of the current frame, the pixel depth value of the depth histogram in the target bounding box of the previous frame, and a preset depth change threshold includes: Step S210: obtaining a depth value variation characteristic according to the pixel depth values ​​of the depth histogram in the target bounding box of the current frame and the pixel depth values ​​of the depth histogram in the target bounding box of the previous frame; Step S220: When the depth value variation characteristic is greater than or equal to a preset depth variation threshold, target occlusion occurs in the current frame; when the depth value variation characteristic is less than the preset depth variation threshold, target occlusion does not occur in the current frame.

4. The method according to claim 3, characterized in that Step S210 is specifically as follows: Where D represents the depth value variation characteristic, Represents the pixel depth value at the (x, y) position in the tth frame, Represents the pixel depth value at the (x, y) position in the t-1th frame, represents the depth difference, μ t -σ t is the threshold value.

5. The method according to claim 4, characterized in that In step S200, if the target is not blocked, the final response map obtained by fusing the response maps of each type of feature is as follows: P=P 1 ⊙P 2 ⊙P 3 ⊙P 4 Among them, P is the final response graph, P 1 is the response diagram of the first feature, P 2 is the response diagram of the second feature, P 3 is the response diagram of the third feature, P 4 is the response graph of the fourth feature, and ⊙ represents the element-wise product.

6. The method according to claim 5, characterized in that In step S400, a spatial weighted graph is obtained according to the target depth binary image and the context weight, specifically: Among them, σ is the scale parameter, and the context weight is set by adjusting the scale parameter. (x, y) represents the center position of the current tracked target, K is the binary mapping of the depth image obtained by optimizing the k-means method, (x t ',y t ') represents the target center position in the tth frame.

7. The method according to claim 6, characterized in that In step S400, the tracking result of the next frame is obtained according to the spatial weighted graph, specifically: Where F represents the FFT function, F -1 represents the inverse FFT function, ⊙ represents the element-wise product, W t represents a spatial weighted graph, H t represents the spatiotemporal context model, (x′ t+1 ,y′ t+1 ) represents the center point position coordinates of the target in the t+1th frame.

8. The method according to claim 7, characterized in that After step S400, the method further includes: Step S500: judging the distance between the mobile robot and the obstacle according to the tracking result of each frame, and controlling the mobile robot to track according to a preset distance-speed-movement mode principle according to the distance between the mobile robot and the obstacle.

9. The method according to claim 8, characterized in that The preset distance-speed-movement mode principle includes: When the distance is within the first range, the mobile robot stops moving forward and rotates in place; When the distance is within the second range, the mobile robot moves straight while rotating; When the distance is within a third range, the mobile robot moves straight without rotating; wherein the third range is larger than the second range, and the second range is larger than the first range.

Images