A visual tracking method and tracking device

By constructing a feature fusion network and a prediction head network, combined with a feature pooling structure and contrastive learning, the problem of tracking box drift in deep learning human tracking was solved, achieving more stable tracking results and improving the robustness and accuracy of robot tracking.

CN114638862BActive Publication Date: 2024-11-22TSINGHUA SHENZHEN INTERNATIONAL GRADUATE SCHOOL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210297392.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-24
Publication Date
2024-11-22
Estimated Expiration
2042-03-24

AI Technical Summary

Technical Problem

Existing deep learning-based person tracking methods are prone to tracking box drift in real-world scenarios, especially when there is interference around the target person, making accurate tracking difficult and affecting the performance of robot planning and control.

Method used

A tracking network is constructed using a feature fusion network and a prediction head network. By combining a feature pooling structure and contrastive learning, the template features are dynamically updated through the feature pooling structure, the relative position constraints between the head and body are increased, and the training network is optimized using an overall tracking loss function to reduce tracking box drift.

Benefits of technology

It effectively reduces cumulative error, improves the robustness of the tracking method, maintains the stability of long-term tracking, reduces tracking box drift, and improves tracking accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114638862B_ABST
    Figure CN114638862B_ABST
Patent Text Reader

Abstract

The present invention provides a visual tracking method and tracking device, the method comprising: acquiring a video to be tested containing a target person in real time; constructing a tracking network and training the tracking network with a collected pedestrian video data set, wherein the tracking network updates template features using a feature pool structure based on a contrastive learning structure; and using the trained tracking network to determine a target person frame of the target person in the video to be tested to obtain a tracking result. By adding a feature pool structure and optimizing the features of the template branch, the feature pool can dynamically update the template at low time complexity, better match the features of subsequent frames, effectively reduce the cumulative error, and alleviate the problem of tracking frame drift; the feature pool structure can keep the tracking network model stable in long-term sequence tracking, thereby improving the robustness of the tracking method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual tracking method and a tracking device. Background Art

[0002] Robotics involves many technologies such as perception, path planning, and mechanical control. As a key link in perception, human tracking technology determines the performance of robots. In recent years, human tracking technology has made great progress. There are two reasons for this: First, the improvement of hardware drive and computing power. Advanced computing methods such as GPU and cloud computing, combined with the storage and processing capabilities of massive data, allow deep learning methods to effectively extract human features in time and space dimensions, which provides possibilities for the development of deep learning in human tracking technology. Second, it is driven by upstream tasks. The upstream tasks of human tracking technology include human detection, pedestrian re-identification, video and image processing, etc. The continuous accumulation of upstream task technologies has pushed human tracking technology towards accurate tracking and real-time tracking. This enables human tracking technology to provide robots with more accurate, real-time and effective information in the planning and control links.

[0003] The deep learning-based person tracking method has strong generalization ability and can learn and process large-scale data. However, in real scenarios, when using deep learning-based methods to track the target person, the problem of tracking frame drift will occur, which directly affects the performance of subsequent robot planning and control. Especially when there are a lot of interference around the target person, the drift phenomenon is more serious, including the following three situations: First, when the target person is blocked by a crowd or object, it is difficult for the tracking frame to locate the blocked target person; second, when the target person walks side by side with other people, there will be a situation where the tracking frame tracks non-target people walking with him, or tracks the target and non-target people at the same time; third, when there are people with similar figures or similar clothes passing by the target person, the tracking frame is easily taken away.

[0004] The prior art lacks a visual tracking method that solves the problem of tracking frame drift.

[0005] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0006] In order to solve the existing problems, the present invention provides a visual tracking method and a tracking device.

[0007] In order to solve the above problems, the technical solution adopted by the present invention is as follows:

[0008] A visual tracking method comprises the following steps: S1: acquiring a video to be tested containing a target person in real time; S2: constructing a tracking network and training the tracking network with a collected pedestrian video data set, wherein the tracking network updates template features using a feature pool structure based on a contrastive learning structure; S3: determining a target person frame of the target person in the video to be tested using the trained tracking network to obtain a tracking result.

[0009] Preferably, the structure of the tracking network is constructed as follows: a feature fusion network is used to enhance and fuse the template features, head features and subsequent frame features in the feature pool to obtain a fused feature map; a prediction head network is used to predict the fused feature map to obtain the tracking results of subsequent frames.

[0010] Preferably, the feature fusion network adopts a Transformer network structure, including two mechanisms of self-attention and mutual attention; the prediction head network includes three parallel structures of a classification branch, a regression branch, and a center measurement branch; the classification branch is used for binary classification of the foreground and background of the image frame in the pedestrian video dataset; the regression branch is used for the regression of the bounding box of the image frame in the pedestrian video dataset; the center measurement branch is used to normalize the distance from the pixels in the prediction box to the target center.

[0011] Preferably, updating the template features using the feature pool structure based on the contrastive learning structure includes: feature pool F = {f i}, where f i is a storage frame, i is a positive integer; the storage frame is stored as a queue structure according to the subscript size, the smaller the subscript, the closer the storage position is; wherein, when i=1, it is a template frame, and when i>1, it is a subsequent frame; for the subsequent frame, the larger the product of the classification branch prediction confidence and the center measurement branch confidence of the frame, the smaller the subscript i; a threshold is set in advance, if the number of frames in the feature pool is greater than or equal to the threshold, the feature vectors corresponding to the frames in the feature pool equal to the threshold number are fused to obtain the template feature; if the number of frames in the feature pool is less than the threshold, the feature vectors corresponding to all frames in the feature pool are fused to obtain the template feature.

[0012] Preferably, a head frame and a full body frame are added to the feature pool, and the ratio and angle of the line segments at the centers of the head frame and the full body frame to the diagonal of the full body frame are used to jointly constrain the relative position constraints; the head frame and the full body frame both maintain a set of the feature pool structure, and the head frame and the full body frame of the target person in the same frame are stored in the same position in their respective pools.

[0013] Preferably, the features in the feature pool are fused in a weighted fusion manner, and the weight coefficient is The power of is obtained, and the weighted sum of the features is adjusted to make the sum of all weight coefficients equal to 1. The specific expression of the fusion result is:

[0014]

[0015] Among them, X k is the fused feature template obtained by the feature pool, and k is the number of frames taken from the fused feature pool.

[0016] Preferably, the collected pedestrian video dataset uses the overall tracking loss function L T Train the tracking network; the overall tracking loss function expression is the head trajectory constraint loss L H and the dense loss L C It consists of two parts:

[0017] L T =βL H +(1-β)L C

[0018] Among them, β is a hyperparameter;

[0019] Among them, the head constraint loss L H The expression is:

[0020]

[0021] Among them, γ1 and γ2 are hyper parameters, l is the distance between the center points of the labeled head frame and body frame, L is the length of the diagonal of the labeled body frame, and θ is the angle between l and L. is the value obtained from the corresponding prediction result;

[0022] The dense loss function L C The expression is:

[0023] L c =L cls +λ1L reg +λ2L cent

[0024] Among them, L cls is the classification loss, L reg is the regression loss, L cent is the center measurement loss, λ1 and λ2 are weight parameters;

[0025] The classification loss and center measurement loss are both expressed in the form of cross entropy loss, and the expression is:

[0026]

[0027] Where a is cls or cent, L a is the classification loss or the center measurement loss, j is the j-th frame sample, y aj is the label of the jth frame, p ajis the prediction confidence of the j-th frame classification branch or center measurement branch;

[0028] The prediction confidence expression of the center measurement branch is:

[0029]

[0030] Among them, l * 、r * ,t * 、b * They are the distances from the predicted center point to the left, right, upper, and lower boundaries of the label box;

[0031] The regression loss expression is:

[0032] L reg =L GIOU +α1L agg -α2L rep

[0033] Among them, L GIOU is the generalized intersection-over-union loss, L agg is the aggregation loss, L rep is the rejection loss, and α1 and α2 are weight parameters.

[0034] The generalized intersection-over-union loss function expression is:

[0035] L GIOU =1-GIOU(gt,b j )

[0036] The generalized intersection-union ratio expression is:

[0037]

[0038] Among them, gt is the label box, b j is the prediction box, C is able to enclose gt and b j The smallest frame.

[0039] The aggregation loss function expression is:

[0040]

[0041] Among them, gt j is the label box of the target person in the jth frame, p i is the predicted box belonging to the label box of the jth frame, |p j+ | is the number of candidate boxes predicted as positive samples in the jth frame;

[0042] The smooth l1 The function expression is:

[0043]

[0044] The rejection loss function expression is:

[0045]

[0046] Among them, b i is the prediction box, g j The box with the largest intersection-and-intersection ratio with the label box of the frame is predicted as the background, |p j+ | is the number of candidate boxes predicted as positive samples in the jth frame, IOG is b i and g j The intersection and union ratio of

[0047] The smooth ln The function expression is:

[0048]

[0049] Among them, σ∈[0,1) represents the smoothing parameter.

[0050] Preferably, using a trained tracking network to determine the target person frame of the target person in the video to be tested includes the following steps: inputting the features of the image frame in the video to be tested and the template features into the trained tracking network at the same time; storing the image frame in the corresponding position of the feature pool according to the product value of the prediction confidence of the classification branch and the prediction confidence of the center measurement branch; using the vector index whose square root is greater than 0.5 based on the product value of the prediction confidence of the classification branch and the prediction confidence of the center measurement branch as a candidate frame index set; searching for a candidate frame set in the regression branch based on the candidate frame index set; and finding the frame with the largest prediction confidence of the classification branch in the candidate frame set as the target person frame.

[0051] Preferably, it also includes: performing motion tracking on the target person based on the target person frame to obtain the tracking result, specifically including: obtaining a depth map from the video to be tested; calibrating the depth map with the image frame in the video to be tested; obtaining the depth value of each pixel point in the corresponding area within the target person frame and calculating the average value of the depth value as the distance of the target person.

[0052] The present invention also provides a tracking device, which adopts any of the above-mentioned visual tracking methods.

[0053] The beneficial effects of the present invention are: providing a visual tracking method and a tracking device, by increasing the feature pool structure and optimizing the features of the template branch, the feature pool can dynamically update the template with low time complexity, better match the subsequent frame features, effectively reduce the cumulative error, alleviate the tracking frame drift problem, and improve the robustness of the tracking method.

[0054] Furthermore, the feature pooling structure enables the tracking network model to remain stable in long-term series tracking and improve the robustness of the tracking method.

[0055] Furthermore, the present invention increases the constraints on the relative positions of the head frame and the full body frame. The head-body constraint assumes that when the same person walks, the relative positions of the head and body remain unchanged. This innovatively introduces common sense assumptions, increases relative position constraints, reduces the search space for solutions, effectively suppresses the drift problem of the tracking frame, and alleviates the interference caused by the sudden inclusion of confusing objects in the tracking frame.

[0056] Furthermore, the present invention uses an overall tracking loss function to optimize the network training, taking into account the cohesive force of the positive sample full-body label frame on the predicted frame and the repulsive force on the predicted background frame. Through training, the cohesive force and repulsive force between samples are increased, the drift problem of the tracking frame is reduced, and the tracking effect is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of a visual tracking method in an embodiment of the present invention.

[0058] Figure 2 Schematic diagram of a robot tracking method in an embodiment of the present invention.

[0059] Figure 3 The present invention provides a method for constructing a tracking network and training the tracking network using a collected pedestrian video data set.

[0060] Figure 4 Schematic diagram of the structure of the feature fusion network in an embodiment of the present invention.

[0061] Figure 5 Schematic diagram of the structure of the prediction head network in an embodiment of the present invention.

[0062] Figure 6 It is a schematic diagram of the structure of the tracking network feature pool module in an embodiment of the present invention.

[0063] Figure 7 It is a schematic diagram of a method for determining a target person frame of a target person in the video to be tested by a trained tracking network in an embodiment of the present invention.

[0064] Figure 8 It is a schematic diagram of a method for obtaining tracking results in an embodiment of the present invention.

[0065] Fig. 9 Schematic diagram of a method for tracking a target person's trajectory in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the technical problems, technical solutions and beneficial effects to be solved by the embodiments of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0067] It should be noted that when a component is referred to as being "fixed to" or "disposed on" another component, it can be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and circuit connection.

[0068] It should be understood that the orientation or position relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0069] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0070] There are three reasons for the problem of tracking frame drift: first, the deep learning model is difficult to converge. On the one hand, it is due to the high complexity of the deep learning network itself, and because the data set is taken from real scenes, the real scenes are relatively complex; on the other hand, compared with image data, video data adds a time dimension, and character tracking requires the use of video data training, which further increases the difficulty of model convergence; second, conventional algorithms ignore the constraints of additional supervisory information, such as walking trajectory information, images of unobstructed parts, ReID information, etc., which makes the algorithm unstable and the tracking frame drifts; third, the tracking algorithm needs to use the non-maximum suppression method for post-processing to select the optimal tracking frame from many candidate frames, but this method is difficult to adjust the threshold for deleting frames because non-maximum suppression is sensitive to the threshold setting.

[0071] Based on the above analysis, the present invention provides a visual tracking method.

[0072] like Figure 1As shown, the present invention provides a visual tracking method, comprising the following steps:

[0073] S1: Acquire the video to be tested containing the target person in real time;

[0074] S2: construct a tracking network and train the tracking network using the collected pedestrian video dataset, wherein the tracking network updates the template features using a feature pooling structure based on a contrastive learning structure;

[0075] S3: Use the trained tracking network to determine the target person frame of the target person in the video to be tested, and obtain a tracking result.

[0076] The present invention increases the feature pool structure and optimizes the features of the template branch. The feature pool can dynamically update the template with low time complexity, better match the features of subsequent frames, effectively reduce the cumulative error, alleviate the problem of tracking frame drift, and improve the robustness of the tracking method.

[0077] Furthermore, the feature pooling structure enables the tracking network model to remain stable in long-term series tracking and improve the robustness of the tracking method.

[0078] In a specific embodiment, the above method is applied to a tracking robot.

[0079] like Figure 2 As shown, the tracking device of the present invention is a tracking robot, and a tracking method of the tracking robot specifically comprises the following steps:

[0080] A1: Get the robot's RGBD camera information and obtain the RGB image and depth image from it;

[0081] A2: Input the RGB image into the tracking network RobTranSim. The tracking network is based on a contrastive learning structure and uses a feature pooling structure to update the template features;

[0082] A3: Get the target person frame;

[0083] A4: Obtain depth information based on the depth map;

[0084] A5: Estimate the trajectory of the target person;

[0085] A6: Drive the robot to track the target person.

[0086] It is understandable that the tracking robot of the present invention is equipped with an RGBD camera to obtain the video to be tested containing the target person.

[0087] like Figure 3 As shown, the present invention provides a method for constructing a tracking network and training the tracking network with a collected pedestrian video dataset, which specifically includes the following steps:

[0088] B1: Collect annotated pedestrian video dataset:

[0089] B2: Constructing the character tracking network RobTranSim:

[0090] B3: Build feature pool and update template features:

[0091] B4: Using the overall tracking loss function L T , the tracking network is trained using the collected pedestrian video dataset.

[0092] B5: Use the trained network to track the target person in the video to be tested and obtain the tracking result.

[0093] In step B1, firstly, the pedestrian video is collected, and then the label frame is annotated frame by frame. The label frame includes the full body frame and the head frame. The full body frame annotates the horizontal and vertical coordinates of the upper left corner of the target person frame and the length and width information of the frame, and the head frame annotates the horizontal and vertical coordinates of the center position of the head. During the tracking process, the network outputs the predicted frame information of each frame, which includes the predicted head frame and the predicted full body frame.

[0094] In step B2, the tracking network is based on contrastive learning and adopts the following structure:

[0095] The feature fusion network is used to enhance and fuse the template features, head features and subsequent frame features in the feature pool to obtain a fused feature map;

[0096] The prediction head network is used to predict the fused feature map to obtain the tracking results of subsequent frames.

[0097] like Figure 4 As shown in the figure, the feature fusion network adopts the Transformer network structure, which includes two mechanisms: self-attention and mutual attention. Compared with the fusion method of the convolutional layer, the Transformer can learn global information, and its nonlinear structure can fuse more effective representations.

[0098] like Figure 5 As shown, the prediction head network includes three parallel structures: a classification branch, a regression branch, and a center measurement branch; the classification branch is used for binary classification of the foreground and background of the image frame in the pedestrian video dataset; the regression branch is used for the regression of the bounding box of the image frame in the pedestrian video dataset; the center measurement branch is used to normalize the distance from the pixels in the prediction box to the target center.

[0099] Among them, the box category whose intersection-over-intersection ratio with the label box marked in step B1 is greater than 0.5 is 1, and the box category whose intersection-over-intersection ratio is less than or equal to 0.5 is 0.

[0100] In step B3, updating the template features using the feature pool structure based on the contrastive learning structure includes:

[0101] Feature pool F = {f i}, where f i is a storage frame, i is a positive integer; the storage frame is stored as a queue structure according to the subscript size, the smaller the subscript, the closer the storage position is; wherein, when i=1, it is a template frame, and when i>1, it is a subsequent frame; for the subsequent frame, the larger the product of the prediction confidence of the classification branch of the frame and the confidence of the center measurement branch, the smaller the subscript i;

[0102] Based on the assumption that the relative position of the pedestrian's head and body remains unchanged, relative position constraints are added to reduce the search space of the solution, and the drift problem of the tracking frame is effectively suppressed; a head frame and a full body frame are added to the feature pool, and the relative position constraints are jointly constrained by the ratio and angle of the line segments of the center of the head frame and the full body frame to the diagonal of the full body frame.

[0103] Use the feature extraction network to extract frame features x for the current frame i , store and update the feature pool; pre-set a threshold, if the number of frames in the feature pool is greater than or equal to the threshold, then fuse the feature vectors corresponding to the frames in the feature pool that are equal to the threshold to obtain the template feature; if the number of frames in the feature pool is less than the threshold, then fuse the feature vectors corresponding to all frames in the feature pool to obtain the template feature. The head frame and the full body frame both maintain a set of the feature pool structure, and the head frame and the full body frame of the target person in the same frame are stored in the same position in their respective pools.

[0104] In a specific embodiment, the feature extraction network is ResNet50.

[0105] In a preferred embodiment, the features in the feature pool are fused in a weighted fusion manner, and the weight coefficient is The power of is obtained, and the weighted sum of the features is adjusted to make the sum of all weight coefficients equal to 1. The specific expression of the fusion result is:

[0106]

[0107] Among them, X k is the fused feature template obtained by the feature pool, and k is the number of frames taken from the fused feature pool.

[0108] The present invention has three advantages in constructing a feature pool. First, it integrates richer information instead of only considering the features of the template frame, and the more important the feature, the greater its weight coefficient; second, the amount of calculation is small. When the feature pool is updated, only the incremental part of the frame features newly entered into the pool needs to be added, and there is no need to recalculate the fusion result in each iteration; third, it is more effective for tracking long-term series, and the longer the time, the more stable the template frame features.

[0109] Then, use the overall tracking loss function LT The tracking network is trained; the overall tracking function considers the aggregation force of the positive sample full body label frame on the prediction frame and the repulsion force on the predicted background frame, and increases the aggregation force and repulsion force between samples through training, reduces the tracking frame drift problem, and achieves better tracking effect.

[0110] In a specific embodiment, the overall tracking loss function expression is composed of the head trajectory constraint loss L H and the dense loss L C It consists of two parts:

[0111] L T =βL H +(1-β)L C

[0112] Among them, β is a hyperparameter; during the training process, the overall tracking loss function moves in a decreasing direction.

[0113] Among them, the head constraint loss L H The expression is:

[0114]

[0115] Among them, γ1 and γ2 are hyperparameters, l is the distance between the center points of the labeled head frame and the full body frame, L is the diagonal length of the labeled full body frame, and θ is the angle between l and L. is the value obtained from the corresponding prediction result;

[0116] The dense loss function L C The expression is:

[0117] L c =L cls +λ1L reg +λ2L cent

[0118] Among them, L cls is the classification loss, L reg is the regression loss, L cent is the center measurement loss, λ1 and λ2 are weight parameters;

[0119] The classification loss and center measurement loss are both expressed in the form of cross entropy loss, and the expression is:

[0120]

[0121] Where a is cls or cent, L a is the classification loss or the center measurement loss, j is the j-th frame sample, y aj is the label of the jth frame, p ajis the prediction confidence of the j-th frame classification branch or center measurement branch;

[0122] The prediction confidence expression of the center measurement branch is:

[0123]

[0124] Among them, l * 、r * ,t * 、b * They are the distances from the predicted center point to the left, right, upper, and lower boundaries of the label box;

[0125] The regression loss expression is:

[0126] L reg =L GIOU +α1L agg -α2L rep

[0127] Among them, L GIOU is the generalized intersection-over-union loss, L agg is the aggregation loss, L rep is the rejection loss, and α1 and α2 are weight parameters.

[0128] The generalized intersection-over-union loss function expression is:

[0129] L GIOU =1-GIOU(gt,b j )

[0130] The generalized intersection-union ratio expression is:

[0131]

[0132] Among them, gt is the whole body label box, b j is the whole body prediction box, C is able to cover gt and b j The smallest frame.

[0133] The aggregation loss function expression is:

[0134]

[0135] Among them, gt j is the full body label frame of the target person in the jth frame, p i is the full body prediction box belonging to the j-th frame label box, |p j+ | is the number of candidate boxes predicted as positive samples in the jth frame;

[0136] The smooth l1 The function expression is:

[0137]

[0138] The rejection loss function expression is:

[0139]

[0140] Among them, b i is the whole body prediction box, g j The frame with the largest intersection and ratio with the full-body label frame of the frame is predicted as the background frame, |p j+ | is the number of candidate boxes predicted as positive samples in the jth frame, IOG is b i and g j The intersection and union ratio of

[0141] The smooth ln The function expression is:

[0142]

[0143] Among them, σ∈[0,1) represents the smoothing parameter.

[0144] In step B5, if Figure 6 In the feature pool structure shown, the template frame is taken as the first frame and stored at the first position of the feature pool, and the position of the template frame is not adjusted afterwards.

[0145] For each subsequent frame fi of the video sequence, it is input into the feature extraction network of the feature pool to obtain the feature vector.

[0146] Set a threshold. If the number of frames in the feature pool is greater than or equal to the threshold, fuse the feature vectors corresponding to the first k frames in the feature pool that are equal to the threshold to obtain the template feature. If the number of frames in the feature pool is less than the threshold, fuse the feature vectors corresponding to all frames in the pool. The threshold is a positive integer greater than or equal to 1. In this embodiment, the threshold is 8, at which time the marginal utility of the feature pool is the largest. In particular, when the threshold is 1, the network degenerates into a standard contrastive learning network.

[0147] like Figure 7 As shown, using the trained tracking network to determine the full body frame of the target person in the video to be tested includes the following steps:

[0148] Inputting the features of the image frame in the video to be tested and the template features into the trained tracking network at the same time;

[0149] Storing the image frame in a corresponding position of the feature pool according to the product value of the prediction confidence of the classification branch and the prediction confidence of the center measurement branch;

[0150] A vector index whose square root value is greater than 0.5 according to the product of the prediction confidence of the classification branch and the prediction confidence of the center measurement branch is used as a candidate box index set; and a candidate box set is searched in the regression branch according to the candidate box index set;

[0151] Find the frame with the highest confidence level predicted by the classification branch in the candidate frame set as the full-body frame of the target person.

[0152] Specifically, the features of frame fi and template features are simultaneously input into the trained network, and the following calculation is performed: Figure 5 The prediction results of the four branches.

[0153] Find the box with the largest prediction confidence of the classification branch in the candidate box set as the tracking result of the frame.

[0154] like Figure 8 As shown, the tracking method of the present invention further includes: tracking the motion of the target person according to the target person frame to obtain the tracking result, specifically including:

[0155] Acquire a depth map from the video to be tested;

[0156] Calibrate the depth map with the image frame in the video to be tested;

[0157] The depth value of each pixel point in the corresponding area within the full body frame of the target person is obtained, and the average value of the depth values ​​is calculated as the distance of the target person.

[0158] like Fig. 9 As shown, corresponding to step A5, the tracking of the target person's trajectory using the device of the present invention includes the following steps:

[0159] C1: Get depth map:

[0160] In this embodiment, the depth map portion is extracted from the information of the RGBD camera.

[0161] C2: Calibrate the depth map with the RGB map:

[0162] In order to obtain a clear depth map, in this embodiment, the opencv library is first used for calibration to obtain the internal and external parameters of the camera, and then the calibration is performed to achieve epipolar alignment. The disparity map is obtained, and then the depth map is calculated by the HashMatch method, and the calculated depth map is spliced ​​with the original camera depth map.

[0163] C3: Capture the depth map within the character frame:

[0164] In this embodiment, in the depth map, the depth of the corresponding position in the prediction frame of each frame of the RGB image is obtained.

[0165] C4: Calculate the average depth:

[0166] In this embodiment, the depth within the frame is predicted based on the RGB image of each frame, and the average distance is calculated.

[0167] C5: Update robot distance and angle.

[0168] The above process also includes the following steps:

[0169] C6: Adjust the camera to keep the character frame in the center of the field of view;

[0170] C7: Calculate the camera's deflection angle relative to the previous position.

[0171] In a specific embodiment, the method and device described above are used for testing. First, a video of the target person is collected in a scene with a dense crowd, and the head frame and the full body frame are annotated for each frame. The tracking network model is trained with the first 70% of the images, and the tracking network model is trained on a server composed of 8 RTX 2080Ti; the tracking effect is tested with the last 30% of the images. By comparing the test results with the single target tracking method in the prior art, it can be seen that the tracking accuracy of this embodiment is higher.

[0172] Table 1 Experimental results

[0173] Tracking Methods ATOM SiamRPN++ TransT Method of the present invention Accuracy (%) 80.5 76.8 83.9 85.3

[0174] An embodiment of the present application also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor is used to execute at least the method described above when executing the computer program.

[0175] An embodiment of the present application also provides a storage medium for storing a computer program, which at least performs the method described above when executed.

[0176] An embodiment of the present application further provides a processor, which executes a computer program and at least performs the method described above.

[0177] The storage medium may be implemented by any type of volatile or non-volatile storage device, or a combination thereof. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAMEnhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0179] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0180] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0181] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0182] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0183] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0184] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0185] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0186] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A visual tracking method, characterized in that: The steps include: S1: Acquire the video to be tested containing the target person in real time; S2: construct a tracking network and train the tracking network using the collected pedestrian video dataset, wherein the tracking network updates the template features using a feature pooling structure based on a contrastive learning structure; S3: using the trained tracking network to determine the target person frame of the target person in the video to be tested, and obtain a tracking result; The collected pedestrian video dataset uses the overall tracking loss function L T training the tracking network; The overall tracking loss function expression is given by the head trajectory constraint loss L H and the dense loss L C Two parts composition: L T =βL H +(1-β)L C Among them, β is a hyperparameter; Among them, the head constraint loss L H The expression is: Among them, γ1 and γ2 are hyper parameters, l is the distance between the center points of the labeled head frame and the full body frame, L is the diagonal length of the labeled full body frame, and θ is the angle between l and L. is the value obtained from the corresponding prediction result; The dense loss function L C The expression is: L c =L cls +λ1L reg +λ2L cent Among them, L cls is the classification loss, L reg is the regression loss, L cent is the center measurement loss, λ1 and λ2 are weight parameters; The classification loss and center measurement loss are both expressed in the form of cross entropy loss, and the expression is: Where a is cls or cent, L a is the classification loss or the center measurement loss, j is the j-th frame sample, y aj is the label of the jth frame, p aj is the prediction confidence of the j-th frame classification branch or center measurement branch; The prediction confidence expression of the center measurement branch is: Among them, l * 、r * ,t * , b * They are the distances from the predicted center point to the left, right, upper, and lower boundaries of the full body label box; The regression loss expression is: L rep =L GIOU +α1L agg -α2L rep Among them, L GIOU is the generalized intersection-over-union loss, L agg is the aggregation loss, L rep is the rejection loss, α1 and α2 are weight parameters; The generalized intersection-over-union loss function expression is: L GIOU =1-GIOU(gt,b j ) The generalized intersection-union ratio expression is: Among them, gt is the whole body label box, b j is the whole body prediction box, C is able to cover gt and b j The smallest frame; The aggregation loss function expression is: Among them, gt j is the full body label frame of the target person in the jth frame, p i is the full body prediction box belonging to the j-th frame label box, |p j+ | is the number of candidate boxes predicted as positive samples in the jth frame; The smooth l1 The function expression is: The rejection loss function expression is: Among them, b i is the whole body prediction box, g j The frame with the largest intersection and ratio with the full-body label frame of the frame is predicted as the background frame, |p j+ | is the number of candidate boxes predicted as positive samples in the jth frame, IOG is b i and g j The intersection and union ratio of The smooth ln The function expression is: Among them, σ∈[0,1) represents the smoothing parameter.

2. The visual tracking method according to claim 1, characterized in that: The structure of the tracking network is as follows: The feature fusion network is used to enhance and fuse the template features, head features and subsequent frame features in the feature pool to obtain a fused feature map; The prediction head network is used to predict the fused feature map to obtain the tracking results of subsequent frames.

3. The visual tracking method according to claim 2, characterized in that: The feature fusion network adopts the Transformer network structure, which includes two mechanisms: self-attention and mutual attention; The prediction head network includes three parallel structures: a classification branch, a regression branch, and a center measurement branch; The classification branch is used for binary classification of foreground and background of image frames in the pedestrian video dataset; The regression branch is used for regression of the bounding box of the image frame in the pedestrian video dataset; The center measurement branch is used to normalize the distance from the pixels in the prediction box to the target center.

4. The visual tracking method according to claim 3, characterized in that: Using the feature pool structure based on the contrastive learning structure to update the template features includes: Feature pool F = {f i }, where f i is a storage frame, i is a positive integer; the storage frame is stored as a queue structure according to the subscript size, the smaller the subscript, the closer the storage position is; wherein, when i=1, it is a template frame, and when i>1, it is a subsequent frame; for the subsequent frame, the larger the product of the prediction confidence of the classification branch of the frame and the confidence of the center measurement branch, the smaller the subscript i; A threshold is set in advance. If the number of frames in the feature pool is greater than or equal to the threshold, the feature vectors corresponding to the frames in the feature pool that are equal to the threshold are fused to obtain the template features; if the number of frames in the feature pool is less than the threshold, the feature vectors corresponding to all frames in the feature pool are fused to obtain the template features.

5. The visual tracking method according to claim 4, characterized in that: Adding a head frame and a full body frame to the feature pool, and using the ratio and angle of the line segments of the center of the head frame and the full body frame to the diagonal of the full body frame to jointly constrain the relative position constraints; The head frame and the full body frame both maintain a set of the feature pool structures, and the head frame and the full body frame of the target person in the same frame are stored in the same position in their respective pools.

6. The visual tracking method according to claim 5, characterized in that: The features in the feature pool are fused using a weighted fusion method, and the weight coefficient is The power of is obtained, and the weighted sum of the features is adjusted to make the sum of all weight coefficients equal to 1. The specific expression of the fusion result is: Among them, X k is the fused feature template obtained by the feature pool, and k is the number of frames taken from the fused feature pool.

7. The visual tracking method according to claim 3, characterized in that: Determining the target person frame of the target person in the video to be tested by using the trained tracking network comprises the following steps: Inputting the features of the image frame in the video to be tested and the template features into the trained tracking network at the same time; Storing the image frame in a corresponding position of the feature pool according to the product value of the prediction confidence of the classification branch and the prediction confidence of the center measurement branch; A vector index whose square root of the product of the prediction confidence of the classification branch and the prediction confidence of the center measurement branch is greater than 0.5 is used as a candidate box index set; Searching for a candidate box set in the regression branch according to the candidate box index set; Find the box with the highest confidence in classification branch prediction in the candidate box set as the target person box.

8. The visual tracking method according to claim 7, characterized in that: Also includes: Tracking the target person's motion according to the target person frame to obtain the tracking result specifically includes: Acquire a depth map from the video to be tested; Calibrate the depth map with the image frame in the video to be tested; The depth value of each pixel point in the corresponding area within the target person frame is obtained, and the average value of the depth values ​​is calculated as the distance of the target person.

9. A tracking device, characterized in that: Adopt the visual tracking method as described in any one of claims 1-8.