A 3D human body tracking method for quadruped robots in complex environments
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2024-03-04
- Publication Date
- 2026-06-09
Smart Images
Figure CN118071798B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video target tracking technology, specifically to a 3D human tracking method for quadruped robots in complex environments. Background Technology
[0002] Single-object tracking is a hot topic in computer vision, with wide applications in many industries, such as intelligent video surveillance, autonomous driving, and human-computer interaction in robotics. Driven by deep learning, single-object tracking tasks are gradually expanding from 2D to 3D. The main approach involves using the point cloud of the first frame of a sequence and its initial bounding box for a specified target, to output the spatial location of the target object in the form of a 3D tracking bounding box in the remaining frames. The 3D human tracking involved in this invention is also a sub-task within single-object tracking.
[0003] The main technical challenges of using the human body as the tracking target are the continuous and arbitrary changes of the human body, the occlusion of the tracked human body by other objects, the rapid movement of the tracked target, and the small size of the human body. These factors result in significant changes in the appearance of the tracked target in each frame and make it easily affected by the appearance of the surrounding environment.
[0004] Currently, the mainstream algorithms for 3D human single-target tracking tasks can be broadly divided into two categories: correlation filter methods and deep learning methods. Correlation filter methods typically utilize the texture features of targets in a sequence to obtain a target feature library, then compare the features of each search block in the search sequence with the library, selecting the most similar target as the moving target. However, this method is usually slow. Deep learning methods use CNN networks combined with activation functions for self-learning and parameter tracking. Recently, with the introduction of infrastructures such as PointNet, training methods and network models have made significant progress in both accuracy and efficiency. However, most current 3D human tracking in this area is geared towards large-scale computer vision datasets, such as the KITTI, WAYMO, and NuScene datasets, with relatively few applications specifically for quadruped robots. Once the application scenario changes to a real-world platform, such as when robot onboard sensors record online foreground data while walking, running, or climbing stairs, the tracking accuracy drops significantly. Therefore, 3D human tracking faces numerous unresolved bottlenecks in practical platform applications and high-order task time. Summary of the Invention
[0005] To address the challenges of tracking accuracy, tracking target, and application platform in 3D human body tracking, this invention provides a 3D human body tracking method for quadruped robots in complex environments. It utilizes point cloud and video as complementary dual inputs to mitigate interference from occlusion and lighting changes in complex environments, exhibiting high robustness. By incorporating an anti-bumping algorithm, this method achieves high tracking accuracy and speed, and can be directly applied to real-time 3D human body tracking tasks for quadruped robots in complex environments.
[0006] This invention provides a 3D human tracking method for quadruped robots in complex environments, characterized in that the method includes:
[0007] Acquire the sequence to be tracked; each frame of the sequence to be tracked includes a point cloud frame and the corresponding video frame;
[0008] Template frames and search frames are pre-determined based on the sequence to be tracked; the template frames include point cloud frames of the target human body object to be tracked and corresponding video frames, and the search frames include point cloud frames of the target human body object to be searched and corresponding video frames.
[0009] The pre-determined template frame and search frame are input into the trained 3D human body tracking model to track the target human body object and output the tracking results;
[0010] The trained 3D human tracking model includes:
[0011] The dual-branch architecture multimodal feature extraction and fusion module is used to input the template frame and the search frame into the corresponding branches for feature extraction and fusion, so as to obtain the template frame features and the search frame features.
[0012] The feature learning encoder under the hierarchical attention mechanism is connected to the multimodal feature extraction and fusion module of the dual-branch architecture, and is used to perform feature interaction and enhancement on the template frame features and search frame features to obtain an attention feature map;
[0013] A human-guided 3D target bounding box proposal and verification decoder is connected to the feature learning encoder under the hierarchical attention mechanism. The initial tracking result is obtained by extracting and verifying the human-guided target from the attention feature map.
[0014] The anti-bump correction unit is connected to the human-guided 3D target box proposal and verification decoder. Based on the robot's recorded frame velocity and its maximum response score, it corrects the initial tracking results using the anti-bump correction algorithm.
[0015] This invention provides a 3D human tracking method for quadruped robots in complex environments. It utilizes point cloud and video as complementary dual inputs to mitigate interference from occlusion and lighting changes in complex environments. It exhibits high robustness and incorporates an anti-bumping algorithm, resulting in high tracking accuracy and speed. This method can be directly applied to real-time 3D human tracking tasks for quadruped robots in complex environments.
[0016] Preferably, the dual-branch architecture multimodal feature extraction and fusion module includes a dual-branch architecture multimodal feature extraction module and a multimodal feature alignment and fusion module based on multi-level interpolation;
[0017] Each branch of the dual-branch architecture multimodal feature extraction module includes a point cloud feature extraction unit and an image feature extraction unit; the point cloud feature extraction unit is used to extract point cloud features; the image feature extraction unit is used to extract video features;
[0018] The multimodal feature alignment and fusion module based on multi-level interpolation is connected to the point cloud feature extraction unit and the image feature extraction unit respectively, and is used for feature alignment and fusion of point cloud features and video features under each branch.
[0019] Preferably, the point cloud feature extraction unit extracts point cloud features through a Point-MAE network.
[0020] The image feature extraction unit extracts video features through the MobileFormer network.
[0021] The feature alignment and fusion steps for point cloud features and video features include:
[0022] The general farthest point sampling algorithm is used to output a total of K point groups and their corresponding point cloud centers. Where K represents the number of point cloud feature groups after point downsampling;
[0023] Using the inverse distance weights w in the original point cloud j Point cloud features Interpolation to each point cloud Point cloud representation after interpolation Right now:
[0024]
[0025] The parameter ∈ is a number that tends towards infinity;
[0026] Convert the interpolated point cloud representation p′ into two-dimensional coordinates. And iterate through all points to form new point cloud features, the new point cloud features having the same dimension as the corresponding video features;
[0027] Multilayer perceptron (MLP) is used to extract interaction information, and the features of the two modalities are merged in a new feature channel to obtain point cloud-video features F. pv .
[0028] The point cloud features and video features after feature extraction differ significantly in dimensionality, and there is no unified alignment method between them. An intuitive solution is to artificially add new dimensions by repeatedly applying the point cloud features until they match the video features. However, this method loses a large number of spatial correspondences between the two modalities. To better achieve information complementarity under multimodal input, this invention first performs feature alignment on the point cloud features and video features after feature extraction to make their dimensions consistent, and then merges them, finally outputting the point cloud-video fused feature F. pv .
[0029] Preferably, the step of inputting the pre-determined template frame and search frame into the trained 3D human body tracking model to track the target human body object and output the tracking result includes:
[0030] For the current search frame, perform the following tracking steps: using the center point of the target box in the template frame as a reference, divide the current search frame into a region with a preset size and distance. Input the template frame and the current search frame after dividing the region into the trained 3D human tracking model to track the target human object and output the tracking result of the current search frame. Use the tracking result of the current search frame as the template frame for the next search frame, and repeat the above tracking steps until all search frames have been traversed and the tracking result is output.
[0031] Preferably, the dual-branch architecture includes a template branch and a search branch; the template frame is input to the template branch for feature extraction and fusion to obtain point cloud-video features under the template branch, which are the template frame features; the search frame is input to the search branch for feature extraction and fusion to obtain point cloud-video features under the search branch, which are the search frame features.
[0032] Preferably, the feature learning encoder under the hierarchical attention mechanism includes a self-attention local enhancement unit and a mutual attention local enhancement unit;
[0033] The self-attention local enhancement unit is used to thoroughly fuse adjacent features of point cloud and video, and performs a multi-head attention mechanism operation Attn(Q,V,K); the point cloud-video features under the template branch and the point cloud-video features under the search branch are input into the self-attention local enhancement unit to calculate the self-attention feature map A after template branch enhancement. t Self-attention feature map A after search branch enhancement s The calculation formula is as follows:
[0034]
[0035]
[0036]
[0037] Here, Q represents all possible features, K represents features that a person might exhibit, V represents the detailed information contained within different features, the superscript T represents the matrix transpose operation, and d k The set floating-point number;
[0038] The mutual attention local enhancement unit is used to further fuse the data output by the self-attention local enhancement unit. It follows the same structure as the self-attention local enhancement unit, adds residual processing to the output, and uses the last layer of self-attention feature map of the template branch as the V and K values, and the last layer of self-attention feature map of the search branch as the Q value to perform multi-head attention mechanism operation and output the enhanced mutual attention feature map.
[0039] Preferably, the initial tracking result is obtained by performing human-guided target extraction and verification on the attention feature map, including:
[0040] The mutual attention feature map of the last layer is input into the human-guided 3D target box proposal and verification decoder, and M candidate boxes are generated through the latent center generation part and clustering part of the P2B-like network.
[0041] A center-based regression head is used to predict several object attributes. The regression head consists of four heads, including a heatmap head. Local offset head Yaw Angle Head and confidence head For each candidate bounding box, 200 points are sampled in the box's 3D space, and the corresponding predicted value H is calculated based on the heatmap value of each point. x,y,θ,c The average value is then calculated using the following formula:
[0042]
[0043] Where d is the Euclidean distance between the center of the 3D candidate box and the point position calculated by the model, and the predicted value H is... x,y,θ,c =1 corresponds to the point being at the center of the human body object, H x,y,θ,c <0.1 corresponds to the background;
[0044] The multi-head perceptron (MLP) is used to select the candidate with the highest predicted value from the filtered candidate boxes. The candidate bounding boxes are used as the initial tracking results for the i-th search frame. i ={xi ,y i ,z i ,l i ,w i ,h i ,θ i ,c i}; where b i The eight dimensions refer to the x-coordinate, y-coordinate, z-coordinate, length, width, height, yaw angle, and confidence level of the target bounding box, respectively.
[0045] Human-oriented 3D bounding box verification is performed. Based on pre-defined prior values of the target human body, M candidate bounding boxes are filtered using human-oriented methods, retaining only those 3D bounding boxes that conform to the human body size. Output the current three-dimensional spatial information of the target human body object. in, The values in parentheses refer to the dimensions. Dimensions 1-3 are the x, y, and z coordinates of the target box in 3D space, respectively.
[0046] Preferably, the initial tracking results are corrected using an anti-bump correction algorithm based on the robot's recorded velocity and maximum response score for each frame, including:
[0047] The robot's built-in speed measurement module records the speed of each frame in real time. and its maximum response score For each frame I t Using the three-dimensional spatial information X of the target human body object t Based on the three-dimensional spatial information X of the preceding ten consecutive frames, the velocity value of the target human object in this frame is calculated:
[0048]
[0049] When the maximum response score R of a certain frame i If the network's pre-defined conditions are met, the initial tracking result is output as the tracking result for the i-th frame; otherwise, the anti-bump correction algorithm is activated, and the variable n is reset to 0.
[0050] The anti-bump algorithm is based on the tracking results generated in the previous frame. Adaptively expand the search area:
[0051]
[0052]
[0053]
[0054] in, The values in parentheses refer to the dimensions, with dimensions 4-6 being the length, width, and height of the target box, respectively.
[0055] With X t-1 Point cloud is extracted from the sampling center. And set the variable n = n + 1, and update the tracker's response in the extracted point cloud. Up to maximum response score R t And move to the next frame t+1;
[0056] Repeat the anti-bump algorithm steps until the maximum detected response R is reached. i If the network's pre-defined conditions are met, i.e. the tracking box catches up with the target object again, then the 3D target box of each frame within that time period, corrected by the anti-bumping correction algorithm, is output. This serves as the tracking result for that period of time.
[0057] Preferably, the training steps of the 3D human body tracking model include:
[0058] The sequence to be tracked is input into the server for reading and divided into training and test sets;
[0059] When reading each frame of the sequence to be tracked, a template frame and a search frame are pre-determined. The point cloud data in the template frame and the search frame are normalized, and the image data is cropped and augmented.
[0060] For a total of T frames to be tracked, the final training loss function is derived from the ground truth. and network model prediction results The calculation yields the following expression:
[0061]
[0062] The loss function is calculated as follows for the tracking results in frame t:
[0063]
[0064] in, This represents the confidence loss used to identify the foreground. Let λ represent the Smooth-L1 loss used to supervise bounding box regression. cls and λ reg These represent the weights of the confidence loss and the L1 loss, respectively. This represents the ground truth value of the 3D bounding box of the target human object in this frame. The 3D bounding box prediction result of the target human object in this frame consists of eight variables: length l, width w, height h, x-coordinate, y-coordinate, z-coordinate, rotation angle θ, and confidence score. The length, width, and height values are consistent with the initial frame template bounding box values. The coordinates and rotation angle are calculated as follows:
[0065] The 3D human tracking model network was trained on the server using the Adam optimizer, which reduced the overall loss value L of the network loss function. total Optimize the network parameters until convergence.
[0066] This invention designs a feature learning encoder based on a hierarchical attention mechanism and conducts human-oriented network training, which can effectively address the special challenges of the target human body object, such as its susceptibility to deformation and small scale.
[0067] This invention also provides a 3D human tracking system for quadruped robots in complex environments, characterized by comprising:
[0068] The LiDAR and camera mounted on the robot are connected to the processor for communication.
[0069] The lidar and camera acquire the sequence to be tracked and send it to the processor;
[0070] The processor contains a pre-trained 3D human tracking network model, which is used to execute a 3D human tracking method for quadruped robots in complex environments.
[0071] Using this system to track target human objects has the advantages of high tracking accuracy and fast tracking speed.
[0072] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a 3D human body tracking method for quadruped robots in complex environments. It utilizes point cloud and video as complementary dual inputs, which alleviates interference from occlusion of the target human body and changes in lighting in complex environments. It has high robustness and integrates an anti-bumping algorithm. The method has high tracking accuracy and fast speed, and can be directly applied to the real-time 3D human body tracking task of quadruped robots in complex environments.
[0073] This invention first performs feature alignment on the extracted point cloud features and video features to ensure they have the same dimensions, then merges them, and finally outputs the point cloud-video fused feature F. pv It retains a large number of spatial correspondences between the two modes, which can better realize information complementarity under multimodal input.
[0074] This invention designs a feature learning encoder based on a hierarchical attention mechanism and conducts human-oriented network training, which can effectively address the special challenges of the target human body object, such as its susceptibility to deformation and small scale.
[0075] The network model of this invention has dense feedback during the learning process and can be successfully transferred to a quadruped robot platform for physical object tracking.
[0076] This invention can stably track target human objects in many complex scenarios, balancing accuracy and real-time requirements, and has a very good 3D human tracking effect compared with other methods. Attached Figure Description
[0077] Figure 1 This is an overall structural diagram of a 3D human tracking method for quadruped robots in complex environments according to the present invention.
[0078] Figure 2 This is a structural diagram of the feature learning encoder under the hierarchical attention mechanism of the present invention;
[0079] Figure 3 This is a structural diagram of the human body-guided 3D target bounding box proposal and verification decoder of the present invention.
[0080] Figure 4 This is a structural diagram of the anti-bump correction unit of the present invention;
[0081] Figure 5 This is a schematic diagram of the 3D human body tracking results of the present invention. Detailed Implementation
[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0083] As robot control, planning, and obstacle avoidance technologies continue to mature, target tracking technology is gradually becoming an effective solution for robots to achieve higher-level task applications. Currently, however, many bottlenecks remain to be addressed in applications involving the practical implementation of 3D human single-target tracking.
[0084] To address the problems and shortcomings of existing technologies, this invention proposes a 3D human tracking method for quadruped robots in complex environments. It utilizes point cloud and video as complementary dual inputs to mitigate interference from occlusion and lighting changes in complex environments, resulting in high robustness. By incorporating an anti-bumping algorithm, this method achieves high tracking accuracy and speed, and can be directly applied to real-time 3D human tracking tasks for quadruped robots in complex environments.
[0085] like Figure 1 As shown, this invention provides a 3D human tracking method for quadruped robots in complex environments, characterized in that the method includes:
[0086] Obtain the sequence to be tracked; each frame of the sequence to be tracked includes a point cloud frame and a corresponding video frame; specifically, point cloud sequences and video sequences recorded in the same time period and scene are used as the sequence to be tracked; given a dynamic point cloud sequence of T consecutive frames. and the corresponding video sequence
[0087] Template frames and search frames are pre-determined based on the sequence to be tracked; the template frames include point cloud frames of the target human body object to be tracked and corresponding video frames, and the search frames include point cloud frames of the target human body object to be searched and corresponding video frames; specifically, the 3D region of the target human body object is specified in the first frame of the sequence as the pre-determined template frame;
[0088] Specifically, the pre-determined template frame and search frame are input into the trained 3D human tracking model to track the target human object and output the tracking result. Specifically, for the current search frame, the following tracking steps are performed: using the center point of the target box in the template frame as a reference, a region larger than the target box in the current search frame is divided into areas with a preset size and distance. The template frame and the current search frame after dividing the region are input into the trained 3D human tracking model to track the target human object and output the tracking result of the current search frame. The tracking result of the current search frame is used as the template frame for the next search frame. The tracking steps are repeated until all search frames are traversed and the tracking result (the 3D tracking box of the target human object) is output.
[0089] Specifically, the template frame is input into the template branch, where the point cloud labels are used as the initial 3D bounding boxes of the algorithm's target objects, and the input image label 2D bounding boxes are used as the high-heat-attention regions of the image in that frame. The search frame is input into the search branch, where the point cloud labels are used as the 3D bounding boxes of the algorithm's target objects to be searched, and the input image label 2D bounding boxes are used as the potential target objects to be searched in the image in that frame.
[0090] The trained 3D human tracking model includes:
[0091] A multimodal feature extraction and fusion module with a dual-branch architecture; the dual-branch architecture includes a template branch and a search branch; used to input template frames and search frames into the corresponding branches for feature extraction and fusion, to obtain template frame features and search frame features;
[0092] Specifically, the dual-branch architecture multimodal feature extraction and fusion module includes a dual-branch architecture multimodal feature extraction module and a multimodal feature alignment and fusion module based on multi-level interpolation;
[0093] Each branch of the dual-branch architecture multimodal feature extraction module includes a point cloud feature extraction unit and an image feature extraction unit; specifically, it includes a point cloud feature extraction unit of the template branch, an image feature extraction unit of the template branch, a point cloud feature extraction unit of the search branch, and an image feature extraction unit of the search branch.
[0094] The point cloud feature extraction unit is used to extract point cloud features. Specifically, each frame of point cloud is processed by the point cloud feature extraction unit. By reading the point cloud labels, the entire scene point cloud is transformed into a local coordinate system centered on a cuboid with the point cloud origin as the center. The sampling center of the training values in the search branch is obtained by randomly selecting (x, y) within the cuboid. In the template branch, the points are normalized to the x-axis and divided into voxel representations according to the regular spatial resolution. The same 3D range is applied to both branches to obtain input pairs, and features are extracted through the Point-MAE network to obtain the point cloud features.
[0095] The image feature extraction unit is used to extract video features; specifically, each frame of video image is processed by the image feature extraction unit. Through appropriate sharpening and preprocessing operations, it addresses difficult situations such as inaccurate focus and image blurring. Image smoothing and noise compensation are used to smooth the image contours and eliminate misleading interference from obvious anomalies such as salt-and-pepper noise. In the 2D coordinate system, with the top-left corner of the image as the origin, it performs four-way translation (±Δx), rotation (±Δθ), and brightness adjustment (±l). Features are extracted using the MobileFormer network to obtain the video features.
[0096] The multimodal feature alignment and fusion module based on multi-level interpolation is connected to both the point cloud feature extraction unit and the image feature extraction unit, and is used for feature alignment and fusion of point cloud features and video features under each branch. Specifically, the feature alignment and fusion steps of point cloud features and video features include:
[0097] The general farthest point sampling algorithm (FPS) is used to output a total of K point groups and their corresponding point cloud centers. Where K represents the number of point cloud feature groups after point downsampling;
[0098] Using the inverse distance weights w in the original point cloud j Point cloud features Interpolation to each point cloud Point cloud representation after interpolation Right now:
[0099]
[0100] The parameter ∈ is a number that tends towards infinity;
[0101] Based on the lidar-camera mapping relationship, the interpolated point cloud representation p′ is converted into two-dimensional coordinates. And iterate through all points to form new point cloud features, the new point cloud features having the same dimension as the corresponding video features;
[0102] Multilayer perceptron (MLP) is used to extract interaction information, and the features of the two modalities are merged in a new feature channel to obtain point cloud-video features.
[0103] Among them, symbols The elements of each feature channel are added together.
[0104] like Figure 1 As shown, the template frame is input into the template branch for feature extraction and fusion to obtain point cloud-video features under the template branch. The template frame features are used as inputs; the search frame is input into the search branch for feature extraction and fusion to obtain the point cloud-video features under the search branch. For search frame features.
[0105] The point cloud features and video features after feature extraction differ significantly in dimensionality, and there is no unified alignment method between them. An intuitive solution is to artificially add new dimensions by repeatedly applying the point cloud features until they match the video features. However, this method loses a large number of spatial correspondences between the two modalities. To better achieve information complementarity under multimodal input, this invention first performs feature alignment on the point cloud features and video features after feature extraction to make their dimensions consistent, and then merges them, finally outputting the point cloud-video fused feature F. pv .
[0106] Feature learning encoder based on hierarchical attention mechanism;
[0107] Connected to the multimodal feature extraction and fusion module of the dual-branch architecture, it is used to perform feature interaction and enhancement on the template frame features and search frame features to obtain an attention feature map; such as Figure 2 As shown, specifically, long-range interactivity of features is obtained through three self-attention local enhancement units and three mutual attention local enhancement units, enabling the encoder to focus more on target perception and feature aggregation in the global view.
[0108] The self-attention local enhancement unit is used to thoroughly fuse adjacent features of the point cloud and video, and performs a multi-head attention mechanism operation Attn(Q,V,K); that is: input point cloud - video fused feature F pvThe values of Q, V, and K in the formula are calculated using a linear projection layer to compute the self-dot product, normalized using Softmax, and the self-attention feature maps enhanced by the template branch are calculated using a feedforward network and a multilayer perceptron (MLP), respectively. Self-attention feature map enhanced with search branch The calculation formula is as follows:
[0109]
[0110]
[0111]
[0112] Here, Q represents all possible features, K represents features that a person might exhibit, V represents the detailed information contained within different features, the superscript T represents the matrix transpose operation, and d k The set floating-point number;
[0113] The mutual attention local enhancement unit is used to further fuse the data output by the self-attention local enhancement unit. It follows the same structure as the self-attention local enhancement unit, adds residual processing to the output, and uses the last layer of self-attention feature map of the template branch. The values of V and K are taken from the self-attention feature map of the last layer of the search branch. As the value of Q, perform the same multi-head attention mechanism operation and output the enhanced mutual attention feature map. The calculation formula is as follows:
[0114]
[0115] In this embodiment, the self-attention local enhancement unit of the template and search region features uses shared weights to avoid a few feature channels with large amplitudes from dominating.
[0116] Human-oriented 3D target bounding box proposal and verification decoder;
[0117] Connected to the feature learning encoder under the hierarchical attention mechanism, initial tracking results are obtained by performing human-guided target extraction and verification on the attention feature map. Following the common-sense prior assumption that the scale of a human object in three-dimensional space is limited (height approximately 1.0-2.0 meters, width approximately 0.1-0.4 meters, and length approximately 0.2-0.6 meters), the proposed target candidate boxes are filtered based on human-preferred values. This step significantly accelerates network training and updates the more accurate target sequence through multiple iterations.
[0118] Specifically, the initial tracking results obtained by performing human-guided target extraction and verification on the attention feature map include:
[0119] like Figure 3 As shown: the mutual attention feature map of the last layer The human-guided 3D target box proposal and verification decoder is input, and through the latent center generation part and clustering part of the P2B-like network, M candidate boxes are roughly proposed (M takes the value 150).
[0120] A center-based regression head is used to predict several object attributes. The regression head consists of four heads, including a heatmap head. Local offset head Yaw Angle Head and confidence head For each candidate bounding box, 200 points are sampled in the 3D space of the box, and the corresponding predicted value H is calculated based on the heatmap value of each point. x,y,θ,c The average value is then calculated using the following formula:
[0121]
[0122] Where d is the Euclidean distance between the center of the 3D candidate box and the point position calculated by the model, and the predicted value H is... x,y,θ,c =1 corresponds to the point being at the center of the human body object, H x,y,θ,c <0.1 corresponds to the background;
[0123] The multi-head perceptron (MLP) is used to select the candidate with the highest predicted value from the filtered candidate boxes. The candidate bounding boxes are used as the initial tracking results for the i-th search frame. i ={x i ,y i ,z i ,l i ,w i ,h i ,θ i ,c i}; where b i The eight dimensions refer to the x-coordinate, y-coordinate, z-coordinate, length, width, height, yaw angle, and confidence level of the target bounding box, respectively.
[0124] Human-oriented 3D bounding box verification is performed. Based on pre-defined prior values of the target human body, M candidate bounding boxes are filtered using human-oriented methods, retaining only those 3D bounding boxes that conform to the human body size. Specifically, M candidate bounding boxes are filtered according to size range requirements (width: 10-40cm, height: 1.0-2.0m, length: 20-60cm). Boxes that are too large or too small are immediately discarded, thus filtering out a portion of candidate bounding boxes that do not meet the verification criteria. The three-dimensional spatial information of the target human object is then output. in, The values in parentheses refer to the dimensions. Dimensions 1-3 are the x, y, and z coordinates of the target box in 3D space, respectively.
[0125] Anti-bump correction unit;
[0126] Connected to the human-guided 3D target box proposal and verification decoder, the initial tracking results are corrected using an anti-bump correction algorithm based on the robot's recorded frame velocity and its maximum response score.
[0127] The proposed anti-bump correction unit is based on three principles: (i) the target object will be re-discovered near its previously observed spatial location; (ii) the tracker response of the recovered object is similar to that of the previous tracking frame; and (iii) each object generally expands at a speed proportional to the average speed of the object before it was lost. The unit first records the velocity of each frame using the robot's built-in velocity measurement module. Once the tracked object disappears due to severe platform shaking, uneven terrain, sudden overexposure or underexposure of light, or occlusion by other objects, the correction algorithm is immediately activated. By adaptively expanding the search area, it continues to track the approximate location of the object until it reappears, while avoiding computational redundancy in handling irrelevant spatial areas.
[0128] Specifically, based on the robot's recorded velocity in each frame and its maximum response score, the initial tracking results are corrected using an anti-bump correction algorithm, including:
[0129] like Figure 4 As shown: The robot's built-in speed measurement module records the speed of each frame at all times. and its maximum response score For each frame I t Using the three-dimensional spatial information X of the target human body object t Based on the three-dimensional spatial information X of the preceding ten consecutive frames, the velocity value of the target human object in this frame is calculated:
[0130]
[0131] When the maximum response R of a certain frame iWhen the response threshold constant δ is greater than the response threshold constant, the initial tracking result is output as the tracking result of the i-th frame; otherwise, once the tracked object disappears due to severe platform shaking, uneven terrain, sudden overexposure or underexposure of light, or occlusion by other objects, the maximum detected response R is lost. i When the response threshold constant δ is less than the response threshold, the anti-bump correction algorithm is activated and the variable n is reset to 0.
[0132] The anti-bump algorithm is based on the tracking results generated in the previous frame. Adaptively expand the search area:
[0133]
[0134]
[0135]
[0136] in, The values in parentheses refer to the dimensions, with dimensions 4-6 being the length, width, and height of the target box, respectively.
[0137] With X t-1 Point cloud is extracted from the sampling center. And set the variable n = n + 1, and update the tracker's response in the extracted point cloud. Up to maximum response score R t And move to the next frame t+1;
[0138] Repeat the anti-bump algorithm steps until the maximum detected response R is reached. i If the network's pre-defined conditions are met, i.e. the tracking box catches up with the target object again, then the 3D target box of each frame within that time period, corrected by the anti-bumping correction algorithm, is output. This serves as the tracking result for that period of time.
[0139] 3D Human Tracking Model Construction: Under the twin network architecture of point cloud-video dual input, a multimodal feature extraction module with dual branch architecture, a multimodal feature alignment and fusion module based on multi-level interpolation, a feature learning encoder under hierarchical attention mechanism, a 3D target box proposal and verification decoder, and an anti-bump correction unit are constructed to form a 3D human tracking model.
[0140] The training steps for a 3D human tracking model include:
[0141] The multimodal feature extraction module, based on a dual-branch architecture, extracts features from both the input point cloud and video in both the template and search branches. Then, it performs multimodal feature alignment and fusion based on multi-level interpolation. Next, it uses self-attention local enhancement units and mutual attention local enhancement units under a hierarchical attention mechanism to encode and enhance the features of the template frame and search frame. Afterward, it obtains human-guided target extraction and verification, outputting the target classification score map, local offset map, and normalized size map of the tracked target, thus obtaining the target object's bounding box. Finally, it uses an anti-bump correction unit for quadruped robots to overcome the special characteristics of the physical platform, supplementing the output of blank object frames, making the tracking more coherent and robust.
[0142] Specifically, the sequence to be tracked is input into the server for reading. The labeled single-target tracking datasets (KITTI dataset, WAYMO dataset, NuScene dataset) are divided into training and test sets according to the official method. Secondly, when reading each frame of the sequence, the image data is cropped and data augmented, and the point cloud data is normalized. The cropping methods are as follows: A rectangular image is cropped centered on the target area and within the target object's size. The portion of the rectangle extending beyond the original video boundary is filled with the average pixel value and scaled to a resolution of 256x256 or 512x512 to form the template frame image. A similar rectangular image is cropped centered on the target area and within a range four times the target object's size. The portion of the rectangle extending beyond the original video boundary is filled with the average pixel value and scaled to a resolution of 256x256 or 512x512 to form the search frame image. Data augmentation methods include horizontally flipping the image with a probability of p=0.4, converting the image to grayscale with a probability of p=0.025, and jittering the image's center point and size with a probability of p=0.1. Point cloud normalization methods include randomizing the point cloud, leaving 1024 points for the template frame and 4096 points for the search frame.
[0143] For a total of T frames to be tracked, the final training loss function is derived from the ground truth. and network model prediction results The calculation yields the following expression:
[0144]
[0145] The loss function is calculated as follows for the tracking results in frame t:
[0146]
[0147] in, This represents the confidence loss used to identify the foreground. Let λ represent the Smooth-L1 loss used to supervise bounding box regression. cls and λ reg These represent the weights of the confidence loss and the L1 loss, respectively. This represents the ground truth value of the 3D bounding box of the target human object in this frame. The 3D bounding box prediction result of the target human object in this frame consists of eight variables: length l, width w, height h, x-coordinate, y-coordinate, z-coordinate, rotation angle θ, and confidence score. The length, width, and height values are consistent with the initial frame template bounding box values. The coordinates and rotation angle are calculated as follows:
[0148] The 3D human tracking model network was trained on the server using the Adam optimizer, which reduced the overall loss value L of the network loss function. total The network parameters are optimized until convergence. The entire parameter training process is completed end-to-end in a one-stage manner. The learning rate is set to 1×10⁻⁶. -4 The batch size is 256, and a total of 100 training rounds are conducted.
[0149] This invention designs a twin decoder encoder based on an attention mechanism and conducts human-oriented network training, which can effectively address the special challenges of the target human body object, such as easy deformation and small scale.
[0150] This invention also proposes a 3D human tracking system for quadruped robots in complex environments, comprising:
[0151] The LiDAR and camera mounted on the robot are connected to the processor for communication.
[0152] The lidar and camera acquire the sequence to be tracked and send it to the processor;
[0153] The processor contains a trained 3D human tracking network model for executing a 3D human tracking method for quadruped robots in complex environments as described in any one of claims 1-9.
[0154] Using this system to track target human objects has the advantages of high tracking accuracy and fast tracking speed.
[0155] This invention provides a 3D human tracking method for quadruped robots in complex environments. It utilizes point cloud and video as complementary dual inputs to mitigate interference from target occlusion and lighting variations in complex environments, further improving the architecture's robustness in such conditions. An attention-based twin decoder encoder is designed, and human-guided network training is conducted to effectively address the unique challenges of human objects, such as their susceptibility to deformation and small scale. Furthermore, an anti-bump correction algorithm is designed in the target bounding box extraction and verification module, making the feedback during the existing network model's learning process denser, enabling successful transfer to quadruped robot platform tracking. This invention can stably track target human objects in numerous complex scenarios, balancing accuracy and real-time performance requirements.
[0156] With dual inputs of image and point cloud, the tracking results output by this invention are closer to the true value, achieving a tracking success rate of 73.3% and a tracking accuracy of 88.9%. It can better track human point clouds and significantly improves accuracy by about 10% compared with the latest methods in the fields of P2B, STNet, PTTR, etc., achieving better 3D human tracking results. Figure 5 The tracking results of this invention and the PTTR method are shown, which can more intuitively see that the 3D tracking box of this patent is closer to the true value and the output tracking results are more accurate.
[0157] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.
Claims
1. A 3D human tracking method for quadruped robots in complex environments, characterized in that, The method includes: Acquire the sequence to be tracked; each frame of the sequence to be tracked includes a point cloud frame and the corresponding video frame; Template frames and search frames are pre-determined based on the sequence to be tracked; the template frames include point cloud frames of the target human body object to be tracked and corresponding video frames, and the search frames include point cloud frames of the target human body object to be searched and corresponding video frames. The pre-determined template frames and search frames are input into the trained 3D human body tracking model to track the target human body object frame by frame and output the tracking results. The trained 3D human tracking model includes: The dual-branch architecture multimodal feature extraction and fusion module is used to input the template frame and the search frame into the corresponding branches for feature extraction and fusion, so as to obtain the template frame features and the search frame features. The feature learning encoder under the hierarchical attention mechanism is connected to the multimodal feature extraction and fusion module of the dual-branch architecture, and is used to perform feature interaction and enhancement on the template frame features and search frame features to obtain an attention feature map; A human-guided 3D target bounding box proposal and verification decoder is connected to the feature learning encoder under the hierarchical attention mechanism. The initial tracking result is obtained by extracting and verifying the human-guided target from the attention feature map. The anti-bump correction unit is connected to the human-guided 3D target box proposal and verification decoder. Based on the robot's recorded frame velocity and its maximum response score, the initial tracking results are corrected using the anti-bump correction algorithm. Based on the robot's recorded velocity and maximum response score for each frame, the initial tracking results are corrected using an anti-bump correction algorithm, including: The robot's built-in speed measurement module records the speed of each frame in real time. and its maximum response score For each frame Through the three-dimensional spatial information of the target human body object And the three-dimensional spatial information of the preceding ten consecutive frames. Calculate the velocity value of the target human object in this frame: ； The maximum response score of a certain frame When the network's pre-defined conditions are met, the initial tracking result is output as the first... If the frame tracking results are not found, then the anti-bump correction algorithm is activated and the variables are reset. =0; The anti-bump correction algorithm is as follows: based on the tracking results generated in the previous frame... Adaptively expand the search area: ； ； ； in, The values in parentheses refer to the dimensions; dimensions 4-6 are the length, width, and height of the target bounding box, respectively. Point cloud is extracted from the sampling center. And set variables = +1, update the tracker's response in the provided point cloud. Up to maximum response score And move to the next frame t+1; Repeat the anti-rolling correction algorithm steps until the maximum response detected If the network pre-set conditions are met, i.e. the tracking frame re-tracks the target object, output each frame of the 3D target frame after correction by the anti-rolling correction algorithm in this period of time As the tracking result in this period of time.
2. The 3D human tracking method for quadruped robots in complex environments according to claim 1, characterized in that: The dual-branch architecture multimodal feature extraction and fusion module includes a dual-branch architecture multimodal feature extraction module and a multimodal feature alignment and fusion module based on multi-level interpolation; Each branch of the dual-branch architecture multimodal feature extraction module includes a point cloud feature extraction unit and an image feature extraction unit; the point cloud feature extraction unit is used to extract point cloud features; the image feature extraction unit is used to extract video features; The multimodal feature alignment and fusion module based on multi-level interpolation is connected to the point cloud feature extraction unit and the image feature extraction unit respectively, and is used for feature alignment and fusion of point cloud features and video features under each branch.
3. The 3D human tracking method for quadruped robots in complex environments according to claim 2, characterized in that: The point cloud feature extraction unit extracts point cloud features through a Point-MAE network The image feature extraction unit extracts video features through a MobileFormer network Point cloud features and video features The feature alignment and fusion steps include: adopting the general farthest point sampling algorithm to output a total of K point groups and corresponding point cloud centers ; wherein K represents the number of point cloud feature groups after point downsampling Using inverse distance weighting in raw point clouds Interpolating point cloud features to each point cloud , generating an interpolated point cloud representation i.e.: ； Parameter A number that tends to infinity; transforming the interpolated point cloud representation to two-dimensional coordinates and traversing all points to form a new point cloud feature having the same dimension as the corresponding video feature; The multi-layer perception MLP is used to extract interaction information, and two modal features are combined on a new feature channel to obtain point cloud-video features .
4. The 3D human body tracking method for quadruped robots in complex environments according to claim 1, characterized in that, The step of inputting pre-determined template frames and search frames into the trained 3D human tracking model to track the target human object and output the tracking results includes: For the current search frame, perform the following tracking steps: using the center point of the target box in the template frame as a reference, divide the current search frame into a region with a preset size and distance. Input the template frame and the current search frame after dividing the region into the trained 3D human tracking model to track the target human object and output the tracking result of the current search frame. Use the tracking result of the current search frame as the template frame for the next search frame, and repeat the above tracking steps until all search frames have been traversed and the tracking result is output.
5. A 3D human tracking method for quadruped robots in complex environments according to claim 1, characterized in that: The dual-branch architecture includes a template branch and a search branch; the template frame is input into the template branch for feature extraction and fusion to obtain point cloud-video features under the template branch, which are the template frame features; the search frame is input into the search branch for feature extraction and fusion to obtain point cloud-video features under the search branch, which are the search frame features.
6. The 3D human body tracking method for quadruped robots in complex environments according to claim 5, characterized in that, The feature learning encoder under the hierarchical attention mechanism includes a self-attention local enhancement unit and a mutual attention local enhancement unit; The self-attention local enhancement unit is used for thoroughly fusing adjacent features of the point cloud and the video, and performing a multi-head attention mechanism operation ; input the point cloud-video features under the template branch and the point cloud-video features under the search branch into the self-attention local enhancement unit to calculate the enhanced self-attention feature map of the template branch and the enhanced self-attention feature map of the search branch ; the calculation formula is as follows: ； ； ； where Q refers to various possible features, K refers to features that a person will have, V refers to detailed information contained within different features, and the superscript T refers to a matrix transpose operation, is a set float number; The mutual attention local enhancement unit is used for further fusing data output by the self-attention local enhancement unit, and the mutual attention local enhancement unit follows the same structure, additionally adds residual processing to the output, and takes the last layer of self-attention feature maps of the template branch as The value, taking the last layer of self-attention feature maps of the search branch as The value, performing a multi-head attention mechanism operation, and outputting enhanced mutual attention feature maps.
7. The 3D human body tracking method for quadruped robots in complex environments according to claim 6, characterized in that, in, The initial tracking results obtained by performing human-guided target extraction and verification on the attention feature map include: The mutual attention feature map of the last layer is input into the human-guided 3D target box proposal and verification decoder, and M candidate boxes are generated through the latent center generation part and clustering part of the P2B-like network. A center-based regression head is used to predict several object attributes. For each candidate bounding box, 200 points are sampled in the box's 3D space, and the predicted value is calculated based on the heatmap value of each point. The average value is then calculated using the following formula: ； Where d is the Euclidean distance between the center of the 3D candidate box and the point position calculated by the model, and the predicted value is... = 1 corresponds to the point being at the center of the human body object. < 0.1 corresponds to the background; The multi-head perceptron (MLP) is used to select the candidate with the highest predicted value from the filtered candidate boxes. The candidate box, as the first Initial tracking results of frame search frames ; Human-oriented 3D bounding box verification is performed. Based on pre-defined prior values of the target human body, M candidate bounding boxes are filtered using human-oriented methods, retaining only those 3D bounding boxes that conform to the human body size. Output the current three-dimensional spatial information of the target human body object. ; in, The values in parentheses refer to the dimensions. Dimensions 1-3 are the x, y, and z coordinates of the target box in 3D space, respectively.
8. A 3D human tracking method for quadruped robots in complex environments according to claim 1, characterized in that, The training steps for the 3D human tracking model include: The sequence to be tracked is input into the server for reading and divided into training and test sets; When reading each frame of the sequence to be tracked, a template frame and a search frame are pre-determined. The point cloud data in the template frame and the search frame are normalized, and the image data is cropped and augmented. For a total of T frames to be tracked, the final training loss function is derived from the ground truth. and network model prediction results The calculation yields the following expression: ； The loss function is calculated as follows for the tracking results in frame t: = ； in, This represents the confidence loss used to identify the foreground. This represents the Smooth-L1 loss used to supervise bounding box regression. and These represent the weights of the confidence loss and the L1 loss, respectively. This represents the ground truth value of the 3D bounding box of the target human object in this frame. This represents the 3D bounding box prediction result of the target human object in this frame, consisting of length l and width l. Height h, x-coordinate, y-coordinate Coordinates, rotation angle The confidence level consists of eight variables, with the length, width, and height values consistent with the initial frame template frame values. The coordinates and rotation angles are calculated as follows: , , , = ; The 3D human tracking model network was trained on the server using the Adam optimizer to reduce the overall loss value of the network loss function. Optimize the network parameters until convergence.
9. A 3D human tracking system for quadruped robots in complex environments, characterized in that, include: The LiDAR and camera mounted on the robot are connected to the processor for communication. The lidar and camera acquire the sequence to be tracked and send it to the processor; The processor contains a trained 3D human tracking network model for executing a 3D human tracking method for quadruped robots in complex environments as described in any one of claims 1-8.