Robot personnel following method based on multi-target tracking

Abstract

The invention provides a mobile robot figure tracking comprehensive framework based on multi-target tracking. Firstly, the framework combines an improved OC-SORT tracking algorithm with a gesture detection technology, and realizes dynamic selection, accurate recognition and position estimation of a plurality of objects. Secondly, accurate path tracking of the omni-directional wheeled mobile robot can be achieved by planning a Dubins path and providing transverse and longitudinal control models based on the omni-directional wheeled mobile robot. According to the improved OC-SORT tracking algorithm, evaluation is carried out on public data sets such as DanceTrack and Person Tracking, the tracking precision and the positioning precision reach 60.5 HOTA and 98.1% respectively, and the tracking precision and the positioning precision exceed existing multi-target tracking OC-SORT and single-target tracking ODFS. The robot personnel tracking framework provided by the invention can be operated on an embedded system equipped with NVIDIA TM # imgabs0 # Jetson Xavier NX. The performance of the system on customized personnel tracking data sets and the successful application of the system in real-world scenes prove the effectiveness of the system in real-time tracking and following of individuals.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-target tracking for complex scene security management and control and human-machine interaction, and in particular, relates to a robot personnel following method based on multi-target tracking. Background Art

[0002] There is a growing demand for service robots that can follow humans around to assist with daily tasks such as guiding, guarding, and elderly care. When robots perform collaborative tasks, they must robustly track the location of the target, which can be problematic when objects occlude or leave the camera's field of view (FOV). To distinguish between visually tracked targets, Multi-Object Tracking (MOT) is essential, which distinguishes between different individuals and retains the position information of the tracked targets so that they can be re-matched.

[0003] Existing tracking algorithms are all based on the OAB tracking framework, where the tracking target is selected in the first frame, and the subsequent frames are based on the extracted target features to distinguish the tracking targets. This framework is not suitable for more complex pedestrian tracking scenarios. Our work proposes a novel framework that is different from these traditional frameworks and presents a robotic human tracking system within a multi-object tracking framework. The system uses target localization technology to accurately identify multiple tracking objects and dynamically switch the tracking ID between multiple tracking objects. Therefore, multi-target tracking is utilized in the method to perceive the robotic visual tracking and effectively manage multiple targets through association algorithms and state estimation.

[0004] Existing multi-target tracking algorithms face challenges in complex scenes where objects frequently interact and occlude each other. In this case, the multiple targets detected by the detector and the multiple state positions predicted by the tracking have a high IoU overlap. This makes it difficult to correctly match targets using IoU alone as the association cost matrix. To address this problem, some algorithms integrate appearance features into the association cost. However, the features of occluded targets are usually unreliable, which may weaken the effectiveness of these methods. Therefore, the present invention designs a robot personnel following method based on multi-target tracking. Summary of the invention

[0005] To achieve the above objectives, the method of the present invention introduces a robot human tracking framework based on a multi-target tracking framework. The framework initially uses an RGB-D camera to capture color images and depth information simultaneously and in parallel. The color images are processed by target detection and multi-target tracking algorithms to achieve real-time visual tracking of the targets. In order to solve the problem of frequent occlusion of objects in densely populated scenes, we designed a deep cost matrix to enhance the tracking performance. In addition, we designed two target positioning methods to identify and dynamically switch the follow-up ID in multi-target tracking. Subsequently, the paths of the robot and the tracked targets are planned in the spatial coordinate system based on the positioning data. Finally, the lateral and longitudinal control models of the omnidirectional wheel robot are proposed to achieve effective real-time tracking of dynamic targets.

[0006] The technical solution adopted by the present invention comprises the following steps:

[0007] S1 Image acquisition: The robot acquires each frame of color image and depth image through the Astro Pro camera. This step ensures high-precision input data in subsequent processing, where the depth image provides key three-dimensional information for subsequent positioning and path planning.

[0008] S2 target detection: The color image obtained by S1 is input into the YOLOX model for target detection. The YOLOX model is an efficient real-time target detection algorithm that can quickly identify targets in complex scenes. The output video frame image includes the regional coordinates of all detected targets and the corresponding confidence, thus providing basic data for multi-target tracking.

[0009] S3 Multi-target Tracking: Input the detection box set output by S2 into the multi-target tracking method to track multiple targets. The multi-target tracking algorithm can not only track the position of each target, but also assign a unique ID to each target to ensure the consistency of the target between different frames. The output includes the ID corresponding to each target and the positioning box of each target, ensuring that the robot can correctly identify and distinguish the target to be followed.

[0010] S4 Target Positioning: All tracking targets acquired in S3 are dynamically positioned according to the predetermined positioning strategy. The output includes the position information of a single follow target to determine the target that the robot is currently following.

[0011] S5 path planning: First, the position information of the target relative to the image obtained by S4 is combined with the depth image to obtain the distance of the robot relative to the target and convert it into position information in the world coordinate system. Then, the Dubins curve is used to plan the path trajectory between the robot and the tracking target. The Dubins curve is a path planning method that takes into account the minimum turning radius and is suitable for path planning of robots in a limited motion space.

[0012] S6 motion control: Based on the tracking trajectory obtained by S5, the trajectory is input into the lateral and longitudinal control algorithm. The control algorithm will dynamically adjust the robot's speed and direction according to the change of the target position to ensure a smooth and stable tracking process. Finally, the robot's three-axis motion speed is output, including forward speed, turning speed and vertical speed, to achieve accurate target tracking.

[0013] S7 ends the tracking of the current frame and enters the next frame: After completing step S6, the robot will continue to process the image data of the next frame and repeat steps S2 to S6. Through such an iterative cycle, the robot can continuously track dynamic targets and achieve real-time following of the target.

[0014] Furthermore, in step S2, in order to deploy the detection model on the Jetson Xavier NX embedded platform, the YOLOX-NANO lightweight model is used, which uses COCO pre-trained weights and finally outputs the detection set Z of the current frame t. t =[u,v,s,r], where u,v,s and r represent the position coordinates, area and aspect ratio of the center point of the detection box, respectively. In order to improve the performance of the model in clustered crowd scenes, we selected CrowdHuman, Cityperson and ETHZ datasets with different characteristics. The CrowdHuman dataset is well-known for its pedestrian images in high-density crowd scenes, which enables the algorithm to better identify and track individuals in crowded environments. The Cityperson dataset contains a variety of urban street scenes, which enhances the model's application capabilities in busy traffic backgrounds. The ETHZ dataset provides pedestrian images from multiple angles and different time periods, which deepens the model's understanding of the dynamic changes of pedestrians. The integration of these datasets aims to improve the generalization ability of the YOLOX-NANO model in practical applications after using COCO pre-trained weights, ensuring its accuracy and robustness in real-world scenarios.

[0015] Furthermore, in step S3, the motion prediction model for multi-target tracking is based on a Kalman filter (KF), which updates the state of the tracked object. The state vector is defined as Represents the position (u, v), area (s), aspect ratio (r) and their respective velocity components. In the same frame, the position z detected by the YOLOX detector t = [u,v,s,r] is used as the observation vector.

[0016] Assuming the object's motion follows a uniform velocity pattern, the following Kalman prediction equation can be used to estimate the state vector at frame t:

[0017]

[0018] Here, F t represents the state transition matrix, W t is the process noise, P t|t-1 is the prediction error covariance, Q t is the process noise covariance matrix. Subsequently, the YOLOX detector corrects the prior estimate according to the observation result of the tracking target z t at frame t and updates the state vector

[0019]

[0020] Here, H t is the observation matrix, R t is the observation noise covariance matrix, K t is the Kalman gain matrix. When the tracked trajectory is occluded and there is no observation update z t the state error may accumulate over time, causing the KF parameters to deviate from the correct path and increasing the prediction error. To reduce this cumulative error, when the object is re-detected after being occluded, we backtrack its state during occlusion, create a virtual trajectory, and update the KF parameters. Assume that the last observation value when the object is occluded is and the first observation value when the object is re-detected is Then the virtual observation state z t between t1 and t2 can be described as:

[0021]

[0022] Here, along the virtual trajectory z t (t1 < t < t2), it is used to recalculate the prediction and update steps of the Kalman filter.

[0023] In the data association part, at frame t, the YOLOX detector generates a set of detection boxes while the Kalman filter generates a set of predicted tracking boxes For any given detection box and prediction box an IoU value can be calculated as the similarity measure in the cost matrix C IoU as follows:

[0024]

[0025] The IoU cost matrix is ​​obtained by calculating the ratio of the intersection area of ​​two boxes to the union area. This method has advantages due to its simplicity and effectiveness, especially when measuring the overlap between boxes. However, IoU may not be effective in cases involving overlap or occlusion, especially when multiple objects are closely arranged along the same line, because it cannot accurately distinguish the actual spatial relationship between individual objects. To solve this problem, this section introduces the deep cost matrix C based on the IoU cost matrix. DCM :

[0026]

[0027] Function F obtains the vertical coordinate of the bottom edge of the box, so as to better compare the relative depths of different objects on the same ground plane. In the case of occlusion or object aggregation, the depth information of the bottom edge can be used as a more stable feature, because the bottom edge is usually still visible when partially occluded, and the depth value before and after occlusion remains consistent.

[0028] Then, the IoU cost matrix C IoU With C DCM Combined, we can get the weighted data association cost matrix C d :

[0029] C d =C IoU (i,j)+ω DCM ·C DCM (i,j)

[0030] Here, ω DCM It is C DCM Matrix coefficients, the Hungarian algorithm is obtained by d The goal is to minimize:

[0031]

[0032] Indicates the trajectory set from the current frame To the detection box Z t The set of all permutations and combinations σ that are perfectly matched. Output result They correspond to the matched activated tracks, unmatched tracks, and unmatched high-score boxes. Use the Kalman filter to update the trajectory parameters and output them as results. Keep it. If there are a certain number of unmatched ones, delete them. Unmatched detection boxes Used to create a new track and assign an ID number for matching in the next frame.

[0033] Further, in step S4, target positioning strategy, in traditional single target tracking, a tracking ID is usually manually selected in the initial frame. In contrast, the target selection process of robot personnel tracking using multi-target tracking technology is more flexible. In simple tracking tasks, the multi-target tracking system can automatically identify and select the tracking ID located at the center of the initial frame image as the main tracking target. This selection process is implemented by the following formula:

[0034]

[0035] Here, z0 represents the detection result in the first frame, and is the horizontal coordinate of the left and right borders of the detection box, and w is the width of the image. The center of the box (x c ,y c )The individual closest to the center of the image It will be identified as the tracking target. Using the depth value d of the RGB-D camera, the three-dimensional spatial coordinates of the tracking target relative to the robot can be calculated as follows:

[0036]

[0037] Where φ represents the horizontal field of view of the camera. In addition, this section also uses MobileNet for gesture detection. The network has been pre-trained on the HaGRID dataset to detect specific gestures. The specific process is as follows: After the robot is started, the robot enters the stop state, waits for 1 second and then transitions to the search state. When a person is detected, the α event is triggered, prompting the robot to enter the candidate state for pedestrian selection. If the pedestrian makes a β gesture, the robot will enter the follow state to determine the follow ID and transmit the position information. If there is no η gesture in the current frame, the β event will be triggered, and the robot will return to the search state to determine the position of the next frame. In the follow state, if the pedestrian makes a τ gesture, or the follow target leaves the camera's field of view for multiple frames, the τ or γ event will be triggered respectively, and the robot will return to the search state to find the position of the next frame.

[0038] Furthermore, in step S5, the path planning solution process: after receiving the position information (x, y, θ) of the tracking target after step S4 (with its own coordinate system as a reference), it is responsible for generating discrete trajectories. Considering the limitations of the camera field of view and the non-global shortest path problem, this section uses Dubins for path planning. The Dubins curve provides six basic path type combinations (LSL, LSR, RSL, RSR, LRL, RLR). Before discretizing the planned curve, it is necessary to determine the shortest path combination. For ease of calculation, coordinate transformation is required. The starting point of the robot is set to s(0,0,φ) and the end point of the tracking target is set to g(d,0,ψ). The specific transformation formula is as follows:

[0039]

[0040] Here φ, d, ψ, and R represent the robot's heading angle, the normalized distance from the robot to the target, the target's heading angle, and the robot's turning radius, respectively. For example, in the LSL path type, the subpath length is calculated as follows:

[0041]

[0042] The length of the LSL path combination can be determined by calculating the starting rotation angle t, the path length p, and the final rotation angle q. Then the shortest path is selected among the six basic combinations for discretization. Discretization is done by interpolation, which decomposes the continuous Dubins path into a series of discrete points. Given a discrete step length l and a starting interpolation state (x o ,y o ,θ o ), the interpolation formula for each path segment is:

[0043]

[0044] in is the turning radius, and the path length is normalized during the calculation process. Iterative interpolation generates a complete set of discrete path points until the planned path length is reached. Finally, this set of points is passed to the path tracking module for further processing.

[0045] Further, in step S6, the robot motion step solution process: The robot motion control is designed based on the omnidirectional wheeled robot model, in which the lateral and longitudinal control models are designed to compensate for the error in the planned tracking trajectory. The lateral controller uses the LQR linear controller to calculate the robot's angular velocity ω and lateral velocity v according to the deviation between the robot's current position and the planned path. y In contrast, the longitudinal controller uses a PID controller to calculate the robot's longitudinal velocity v according to the distance between the robot and the tracking target. x .

[0046] The robot motion control is designed based on an omnidirectional wheeled robot model, where the lateral and longitudinal control models are designed to compensate for the error in the planned tracking trajectory. The lateral controller uses an LQR linear controller to calculate the robot's angular velocity ω and lateral velocity v according to the deviation between the robot's current position and the planned path. y In contrast, the longitudinal controller uses a PID controller to calculate the robot's longitudinal velocity v according to the distance between the robot and the tracking target. x .

[0047] The lateral control system uses an LQR linear controller to adjust the lateral error d of the robot relative to the trajectory. p and heading error θ p These deviations are determined by comparing the actual position of the robot with its projection on the intended path. The lateral control error is defined as The control input is u=[v y ,ω] T .

[0048] First, a mathematical representation of the difference between the actual position of the robot and its projection point on the planned path is established. This model relies on the geometric relationship between the earth coordinate system and the planned trajectory. It is not difficult to see from the figure that the heading error e θ and its rate of change

[0049] e θ =θ-θ r

[0050]

[0051] Then the lateral error e is defined d As the robot's current position vector Matching point position vector with planned trajectory In the normal direction Projection difference on :

[0052]

[0053] Then, by changing the robot's position vector The position change vector of the matching point with the planned trajectory The difference in projection in the direction, and Rate of change over time The influence of lateral error change rate is derived

[0054]

[0055] Here, the robot's position change vector Expressed as the velocity component in the body coordinate system and The linear combination of the planned trajectory matching point position change vector The velocity along the trajectory is taken into account

[0056]

[0057] Consider the curvature k of the planning curve with respect to the normal vector The effect of the rate of change, according to Frenet's formula have

[0058]

[0059] Substitute the formula, expand and simplify to get:

[0060]

[0061] Assume θ-θ r is the minimum value, then the lateral error change rate for:

[0062]

[0063] Then according to the linear state space model There is lateral control error The lateral control problem is framed and the state transfer matrix is ​​introduced With control matrix And discretize it to form the lateral control error e at the next moment t+1 The model is expressed by the following matrix equation:

[0064]

[0065] The lateral control error The horizontal control variable u=[a y ,α] T , then we have

[0066]

[0067] Therefore, the state transition matrix and control matrix It can be determined by geometric relations. By defining the cost function J and minimizing the sum of the weighted long-term cumulative error and the control effort, we propose a discrete LQR problem:

[0068]

[0069] The weight matrices Q and R are introduced and given the dynamic constraints Next, by applying the Lagrange multiplier method, the Riccati equation is finally derived:

[0070]

[0071] By continuously iterating to solve the P matrix, the feedback gain matrix K is then solved:

[0072]

[0073] Finally, the control matrix u is solved k :

[0074] u t =-Ke t

[0075] Throughout the process, the lateral control system is simulated as a linear system that adjusts the control input u to reduce the error e.

[0076] The longitudinal control system uses a PID controller to maintain a predetermined distance D between the robot and the tracking target. The longitudinal velocity v of the robot x It is proportional to the error between the current tracking target depth and the set distance. The relationship between the longitudinal velocity and the error term is shown in the following formula:

[0077] v x =K p *(dD)+K i *∫(dD)dt

[0078] Among them, K p and K i To control the gain, dD is the error between the current depth of the target and the predetermined distance, and dt is the time interval between consecutive frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a flow chart of the robot personnel following method based on multi-target tracking disclosed in the present invention.

[0080] Figure 2 This is a state switching diagram of the gesture-based target positioning method disclosed by the present invention.

[0081] Figure 3 It is a schematic diagram of the lateral and longitudinal control model of the omnidirectional wheel according to the method disclosed in the present invention.

[0082] Figure 4 A physical picture of the robot hardware built for the experiment of the method disclosed in the present invention.

[0083] Figure 5 This is a visual comparison diagram of the method disclosed in the present invention before and after the improvement of the multi-target tracking method.

[0084] Figure 6 This is a comparison chart of the target positioning accuracy of the method disclosed in the present invention based on multi-target tracking on the Person Tracking dataset.

[0085] Figure 7 This is a comparison chart of the positioning accuracy of the method disclosed in the present invention under different sequences of the Person Tracking dataset.

[0086] Figure 8This is a diagram showing the effect of the following path planning and tracking accuracy of a specified dynamic target under the Person Following data set using the method disclosed in the present invention.

[0087] Fig. 9 This is a diagram of robot-human tracking results of a mobile robot using a multi-target tracking algorithm in an outdoor environment using the method disclosed by the present invention. DETAILED DESCRIPTION

[0088] The present invention will be further described below in conjunction with the accompanying drawings.

[0089] refer to Figure 1-9 ,A robot tracking method based on multi-target tracking mainly includes five main modules: detection module, multi-target tracking module, target positioning module, path planning module and motion control module, and according to the above five modules, the following steps are adopted:

[0090] S1 Image Acquisition: The robot acquires each frame of color image and depth image through the Astro Pro camera to ensure high-precision input data for subsequent processing. The depth image provides key three-dimensional information for subsequent positioning and path planning.

[0091] S2: Target detection: input the color image obtained in step S1 into the YOLOX model for target detection. The YOLOX model is an efficient real-time target detection algorithm that can quickly identify targets in complex scenes. The output video frame image includes the regional coordinates of all detected targets and the corresponding confidence levels, providing basic data for multi-target tracking.

[0092] S3: Multi-target tracking: Input the detection frame set output by the target detection in step S2 into the multi-target tracking algorithm to track multiple targets. The multi-target tracking algorithm can not only track the position of each target, but also assign a unique ID to each target to ensure the consistency of the target between different frames. The output includes the ID corresponding to each target and the positioning frame of each target, ensuring that the robot can correctly identify and distinguish the target to be followed;

[0093] S4: Target positioning: all tracking targets acquired in S3 are dynamically positioned according to a predetermined positioning strategy, and the position information of a single follow target is output to determine the target that the robot is currently following;

[0094] S5: Path planning. First, the position information of the target relative to the image obtained by the target positioning clock in step S4 is combined with the depth image to obtain the distance of the robot relative to the target and convert it into position information in the world coordinate system. Then, the path trajectory between the robot and the tracking target is planned using the Dubins curve.

[0095] S6: motion control, according to the tracking trajectory obtained by the path planning in step S5, the trajectory is input into the lateral and longitudinal control algorithm. The control algorithm will dynamically adjust the speed and direction of the robot according to the change of the target position to ensure a smooth and stable tracking process, and finally output the three-axis motion speed of the robot, including the forward speed, the turning speed and the vertical speed, to achieve accurate target tracking;

[0096] S7: End the tracking of the current frame and enter the next frame. After completing the motion control of step S6, the robot continues to process the image data of the next frame and repeats steps S2 to S6. Through the iterative cycle, the robot can continuously track the dynamic target and achieve real-time following of the target.

[0097] like Figure 1 As shown, in the framework of this method, represents the trajectory set at frame t-1, The Kalman filter predicted state t and Z representing the existing trajectory at frame t t Corresponding to the state detected at frame t, the dashed line represents the robot's planned path to follow the goal.

[0098] like Figure 2 The state transition diagram of gesture-based target localization mainly includes four states: stop, search, candidate and follow, and five state transition events: α, β, γ, η (follow gesture) and τ (stop gesture).

[0099] like Figure 3 Schematic diagram of the omnidirectional wheel lateral and longitudinal control model, mathematically represented in the global coordinate system, considering the planned path indicated by the dotted line. p and heading deviation θ p As the input parameter of the model, the output velocity v along the y-axis y and the angular velocity ω about the z-axis.

[0100] like Figure 4 A physical picture of a mobile robot equipped with Jetson Xavier NX. In order to increase the shooting height, the Astro Pro camera module is equipped with an elevated mounting bracket. It is installed on the robot's omnidirectional wheel chassis together with the Jetson Xavier NX board for easy wiring.

[0101] In order to accurately evaluate the performance of RPF systems in terms of individual position recognition and real-time tracking in video frames, this paper conducts a comprehensive evaluation on two public datasets and one custom dataset. First, the DanceTrack dataset\cite{sun2022dancetrack} is used to evaluate the multi-target tracking algorithm. This dataset mainly consists of collective dance videos, with 40 training sets, 25 validation sets, and 35 2D video sequence test sets. This dataset is known for its complex human motion patterns, similar appearance features, frequent human interactions, and occlusions, which poses a significant challenge to motion analysis capabilities while deliberately reducing the reliance on appearance features. Secondly, the pedestrian tracking dataset\cite{chen2017integrating} contains indoor and outdoor individual pedestrian tracking video sequences captured by stereo cameras bumblebee2 and ZED, providing 9 indoor and 2 outdoor stereo video sequences, including left, right, and depth images. The challenges faced by this dataset include changes in human poses, changes in lighting, changes in appearance, and human occlusion. This paper uses it to evaluate the robot's human tracking algorithm. In addition, unlike the two public datasets mentioned above, the “People Tracking” dataset was developed specifically for this paper to evaluate the robot’s ability to track individuals. This custom dataset is designed to test the responsiveness and reliability of the robot’s lateral and longitudinal control algorithms in various scenarios, including dynamic heading angle and velocity changes of the target individual. It is updated at a frequency of 5Hz and includes global coordinates based on heading angle and velocity updates, which is critical for algorithm implementation. The dataset includes two regular paths (rectangular and circular) and two irregular paths (alternating between static and moving, and a continuous moving path), where the irregular paths show that the deviation of the heading angle of the person’s movement is within 10 degrees, which closely mimics the movement of the real world. Finally, in order to verify the effectiveness of the robot follower based on multi-target tracking in real applications, a series of robot following experiments were conducted in outdoor and corridor scenarios. These experiments are designed to test the robot’s ability to follow the target when it is repeatedly occluded and disappears and then reappears. In addition, by integrating gesture detection, we further evaluate the robot’s following performance in complex human-robot interaction scenarios.

[0102] For the DanceTrack dataset, the experiment uses high-order tracking accuracy HOTA as the main indicator, integrating multiple sub-indicators, from association accuracy AssA and detection accuracy DetA. In addition, in order to more accurately evaluate the association performance between objects, the experiment places special emphasis on the use of IDF1. Classic verification indicators include multi-object tracking accuracy MOTA and IDF1 score, which are also used to provide a comprehensive evaluation of the performance of the algorithm on the verification set and test set.

[0103] We use the person tracking dataset to evaluate the accuracy P of target localization. The evaluation criterion is based on the tracking target center (xi ,y i ) and the ground truth center (x gt ,y gt ) between them. If the distance is equal to or less than the positioning error threshold T θ =50, the target tracking is considered successful. The evaluation formula is as follows:

[0104]

[0105] Here, the indicator function The position error threshold T used to determine whether the target is in frame i θ = 50. The variable n represents the total number of frames in the evaluation sequence.

[0106] For the PersonFollowing dataset, the task of the robot tracking algorithm is to maintain a specific tracking distance d o and the angle θ relative to the target motion direction o This paper uses d o and θ o As the evaluation index, d in the simulated environment o is calculated using the Euclidean distance in the global coordinate system, while in the real environment it is obtained in real time using the depth measured by the RGB-D camera. Figure 3 In addition, the lateral deviation d is introduced p and heading deviation θ p As evaluation indicators, the performance of lateral and longitudinal control models under the planned path is evaluated.

[0107] The evaluation experiments on the DanceTrack and PersonTracking public datasets were conducted on a desktop computer equipped with an Intel Core i5-12490F processor (4.60GHz) and an NVIDIA GeForce RTX 3060Ti GPU (8GB GDDR6X). Pedestrian detection in these experiments is based on the YOLOX model, implemented in PyTorch. The deep cost matrix ω d The weight of is set to 0.4, the number of matches of the depth cost matrix v is set to 3, and the IoU association threshold is set to 0.3. The test of the PersonFollowing dataset is conducted using an omnidirectional wheel robot platform equipped with JetsonXavierNX. The robot is also tested in practice, such as Figure 4As shown. The size of the robot is 0.4m×0.4m×1.2m and it is equipped with an AstroPro model RGB-D camera for environmental data collection and personnel tracking. By using the same parameters for both scenarios, the consistency between personnel tracking and actual tests can be maintained. In the lateral controller, the iterative convergence threshold ∈ is set to 0.01, the cost matrix Q is a diagonal matrix with elements [1,0,1,0], and R is a diagonal matrix with elements [0.01, 0.2]. In the longitudinal controller, the proportional coefficient K p Set to 0.7, the integral coefficient K i Set to 0.01.

[0108] In the DanceTrack dataset, we compare our proposed multi-object tracking algorithm with various existing motion model-based algorithms, ensuring a fair comparison by using the same YOLOX detector in all evaluation methods. The detailed results are shown in Table 1. This shows that our algorithm obtains 60.5 points in HOTA, 63.0 points in IDF1, and 45.9 points in AssA, which is nearly 6% higher than OC-SORT. This enhancement is mainly attributed to the introduction of the deep cost matrix, which combines the depth information to reduce IoU conflicts and improve the track association accuracy.

[0109] Table 1

[0110]

[0111] To demonstrate the tracking performance intuitively, we selected two sequences from the DanceTrack training and validation sets: DanceTrack0097 and DanceTrack0039. Figure 5 As shown. In sequence DanceTrack0097, #2 is occluded by #3. Under this occlusion, the detector cannot recognize the occluded #3, resulting in a high IoU overlap between #2 and #3. This situation usually leads to the track switching problem, where a single detection is incorrectly matched to multiple tracking tracks. The deep cost matrix solves this problem by exploiting the depth relationship to prevent confusion in the corresponding box matching of tracks and targets that are located in front of or behind each other. As for DanceTrack0039, #1 was occluded by #3 for a long time. When #1 is detected again, its pre-occlusion Kalman filter state has a lower IoU value, resulting in the creation of a new #6 in the left image. Combined with C DCM The association between re-detection and pre-occlusion state is enhanced to rely not only on IoU but also on the depth dimension, which remains consistent despite the change of object position before and after occlusion.

[0112] In the PersonTracking dataset, each sequence starts with a person standing in front of the robot and the tracking target is identified from the initial frame. Our experiments exclusively use left-eye images to test the selected person tracking algorithms. Figure 6 As shown in (a), we compare our proposed framework with various tracking algorithms, including single target tracking algorithms OAB, ASE, and CNNv3 based on monocular cameras, and single target tracking algorithms SOAB, CNNv1, and CNNv2 based on stereo cameras. The multi-target tracking algorithm OC-SORT also uses a monocular camera and is also included in the comparison. The results show that when the positioning error threshold T θ When set to 50 pixels, the state-of-the-art multi-tracking algorithm OC-SORT achieves 85.8% localization accuracy in the RPF evaluation, surpassing the previous best performing algorithm CNNv1. Through the fusion of the deep cost matrix, the localization accuracy is further improved to 95.9%. Figure 6 (b) shows the localization accuracy under different position error thresholds, where our framework exhibits higher accuracy at lower thresholds and achieves 98.1% accuracy at higher thresholds, comparable to the state-of-the-art tracking algorithms.

[0113] Figure 7 11 sets of accuracy maps from the person tracking dataset are shown. It is worth noting that our multi-object tracking algorithm selects the target only in the first frame and does not rely on appearance features, but instead relies on the motion model of the target in the monocular image.

[0114] In this PersonFollowing experiment, the robot extracts pedestrian position information at a sampling frequency of 5Hz to update the target path. At the same time, the robot's lateral and longitudinal control systems are adjusted at a rate of 20Hz to achieve accurate path tracking. The experimental results are shown in the figure: Figure 8 (a)-(d) show the target’s trajectory and the robot’s tracking path under different conditions. The robot’s clear tracking path in the figure shows its ability to successfully track moving targets on various paths.

[0115] Figure 8 (e)-(h) show the target distance d under diverse tracking paths o and heading error θ o The target heading error fluctuates within the range of ±0.2, which is acceptable within the horizontal field of view of the depth camera φ = 58.4°. This shows that the robot can adapt to changes in the direction of pedestrians in real time, ensuring that the tracked object remains within the camera's field of view and continues to track while maintaining a safe distance. In addition, the figure also shows the lateral deviation d of the robot relative to the planned path p and heading deviation θ p, and shows small fluctuations around zero under dynamic tracking conditions. These results confirm the effectiveness of the proposed omnidirectional wheel-based lateral control model. We p and K i The parameters provide optimized values, which enable the robot to quickly reach a stable state under dynamic conditions and achieve real-time tracking of moving targets.

[0116] The method of the present invention was tested four times outdoors using an omnidirectional wheeled mobile robot. Fig. 9 As shown in Figure 2, the results demonstrate the robustness of the proposed multi-target tracking algorithm in terms of stable tracking of a single target. Fig. 9 (a). Although the subsequent target is blocked multiple times, the robot can still effectively track and follow the target. Fig. 9 (b). Fig. 9 As shown in (c), when the subsequent target temporarily leaves the camera's field of view, the tracker maintains the original trajectory and resumes tracking immediately when the target re-enters the frame. Fig. 9 In (d), the robot detects and locates the target based on the gesture. Only when the target makes a follow-up gesture will the robot initiate real-time tracking. Without gesture signals, the robot will not track. It is worth noting that the robot only uses Jetson Xavier NX to perform perception and control tasks throughout the experiment without any additional computing devices, highlighting the simplicity of the system.

[0117] In summary, the proposed method proposes a multi-target tracking framework designed to enhance the robot tracking capability in complex environments. The framework introduces a target localization strategy to optimize the adaptability and flexibility of the multi-target tracking algorithm. In addition, to ensure stable following, lateral and longitudinal control models are developed based on the omnidirectional wheel motion model. Evaluation tests and real-world robot following experiments show that the proposed visual tracking method significantly outperforms the existing multi-target tracking OC-SORT and single target tracking ODFS, confirming the efficiency and practicality of the proposed robot following framework. Real-world human-robot interaction tracking scenarios.

Claims

1. A robot personnel following method based on multi-target tracking, characterized in that: It mainly includes five main modules: detection module, multi-target tracking module, target positioning module, path planning module and motion control module. The work of the above five main modules includes the following steps: S1 Image Acquisition: The robot acquires each frame of color image and depth image through the Astro Pro camera to ensure high-precision input data for subsequent processing. The depth image provides key three-dimensional information for subsequent positioning and path planning. S2: Target detection: input the color image obtained in step S1 into the YOLOX model for target detection. The YOLOX model is an efficient real-time target detection algorithm that can quickly identify targets in complex scenes. The output video frame image includes the regional coordinates of all detected targets and the corresponding confidence levels, providing basic data for multi-target tracking. S3: Multi-target tracking: Input the detection frame set output by the target detection in step S2 into the multi-target tracking algorithm to track multiple targets. The multi-target tracking algorithm can not only track the position of each target, but also assign a unique ID to each target to ensure the consistency of the target between different frames. The output includes the ID corresponding to each target and the positioning frame of each target, ensuring that the robot can correctly identify and distinguish the follow-up target. S4: Target positioning: all tracking targets acquired in S3 are dynamically positioned according to a predetermined positioning strategy, and the position information of a single follow target is output to determine the target that the robot is currently following; S5: Path planning. First, the position information of the target relative to the image obtained by the target positioning clock in step S4 is combined with the depth image to obtain the distance of the robot relative to the target and convert it into position information in the world coordinate system. Then, the path trajectory between the robot and the tracking target is planned using the Dubins curve. S6: motion control, according to the tracking trajectory obtained by the path planning in step S5, the trajectory is input into the lateral and longitudinal control algorithm. The control algorithm will dynamically adjust the speed and direction of the robot according to the change of the target position to ensure a smooth and stable tracking process, and finally output the three-axis motion speed of the robot, including the forward speed, the turning speed and the vertical speed, to achieve accurate target tracking; S7: End the tracking of the current frame and enter the next frame. After completing the motion control of step S6, the robot continues to process the image data of the next frame and repeats steps S2 to S6. Through the iterative cycle, the robot can continuously track the dynamic target and achieve real-time following of the target.

2. The robot personnel following method based on multi-target tracking as claimed in claim 1, characterized in that: In the process of target detection in step S2, the YOLOX-NANO lightweight model is first used and deployed on the Jetson XavierNX embedded platform. The model uses COCO pre-trained weights and finally outputs the detection set Z of the current frame t. t =[u,v,s,r], where u,v,s and r represent the position coordinates, area and aspect ratio of the center point of the detection box respectively. In order to improve the performance of the model in crowd clustering scenarios, the CrowdHuman, Cityperson and ETHZ datasets were selected to improve the generalization ability of the YOLOX-NANO model in practical applications after using the COCO pre-training weights, ensuring its accuracy and robustness in real scenarios.

3. The robot personnel following method based on multi-target tracking as claimed in claim 1, characterized in that: In the process of multi-target tracking in step S3, the motion prediction model of multi-target tracking is based on the Kalman filter (KF) to update the state of the tracked object, where the state vector is defined as Represents the position (u, v), area (s), aspect ratio (r) and their respective velocity components. In the same frame, the position z detected by the YOLOX detector t = [u,v,s,r] is used as the observation vector; Assuming that the object moves at a constant speed, the following Kalman prediction equation is used to estimate the state vector at frame t: Among them, F t represents the state transition matrix, W t is the process noise, P t|t-1 is the forecast error covariance, Q t is the process noise covariance matrix. Then, the YOLOX detector tracks the target z t The observations at frame t correct the prior estimate and update the state vector Among them, H t is the observation matrix, R t is the observation noise covariance matrix, K t is the Kalman gain matrix. When the track is blocked and there is no observation update z t When the state error is , it may accumulate over time, causing the KF parameters to deviate from the correct path, thereby increasing the prediction error. In order to reduce this accumulated error, when the object is re-detected after being occluded, its state during the occlusion is traced back, a virtual trajectory is created, and the KF parameters are updated, assuming that the last observation value when the object was occluded is The first observation value when the object is re-detected is Then the virtual observation state z between t1 and t2 t It can be described as: Here, along the virtual trajectory z t (where t1 < t < t2), for recalculating the prediction and update steps of the Kalman filter; In the data association part, at frame t, the YOLOX detector generates a set of detection boxes The Kalman filter generates a set of predicted tracking boxes For any given detection box and prediction box An IoU value can be calculated as the cost matrix C IoU Similarity measure in: Based on the IoU cost matrix, the depth cost matrix C is introduced DCM : Function F can obtain the vertical coordinate of the bottom edge of the box, so as to better compare the relative depths of different objects on the same ground plane; Then, the IoU cost matrix C IoU With C DCM Combined, we can get the weighted data association cost matrix C d : C d =C IoU (i,j)+ω DCM ·C DCM (i,j) Among them, ω DCM It is C DCM Matrix coefficients, the Hungarian algorithm is obtained by d The goal is to minimize: Indicates the trajectory set from the current frame To the detection box Z t The set of all permutations and combinations σ that perfectly match, output the result Corresponding to the matched activation track, unmatched track and unmatched high-score box, respectively, for the matched track Use the Kalman filter to update the trajectory parameters and output them as results. Keep it. If there are a certain number of unmatched ones, delete them. Unmatched detection boxes Used to create a new track and assign an ID number for matching in the next frame.

4. The robot personnel following method based on multi-target tracking as claimed in claim 1, characterized in that: In step S4 target positioning, the multi-target tracking system can automatically identify and select the tracking ID located at the center of the initial frame image as the main tracking target. The above process is implemented by the following formula: Among them, z0 represents the detection result in the first frame, and are the horizontal coordinates of the left and right edges of the detection box, w is the width of the image, and the center of the box (x c ,y c )The individual closest to the center of the image It will be identified as the tracking target. Using the depth value d of the RGB-D camera, the three-dimensional spatial coordinates of the tracking target relative to the robot can be calculated as follows: Where φ represents the horizontal field of view of the camera; MobileNet is used for gesture detection. The network is pre-trained on the HaGRID dataset to detect specific gestures.

5. The multi-target tracking method based on pseudo depth estimation and online trajectory classification according to claim 1, characterized in that: In the solution process of the path planning in step S5: after receiving the position information (x, y, θ) of the tracking target after step S4 (with its own coordinate system as a reference), it is responsible for generating discrete trajectories and using Dubins for path planning. The Dubins curve provides six basic path type combinations (LSL, LSR, RSL, RSR, LRL, RLR). Before discretizing the planned curve, determine the shortest path combination and perform coordinate transformation. Set the starting point of the robot to s(0,0,φ) and the end point of the tracking target to g(d,0,ψ). The specific transformation formula is as follows: Among them, φ, d, ψ and R represent the robot's heading angle, the normalized distance from the robot to the target, the target's heading angle and the robot's turning radius, respectively. In the LSL path type, the subpath length is calculated as follows: By calculating the starting rotation angle t, path length p and final rotation angle q, the length of the LSL path combination is determined. Then, the shortest path is selected from the six basic combinations for discretization. The discretization is completed by using the interpolation method. The continuous Dubins path is decomposed into a series of discrete points. Given the discrete step length l and the starting interpolation state (x o ,y o ,θ o ), the interpolation formula for each path segment is: in is the turning radius, the path length has been normalized during the calculation process, and an iterative interpolation is performed to generate a complete set of discrete path points until the planned path length is reached. Finally, this set of points is transmitted to the path tracking module for further processing.

6. The multi-target tracking method based on pseudo depth estimation and online trajectory classification according to claim 1, characterized in that: In step S6, the robot motion control is designed based on the omnidirectional wheeled robot model, where the lateral and longitudinal control models are designed to compensate for the error in the planned tracking trajectory. The lateral controller uses the LQR linear controller to calculate the robot's angular velocity ω and lateral velocity v according to the deviation between the robot's current position and the planned path. y On the contrary, the longitudinal controller uses a PID controller to calculate the longitudinal velocity v of the robot according to the distance between the robot and the tracking target. x ; The lateral control system uses an LQR linear controller to adjust the lateral error d of the robot relative to the trajectory. p and heading error θ p , these deviations are determined by comparing the actual position of the robot with its projection on the intended path. The lateral control error is defined as The control input is u=[v y ,ω] T ; First, a mathematical representation of the difference between the actual position of the robot and its projection point on the predetermined path is established, and its heading error e θ and its rate of change It is expressed as follows: e θ =θ-θ r Then, define the lateral error e d As the robot's current position vector Matching point position vector with planned trajectory In the normal direction Projection difference on : Then, by changing the robot's position vector The position change vector of the matching point with the planned trajectory The difference in The projection in the direction, and Rate of change over time The influence of lateral error change rate is derived Here, the robot's position change vector Expressed as the velocity component in the body coordinate system and The linear combination of the planned trajectory matching point position change vector The velocity along the trajectory is taken into account Consider the curvature k of the planning curve with respect to the normal vector The effect of the rate of change, according to Frenet's formula have Substitute the formula, expand and simplify to get: Assume θ-θ r is the minimum value, then the lateral error change rate for: Then according to the linear state space model There is lateral control error The lateral control problem is framed and the state transfer matrix is ​​introduced With control matrix And discretize it to form the lateral control error e at the next moment t+1 The prediction of , the model is expressed by the following matrix equation: The lateral control error The horizontal control variable u=[a y ,α] T , then we have Therefore, the state transition matrix and control matrix It can be determined by geometric relations. By defining the cost function J and minimizing the sum of the weighted long-term cumulative error and the control effort, we propose a discrete LQR problem: The weight matrices Q and R are introduced and given the dynamic constraints Finally, the Riccati equation is derived by applying the Lagrange multiplier method: By continuously iterating to solve the P matrix, the feedback gain matrix K is then solved: Finally, the control matrix u is solved k : you t =-I am t Throughout the process, the lateral control system is simulated as a linear system that reduces the error e by adjusting the control input u; The longitudinal control system uses a PID controller to maintain a predetermined distance D between the robot and the tracking target, and the longitudinal velocity v of the robot x It is proportional to the error between the current tracking target depth and the set distance. The relationship between the longitudinal velocity and the error term is shown in the following formula: v x =K p *(d-D)+K i *∫(d-D)dt Among them, K p and K i To control the gain, dD is the error between the current depth of the target and the predetermined distance, and dt is the time interval between consecutive frames.

Images