A method and related equipment for target localization and tracking of a mobile robot

Abstract

This application discloses a method and related equipment for target localization and tracking of a mobile robot, including: pre-localizing the robot itself and constructing a scene map; acquiring scene image data using the robot; performing target detection on the acquired scene image data based on the YOLO algorithm; inputting the target detection results of the YOLO algorithm into an improved tracker, StrongSORT, for target localization. The improved tracker, StrongSORT, includes an appearance branch and a motion branch. The appearance branch is used to detect the appearance of the target and extract its appearance features, while the motion branch is used to predict the target's tracking information. Based on the predicted tracking information and the extracted appearance features, the tracked target is matched against all target detection results in the current frame to determine the target localization result. The method of this application solves the problems in the prior art of handling target occlusion and the planned tracking trajectory not conforming to the dynamic characteristics of legged robots.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method and related equipment for target localization and tracking of a mobile robot. Background Technology

[0002] Target localization and tracking technology has long been a crucial research area in mobile robotics. Accurate target localization and effective tracking control methods enable mobile robots to autonomously perceive target and environmental information, and generate appropriate motion trajectories for tracking control to achieve target tracking. This allows mobile robots to undertake various practical tasks such as following people and vehicles. However, the complex environments of various application scenarios also present many challenges to robot target localization and tracking, such as localization methods when targets are occluded and trajectory generation strategies suitable for legged robot tracking control. The ability to accurately capture and locate targets and generate appropriate tracking trajectories in complex scenarios is a key indicator of a mobile robot's good target localization and tracking capabilities. Therefore, sufficiently robust target localization and trajectory generation suitable for legged robots in target tracking are of great significance for the application of legged robots in real-world scenarios. Vision-based target localization can be achieved through target tracking. Target tracking refers to identifying a specified target in an image sequence and achieving accurate target localization. Motion trajectory generation for tracking control can generate a suitable motion trajectory based on the target's localization information and environmental information to control the mobile robot to track the target. This enables the robot to have accurate target localization capabilities and obtain a tracking trajectory suitable for the motion control of the legged robot, so as to continuously and stably complete various practical tasks.

[0003] Currently, the mainstream methods used in related fields to achieve target localization and tracking control include the following:

[0004] (1) Target tracking based on correlation filtering

[0005] In industry, traditional methods based on correlation filtering remain the mainstream solution for target tracking. These algorithms typically model the target's motion or appearance features and search for targets matching the model in subsequent frames. Among these, Kalman filtering is the most common method in traditional algorithms. It can use the target's localization information to represent the target's state and calculate the current state estimate based on the target state estimate from the previous moment, the estimated motion model, and the current observation state. However, traditional filtering has poor robustness and struggles to handle various common scenarios such as background interference, target occlusion, and changes in illumination, failing to meet the requirements for persistence and stability in target localization tasks. Furthermore, traditional Kalman filters have limited ability to handle complex motion models.

[0006] (2) Target tracking based on deep learning

[0007] In recent years, with the development of deep learning technology, deep learning-based target tracking methods have also made significant progress. These methods learn feature representations of targets by training on large amounts of data. Deep models no longer require manual parameter tuning and can extract depth representations of images to adapt to various scenarios. For example, DeepSORT is a classic deep learning target tracking method. It uses a deep object detection network to identify and locate all targets in each frame, then uses a deep re-identification network to extract feature representations of these targets. By calculating similarity, it matches the tracked target to the corresponding detected target, transforming the target tracking problem into a data association problem between the tracked target and the target detection result, thus achieving the localization of the specified target. With the help of extracting depth representations of images and online update strategies, the DeepSORT method has a certain ability to handle image interference such as background noise, lighting changes, and target deformation. However, its effectiveness depends on the target detection results. When target occlusion causes target loss or target detection failure, DeepSORT still struggles to achieve continuous target tracking.

[0008] (3) Sampling-based path planning method

[0009] Sampling-based path planning methods are model-independent and degree-of-freedom-independent path planning approaches. They are applicable to various complex constraints and can plan paths to target locations in both 2D and 3D maps, enabling robot tracking and control of specified targets. However, due to the sparsity of sampling points, the paths constructed through random sampling are often of low quality, exhibiting problems such as sharp edges and lack of smoothness. These tracking paths and trajectories, which do not conform to the dynamics of legged robots, will affect the robot's motion control, making it difficult to achieve stable tracking control in practical applications.

[0010] (4) Optimized trajectory generation strategy

[0011] Optimization-based trajectory generation strategies generate paths that satisfy constraints by optimizing the objective function, producing high-quality, smooth motion trajectories for tracking control of legged robots. For example, polynomial trajectory generation is a commonly used optimization-based path planning method that generates polynomial trajectories that meet constraints by solving a convex optimization problem. However, optimization-based trajectory generation strategies require setting corresponding objective functions based on the motion patterns of different robot types.

[0012] Existing technologies have the following drawbacks:

[0013] 1) For a long time, traditional methods based on correlation filtering have been the mainstream methods for target tracking and localization in industry. The main problems they face are: these methods have poor robustness and are difficult to handle common problems such as background interference, target occlusion, and changes in illumination. Therefore, these methods are difficult to continuously and stably locate targets in complex environments.

[0014] 2) Deep learning-based target tracking algorithms have made some progress in recent years, and DeepSORT is one of the most classic methods among deep learning-based multi-target tracking algorithms. It uses features extracted by a simple re-identification network as appearance features based on target detection results, and achieves target tracking and localization by associating the target with the target detection results. Its main problem is that this type of method directly uses the re-identification features of the target detection results as appearance features for data association, making its target localization performance highly dependent on the target detection results and difficult to handle target detection failures such as target occlusion.

[0015] 3) Sampling-based path planning methods establish a path network graph in the map through random sampling, and then perform path planning on the path network graph based on the target's location information. The main problem it faces is that, since the sampling is random and unconstrained, sampling-based path planning methods may generate unsmooth, low-quality paths, which will affect the robot's motion control and make it difficult to achieve stable target tracking.

[0016] 4) Optimization-based trajectory generation strategies generate trajectories that satisfy constraints by optimizing the objective function. This type of method was first widely applied in the field of unmanned aerial vehicles (UAVs). It designs constraints based on the dynamic characteristics of UAVs and solves the optimization function to generate high-quality, smooth trajectories in 3D space. Later, it was extended to various robotics fields. The main problem it faces is that the dynamic characteristics of UAVs differ from those of legged robots. When constraints designed for UAVs are directly applied to legged robots, they sometimes result in trajectories that contradict the dynamic characteristics of the legged robot. For example, if the generated trajectory satisfies the dynamic characteristics of the UAV, causing other motion parameters to be discontinuous, the legged robot may require infinite torque to track this trajectory, making long-term stable motion control difficult and leading to target tracking failure. Summary of the Invention

[0017] This application provides a mobile robot target localization and tracking method and related equipment to solve the problems in the prior art of dealing with target occlusion and the planned tracking trajectory not conforming to the dynamic characteristics of legged robots.

[0018] This application provides a method for target localization and tracking of a mobile robot, including the following steps:

[0019] Pre-positioning of the robot and construction of scene maps;

[0020] The robot is used to acquire scene image data;

[0021] Target detection is performed on the acquired scene image data based on the YOLO algorithm;

[0022] The target detection results from the YOLO algorithm are input into the improved tracker StrongSORT for target localization. The improved tracker StrongSORT includes an appearance branch and a motion branch. The appearance branch is used to extract the appearance features of the target, and the motion branch is used to predict the target's tracking information. Based on the predicted tracking information and the extracted appearance features, the tracked target is matched against all target detection results in the current frame to determine the target localization result.

[0023] Optionally, the robot's self-localization and scene map construction are performed in advance based on FAST-LIO2.

[0024] Optionally, the target detection results from the YOLO algorithm can be input into the improved tracker StrongSORT for target localization, including:

[0025] In the appearance branch, the feature extractor for the target detection results is a lightweight re-identification network, OSNet. OSNet is used to extract multi-scale features of the target's appearance. These extracted multi-scale features are fused and input into the re-identification network. The appearance features extracted by the re-identification network are then processed using an exponential moving average (EMA) method to incorporate temporal information, resulting in a comprehensive feature e containing both appearance and temporal motion information, satisfying the following:

[0026]

[0027] In the formula, This represents the comprehensive feature of the i-th trajectory in the t-th frame of the image. α is the momentum term, representing the feature extracted by the re-identification network from the target detection result corresponding to the i-th trajectory in the t-th frame.

[0028] In the motion branch, an NSA Kalman filter is used to predict the target state, and this NSA Kalman filter is configured with an adaptive covariance matrix. satisfy:

[0029]

[0030] Where R k c represents the pre-defined constant noise covariance.k This represents the detection confidence score under state k.

[0031] Optionally, for target detection results from the YOLO algorithm, inputting them into the improved tracker StrongSORT for target localization also includes:

[0032] Combine the comprehensive feature e of the target detection result with the previous L k The cosine distance calculated from the tracking results of each iteration is used as the cost of the appearance branch, satisfying the following:

[0033]

[0034] in, This represents the transpose of the comprehensive feature e extracted from the detection result of the j-th target. L represents the prior L of the i-th tracking result. k Synthetic features in the frame;

[0035] The Mahalanobis distance between the predicted state of the NSA Kalman filter and the state of the target detection box is calculated and used as the cost of the motion branch, satisfying the following:

[0036]

[0037] Where, d j y i These represent the target detection result state and the tracking state predicted by the Kalman filter, respectively. (u,v) represents the coordinates of the target in the image, and (s,r) represents the scale and aspect ratio of the detection box. This represents the rate of change of the corresponding parameter. For d j y i The inverse of the covariance matrix;

[0038] The cost matrix is ​​then:

[0039] C=λA a +(1-λ)A m

[0040] Where λ is the coefficient;

[0041] The target detection results, predicted tracking information, cost matrix, and preset threshold matrix are matched, and each target detection result is matched with the corresponding target tracking result to achieve target tracking.

[0042] Optionally, for target detection results from the YOLO algorithm, inputting them into the improved tracker StrongSORT for target localization also includes:

[0043] Based on the point cloud information of the target tracking depth image, the ground point cloud within the detection box that serves as the background is removed to obtain the target localization result.

[0044] Optionally, it also includes generating the target's tracking trajectory by calculating the trajectory generated by the minimum jerk based on the robot's joint motion, wherein one of the minimum jerk trajectories is obtained by solving the following constrained optimization problem:

[0045]

[0046] Where j represents the third derivative of position with respect to time, jerk,x(t), x * (t) represent the robot position and the optimal jerk trajectory as the rolling time T changes, respectively. It is the objective function. Let x(t) be the third derivative of x(t), and T be the rolling time.

[0047] The multiple candidate trajectories obtained from the solution are filtered, the score of each trajectory is calculated, and the trajectory with the highest score is selected as the motion trajectory of the generated target.

[0048] Optionally, calculating the score of each trajectory and selecting the trajectory with the highest score as the generated target's motion trajectory includes heuristic scoring based on the grid map and scoring based on target distance. Heuristic scoring encourages the planned trajectory to pass through free areas in the grid map and avoids selecting trajectories that occupy areas as much as possible. The target distance-based scoring method directly calculates the Euclidean distance between the endpoint of the candidate trajectory and the target.

[0049] This application also proposes a mobile robot target localization and tracking device, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the mobile robot target localization and tracking method described above.

[0050] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0051] This application also proposes a robot, including the aforementioned mobile robot target localization and tracking device or the aforementioned computer-readable storage medium.

[0052] This application addresses the problems of current mainstream technologies, such as difficulty in handling target occlusion and the inability of planned tracking trajectories to conform to the dynamic characteristics of legged robots. It proposes target localization based on target tracking and target tracking based on trajectory generation, thus solving the problems in the field of target localization and tracking technology for mobile robots, such as the poor ability of legged robots to handle target occlusion and the difficulty in generating motion trajectories that conform to the dynamic characteristics of legged robots.

[0053] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0054] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0055] Figure 1 This is a schematic diagram of the basic process of the mobile robot target localization and tracking method according to an embodiment of this application;

[0056] Figure 2 This is a schematic diagram of the OSNet re-identification network model structure in an embodiment of this application;

[0057] Figure 3 This is a schematic diagram of the StrongSORT target tracking algorithm used in the embodiments of this application;

[0058] Figure 4 The effect of the target tracking algorithm is shown in the example of the method application in the embodiments of this application;

[0059] Figure 5 A ROS-based simulation map serving as an example of the application of the method in the embodiments of this application;

[0060] Figure 6 This is a simulation illustration of the target localization method as an example of the application of the method in the embodiments of this application.

[0061] Figure 7 A visual illustration of the trajectory generation strategy used in the application examples of the method in this application;

[0062] Figure 8 This is a schematic diagram illustrating the motion trajectory effect of a target tracking task, which is an example of the application of the method in this embodiment of the application. Detailed Implementation

[0063] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0064] This application provides a method for target localization and tracking of a mobile robot, such as... Figure 1 As shown, it includes the following steps:

[0065] In step S101, the robot itself is pre-localized and a scene map is constructed. In some embodiments, the pre-localization of the robot and the construction of the scene map are implemented based on FAST-LIO2. In some specific examples, FAST-LIO2 is a fast and efficient SLAM algorithm that can construct a point cloud map of the environment based on the measurement data of the LiDAR and inertial measurement unit of the legged robot, and calculate the robot's position, attitude, and velocity data relative to the starting point, thereby achieving state estimation of the robot. A grid map is a probabilistic map that uses grids as basic units to represent environmental elements. Each grid is calculated to determine whether it is a passable area. The grid map construction method based on point cloud data is relatively mature and can be directly called in various robot programming libraries (such as the ORB-SLAM2 library). The robot completes its own state estimation based on FAST-LIO2 and obtains a map containing passability information.

[0066] In step S102, the robot is used to acquire scene image data.

[0067] Target localization consists of three steps: target detection in the image using a target detection algorithm, target tracking using a target tracking algorithm based on the target detection results, and final target localization calculated from the point cloud information based on the target tracking results. In step S103, target detection is performed on the acquired scene image data using the YOLO algorithm. In some specific applications, target detection can be implemented using a lightweight neural network, YOLOv4-tiny. RGB images are input into the YOLOv4-tiny backbone network for feature extraction. The extracted features are then fused using a feature pyramid and input into the YOLO detection head for target detection, ultimately obtaining the final target detection result pred = (P c ,b x ,b y ,b h ,b w Pc represents the confidence level of this target detection, and (b) represents the confidence level of the target detection. x ,b y ,bh ,b w The coordinates of the center point of the detection box, along with its height and width, describe the location of the target detection result. C represents the target detection category.

[0068] In step S104, the target detection results of the YOLO algorithm are input into the improved tracker StrongSORT for target localization. The improved tracker StrongSORT includes an appearance branch and a motion branch. The appearance branch is used to extract the appearance features of the target, and the motion branch is used to predict the tracking information of the target. Based on the predicted tracking information and the extracted appearance features, the tracked target is matched with all target detection results in the current frame to determine the target localization result.

[0069] In other words, in this embodiment, the motion branch of the improved tracker StrongSORT is used to obtain predicted tracking information, and the appearance branch is used to extract appearance features. Then, based on the tracking information and appearance features obtained from these two branches, the tracked target is matched against all target detection results in the current frame. An exemplary matching method involves calculating the distance between the tracking information and appearance features of each target detection result and the tracking information and appearance features of the tracked target in the previous frame, thereby achieving target tracking.

[0070] In this embodiment, the tracking information can be understood as the target's identity identifier (ID) plus motion information. The ID is a unique identifier for each target, used to specify a target within consecutive time frames. The motion information refers to the target's position and rate of change of position; the Mahalanobis distance is calculated based on this motion information. Essentially, the matching process in the tracker assigns the target's ID to the correct target detection result based on the calculated distance (ultimately weighted to calculate the cost matrix).

[0071] The target localization described in this application embodiment is completed based on target tracking. That is, after the tracked target is matched to the target detection box, in some embodiments, the target detection results from the YOLO algorithm are input into the improved tracker StrongSORT for target localization, which further includes:

[0072] Based on the point cloud information of the depth image for target tracking, ground point clouds within the detection box that serve as background are removed to obtain the target localization result. Specifically, based on the point cloud information of the depth image, the ground point cloud within the detection box that serves as background is simply removed. The ground is approximated as a plane aX + bY + cZ = 1. Points with a distance less than a certain threshold T are approximately classified as ground point clouds. The effective target point cloud includes the ground point cloud, and the remaining point cloud within the box is then calculated. The calculation process satisfies the following:

[0073]

[0074] Where P represents the overall point cloud within the detection box for target tracking, G represents the ground point cloud, Q represents the effective point cloud, and the centroid of Q will be used as the final state estimate of the target, i.e. the target localization result. The parameters of the plane equation aX+bY+cZ=1 can be calculated by randomly sampling the perigee near the detection box.

[0075] This application addresses the problems of current mainstream technologies, such as difficulty in handling target occlusion and the inability of planned tracking trajectories to conform to the dynamic characteristics of legged robots. It proposes target localization based on target tracking and target tracking based on trajectory generation, thus solving the problems in the field of target localization and tracking technology for mobile robots, such as the poor ability of legged robots to handle target occlusion and the difficulty in generating motion trajectories that conform to the dynamic characteristics of legged robots.

[0076] In some embodiments, target localization based on the target detection results of the YOLO algorithm includes:

[0077] In the appearance branch, the feature extractor for the target detection results is a lightweight re-identification network, OSNet, which extracts multi-scale features of the target's appearance. These extracted multi-scale features are then fused and input into the re-identification network. The improved StrongSORT tracker used in this embodiment is a further optimization based on DeepSORT, including the following optimizations: In the appearance branch, the feature extractor for the target detection results is replaced by a simple CNN network from DeepSORT with the more powerful lightweight re-identification network, OSNet. This network model performs multi-scale feature extraction simultaneously by setting convolutional flows with different receptive fields. The extracted features at each scale are fused through a feature aggregation gate, ultimately obtaining features with an omni-scale (Omin-Scale) for re-identification after target occlusion. Its structural diagram is shown below. Figure 2 As shown, OSNet is pre-trained on a pedestrian re-identification dataset, and the model weights only require about 2MB, while ensuring lightweight deployment and re-identification performance.

[0078] Then, the appearance features extracted by the re-identification network are combined with temporal information using the exponential moving average (EMA) method to obtain a comprehensive feature e containing both appearance and temporal motion information, satisfying:

[0079]

[0080] In the formula, This represents the comprehensive features of the i-th trajectory in the t-th frame of the image. α is the momentum term, representing the feature extracted by the re-identification network from the target detection result corresponding to the i-th trajectory in the t-th frame.

[0081] In the motion branch, the Kalman filter used to predict the target state is replaced with an NSA Kalman filter to predict the target state, the NSA Kalman filter being configured with an adaptive covariance matrix. satisfy:

[0082]

[0083] Where R k c represents the pre-defined constant noise covariance. k This represents the detection confidence score under state k.

[0084] Motion information is extracted from the motion branch. In this embodiment, the predicted target state is the motion information extracted from the motion branch. It can be considered that the closer the state of the target detected in the current frame is to the state predicted based on Kalman filtering, the greater the probability of a successful match between the tracked target and the target. Kalman filtering is based on the motion information of the previous frame (including the target's position, rate of change of position, rate of change of velocity, etc.) for prediction, i.e., motion information. In specific applications, an identity identifier (ID) is maintained for this motion information, i.e., tracking information is formed.

[0085] In some embodiments, the target detection results from the YOLO algorithm are input into the improved tracker StrongSORT for target localization, which further includes:

[0086] The cosine distance between the comprehensive feature e of the target detection result and the tracking results of the previous Lk iterations is calculated and used as the cost of the appearance branch, satisfying the following:

[0087]

[0088] in, This represents the transpose of the comprehensive feature e extracted from the detection result of the j-th target. This represents the comprehensive feature of the i-th tracking result in the preceding Lk frames. Lk can be specifically selected according to the actual task.

[0089] The Mahalanobis distance between the predicted state of the NSA Kalman filter and the state of the target detection box is calculated and used as the cost of the motion branch, satisfying the following:

[0090]

[0091] Where, d j y iThese represent the target detection result state and the tracking state predicted by the Kalman filter, respectively. (u, v) represents the coordinates of the target in the image, and (s, r) represents the scale and aspect ratio of the detection box. This represents the rate of change of the corresponding parameter. For d j y i The inverse of the covariance matrix;

[0092] The cost matrix is ​​then:

[0093] C=λA a +(1-λ)A m

[0094] Where λ is a coefficient, and λ can be 0.98.

[0095] The target detection results, predicted tracking information, cost matrix C, and preset threshold matrix B are cascaded and matched, and each target detection result is matched with the corresponding target tracking result to achieve visual target tracking. The cascaded matching process is shown in Table 1.

[0096] Table 1. Schematic diagram of the cascaded matching process used in the tracker.

[0097]

[0098] The method of this application also includes target tracking based on trajectory generation. The joint motion of the legged robot is algebraically related to the third derivative (jerk) of the position. In some embodiments, it also includes generating a target tracking trajectory by calculating the trajectory generated by the minimum jerk based on the robot's joint motion. One of the minimum jerk trajectories is obtained by solving the following constrained optimization problem:

[0099]

[0100] Where j represents the third derivative of position with respect to time, jerk, x(t), x * (t) represent the robot position and the optimal jerk trajectory as the rolling time T changes, respectively. It is the objective function. Let x(t) be the third derivative of x(t), and T be the rolling time. The rolling time T can be set as the trajectory length l. traj And the robot's maximum speed v max The merchant, that is

[0101] This function can be minimized by solving the following Euler-Lagrange equation:

[0102]

[0103] By solving the Euler-Lagrange equations in the X and Y directions, we can obtain polynomial functions of the desired position with respect to time in each direction:

[0104] x * (t)=c k t k +c k-1 t k - 1 +…+c1t+c0,k=2n-1

[0105] Using the above equation to solve a linear system of the form Ax = b, we obtain the desired position, velocity, acceleration, and acceleration constraints at time t. By setting the linear velocity and the yaw range of the search, we obtain the constraints:

[0106]

[0107] Where v represents linear velocity, ψ i Represent the set of yaw angles in the same dimension using a fractional matrix. Set row to 9 to cover the yaw range of [-60°, 60°]. Similarly, derive the constraints from the first trajectory to the second trajectory as follows:

[0108]

[0109] Set the branch to 7 to cover the yaw range of [-27°, 27°] from the focus of the first trajectory. This generates a series of candidate trajectories with the minimum jerk. Using the constraints derived above, the minimum jerk trajectory can be pre-calculated offline using C or MATLAB in approximately one millisecond.

[0110] Then, the multiple candidate trajectories obtained from the solution are filtered, the score of each trajectory is calculated, and the trajectory with the highest score is selected as the motion trajectory of the generated target.

[0111] In some embodiments, calculating the score of each trajectory and selecting the trajectory with the highest score as the motion trajectory of the generated target includes heuristic scoring based on a grid map and target distance scoring. Heuristic scoring encourages the planned trajectory to pass through free areas in the grid map and avoids selecting trajectories that pass through areas that are occupied as much as possible. Target distance scoring directly calculates the Euclidean distance between the endpoint of the candidate trajectory and the target.

[0112] In some specific examples, heuristic scoring is shown in Table 2.

[0113] Table 2 illustrates the heuristic scoring method used in this application example.

[0114]

[0115] The scoring method based on target distance directly calculates the Euclidean distance between the endpoint of the candidate trajectory and the target. The closer the distance, the higher the score. The scoring method is as follows:

[0116]

[0117] Among them, (x i y i , z i Let (x) be the coordinates of the endpoint of the i-th trajectory. t y t , z t ) is the state estimate coordinate of the target being followed.

[0118] Therefore, the heuristic score matrix S of each generated trajectory is calculated. h Distance score matrix S to target e Then, the two matrices are multiplied using the Hadamard product operation to obtain the final comprehensive score matrix S:

[0119] S = S h *S e

[0120] Based on the comprehensive score matrix S, the trajectory with the highest score is selected as the final generated motion trajectory. Thus, the legged robot can move along the planned optimal trajectory to achieve target tracking.

[0121] This application presents a target localization and tracking method suitable for mobile robots. Unlike existing traditional methods, to verify its effectiveness, the flow and performance of the target tracking algorithm for target localization are as follows: Figure 3 , Figure 4 As shown, Figure 4 In frame 1, the pedestrian with ID 93 is obscured by the pedestrians side-by-side within the red box in frame 2. When the pedestrian reappears in frame 3, they are immediately captured and successfully re-identified as pedestrian number 93. The simulation environment built based on ROS is as follows: Figure 5 As shown, Figure 5 In the process, the robot will attempt to locate and track the target car on the map.

[0122] The target localization effect used in the simulation is as follows: Figure 6 As shown, the target being tracked is a jackal car with a tracking ID of 1. The target was detected in the second frame because it was partially occluded by the edge. However, it was recaptured and successfully re-identified as the jackal car with tracking ID 1 in the third frame, thus achieving sustainable localization under target occlusion.

[0123] The effect of the minimum jerk trajectory generation strategy used is visualized as follows: Figure 7As shown, in the constructed OctoMap map, the sphere represents the tracking target, the arrow points to the location of the legged robot, the radiating lines are the generated minimum jerk trajectory clusters, and the lines connecting the legged robot to the target are the best candidate trajectories selected through scoring.

[0124] The motion trajectory of a legged robot performing target localization and target tracking tasks is as follows: Figure 8 As shown, each trajectory represents the motion trajectory of the target being tracked, the planned path, and the actual motion trajectory of the robot. The arrows indicate the starting position of the target and the end point of the current movement, respectively. The legged robot continuously moves, traversing various scenes in the simulation environment, ultimately achieving a complete target localization and tracking operation.

[0125] Thanks to the neural network and NSA Kalman filter trained on the re-identification dataset used by StrongSORT, the robot can better handle the situation of target occlusion when performing target localization, and quickly find the original target again after a brief loss of the target. At the same time, based on the trajectory of the minimum jerk, the legged robot's movement is more stable and continuous when performing target tracking.

[0126] The neural network models used in the target localization method of this application can all be trained on various publicly available datasets. Only the downstream parameters of the network need to be fine-tuned or directly applied to target tracking, which greatly simplifies the data collection and labeling process.

[0127] The robot's dynamic characteristics are optimized, and the minimized jerk trajectory generated by this trajectory generation strategy can be well adapted to the movement of legged robots, enabling them to perform stable and continuous target tracking in relevant practical tasks.

[0128] This application's embodiments innovatively design and employ the StrongSORT target tracking algorithm and a trajectory generation strategy based on minimum jerk constraints to achieve an efficient and stable target following method for legged robots. By using the StrongSORT target tracking algorithm, the ability of legged robots to handle target occlusion problems in target following tasks is improved. By employing the trajectory generation strategy based on minimum jerk constraints, the problem of generating motion trajectories that conform to the dynamic characteristics of legged robots is solved, and the trajectory generation speed is improved to a certain extent. The StrongSORT target tracking algorithm and the trajectory generation strategy based on minimum jerk constraints used in this invention have originality and broad application prospects in the field of vision-based legged robots.

[0129] This application also proposes a mobile robot target localization and tracking device, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the mobile robot target localization and tracking method described above.

[0130] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0131] This application also proposes a robot, including the aforementioned mobile robot target localization and tracking device or the aforementioned computer-readable storage medium.

[0132] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0133] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0135] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A mobile robot target positioning, tracking method, characterized by, Comprising the following steps: Pre-positioning the robot itself and scene map construction; Obtaining scene image data by the robot; Target detection on the obtained scene image data based on YOLO algorithm; For the target detection result of YOLO algorithm, input into the improved tracker StrongSORT for target positioning, the improved tracker StrongSORT comprising an appearance branch and a motion branch, the appearance branch being used to extract the appearance features of the target, and the motion branch being used to predict the tracking information of the target, so as to match the tracked target among all target detection results in the current frame based on the predicted tracking information and the extracted appearance features, to determine the target positioning result; For the target detection result of YOLO algorithm, input into the improved tracker StrongSORT for target positioning comprises: In the appearance branch, the feature extractor of the target detection result is a lightweight re-identification network OSNet, so as to extract multi-scale features of the appearance of the target by using the lightweight re-identification network OSNet, and the extracted multi-scale features are input into a re-identification network after fusion, and the appearance features extracted by the re-identification network are added with time sequence information by an exponential moving average (EMA) method, so as to obtain comprehensive features with appearance information and time sequence motion information satisfies: wherein, represents the comprehensive feature of the i-th track in the t-th frame image, is the feature extracted by the re-identification network for the target detection result corresponding to the i-th track in the t-th frame, is the feature extracted by the re-identification network for the target detection result corresponding to the i-th track in the t-th frame, is the feature extracted by the re-identification network for the target detection result corresponding to the i-th track in the t-th frame, is the momentum term; In the motion branch, a NSA Kalman filter is used to predict the target state, which is configured with an adaptive covariance matrix satisfies: wherein represents a pre-set constant measurement noise covariance, represents a state under detection confidence score; For the target detection result of YOLO algorithm, input into the improved tracker StrongSORT for target positioning further comprises: combining features of target detection results L k the cosine distance of the previous L in, Indicates the first The comprehensive features extracted from the target detection results transpose, Indicates the first The tracking results in the previous L k Synthetic features in the frame; Calculate Mahalanobis distance between the predicted state of the NSA Kalman filter and the state of the target detection box as the cost of the motion branch: wherein, , are the state of the target detection result and the tracking state predicted by the Kalman filter, respectively, the state is , represents the coordinates of the target on the image, represents the scale and the aspect ratio of the detection box, then represents the rate of change of the respective corresponding parameters, is , the inverse matrix of the covariance matrix of Then the cost matrix is: wherein, are coefficients; Match the target detection result, the predicted tracking information, the cost matrix and the preset threshold matrix, and match each target detection result with the corresponding target tracking result, to realize target tracking; Further comprising generating a tracking trajectory of the target based on the joint motion of the robot, wherein one minimum jerk trajectory is obtained by solving the following constrained optimization problem: in, The third derivative of position with respect to time, jerk. , These represent the time as the scroll progresses. Changing robot position and optimal jerk trajectory, It is the objective function. for The third derivative, where T is the rolling time; Screen the plurality of candidate trajectories obtained by solving, calculate the score of each trajectory and select the trajectory with the highest score as the generated motion trajectory of the target.

2. The mobile robot target locating and tracking method of claim 1, wherein, The pre-positioning of the robot itself and the scene map construction are realized based on FAST-LIO2.

3. The mobile robot target locating and tracking method of claim 1, wherein, For the target detection result of YOLO algorithm, input into the improved tracker StrongSORT for target positioning further comprises: Based on the point cloud information of the target tracking depth image, remove the ground point cloud in the detection box as background to obtain the target positioning result.

4. The mobile robot target locating and tracking method of claim 1, wherein, Calculating the score of each trajectory and selecting the trajectory with the highest score as the generated motion trajectory of the target comprises heuristic scoring based on the grid map and target distance-based scoring, wherein the heuristic scoring does not select trajectories that occupy areas, but selects free areas in the grid map along the planned trajectory path, and the target distance-based scoring directly calculates the Euclidean distance between the end point of the candidate trajectory and the target.

5. A mobile robot target positioning, tracking apparatus, characterized by, Comprising a processor and a memory, the memory storing a computer program, the computer program being executed by the processor to realize the steps of the mobile robot target positioning and tracking method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program being executed by the processor to realize the steps of the method according to any one of claims 1 to 4.

7. A robot, characterized in that A mobile robot target location, tracking device as claimed in claim 5 or a computer readable storage medium as claimed in claim 6.

Images