Autonomous mobile robot multi-target tracking method and system based on dynamic adaptive motion model

Abstract

The invention belongs to the technical field of intelligent vehicles, and discloses an autonomous mobile robot multi-target tracking method and system based on a dynamic adaptive motion model. Detecting object information in the environment, and obtaining detection frame data of each frame of laser radar and camera; laser and camera data are projected to the ground to calculate the Euclidean distance so as to realize multi-sensor fusion and different frame data association operation. A corresponding adaptive state transition matrix is set for a previous frame trajectory, and a prediction bounding box and a covariance matrix of the previous frame trajectory in a current frame are calculated by using a Kalman filtering technology. The long-term and short-term memory module stores historical information of a management target and dynamically adjusts a tracking strategy and parameters. According to the invention, the tracking effect of the multi-target tracking algorithm on the autonomous mobile robot is successfully improved, powerful technical support and guarantee are provided for the intelligent robot to realize accurate multi-target tracking and efficient operation in a complex environment, and further development of the intelligent robot technology in practical application is powerfully promoted.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent vehicles, and particularly relates to a multi-target tracking method and system for an autonomous mobile robot based on a dynamic adaptive motion model. Background Art

[0002] In the rapid development process of modern industrial logistics, the application of autonomous mobile robots is becoming increasingly widespread, becoming a key support for improving logistics efficiency and intelligent level. Their operation scenarios cover complex environments such as large warehouses, logistics parks, and production workshops. Achieving multi-target tracking in these scenarios is one of the core technologies to ensure the efficient operation of the robot.

[0003] Multi-target tracking aims to accurately identify and continuously track multiple moving targets in the environment. It mainly obtains target information by fusing various sensor data and uses specific algorithms to maintain the accuracy of target identity and trajectory. Among them, the motion model plays a key role in predicting the target position. Common methods such as the Kalman filter based on certain assumptions are widely used. However, in industrial logistics scenarios, there are many complex factors that pose challenges to traditional tracking algorithms.

[0004] From the perspective of the robot's own characteristics, industrial logistics robots usually carry goods of different weights when performing tasks, and their load changes frequently and significantly. This load change causes obvious vibrations in the robot chassis, which are then transmitted to the sensors installed on it. Especially key sensors such as cameras and lidar, the vibrations cause deviations and noises in the data they collect. For example, the images captured by the camera will be blurred and displaced due to jitter, seriously interfering with the accuracy of target positioning and tracking, resulting in traditional algorithms being difficult to accurately determine the true position of the target.

[0005] From the perspective of the operation environment analysis, the target motion patterns in industrial logistics scenarios are extremely complex and diverse. In the warehouse, forklifts shuttle quickly between the shelves, frequently performing operations such as accelerating, decelerating, and turning; workers walk and carry goods in different areas, and their movement routes and speeds change irregularly; at the same time, there are other mobile robots moving according to their respective task plans. The dynamic characteristics of these targets make traditional tracking algorithms based on simple motion assumptions (such as constant speed or fixed acceleration models) unable to effectively adapt, resulting in a large deviation between the predicted trajectory and the actual motion trajectory, and the tracking effect is greatly reduced.

[0006] In addition, in some specific areas of logistics operations, such as cargo loading and unloading areas, sorting centers, etc., the targets are densely distributed. A large number of targets move in close proximity and cross each other within a limited space, resulting in frequent occlusions between targets. When traditional multi-target tracking algorithms handle such situations, they often tend to have problems with target identity confusion. For example, when multiple workers or forklifts approach and cross each other in a short period of time, the algorithm may incorrectly associate the target detection results, assigning the trajectory originally belonging to one target to another target, thus causing chaos in subsequent logistics operations, reducing the overall operation efficiency, and even potentially causing damage to goods or safety accidents.

[0007] In summary, the industrial logistics scenario poses extremely high requirements for the multi-target tracking technology of autonomous mobile robots. Existing traditional algorithms have obvious limitations when facing these complex situations. Therefore, it is of extremely important practical significance and urgency to develop a new multi-target tracking algorithm that can effectively overcome the above problems and adapt to the industrial logistics environment, which has become a key technological breakthrough point for promoting the intelligent development of industrial logistics. Summary of the Invention

[0008] The present invention proposes a multi-target tracking method for autonomous mobile robots based on a dynamic adaptive motion model, aiming to overcome the severe challenges faced by multi-target tracking in the industrial logistics scenario.

[0009] The technical solution of the present invention is as follows: A multi-target tracking method for autonomous mobile robots based on a dynamic adaptive motion model specifically includes the following steps:

[0010] Step 1: Obtain target detection information at the image level through a 2D detection box and obtain detection information at the laser level through a 3D detection box;

[0011] Step 2: Construct a 2D information data association matrix and a 3D information data association matrix respectively; the 2D information association matrix is constructed based on the Euclidean distance of the midpoint at the bottom of the 2D detection box projected onto the ground; the 3D information association matrix is constructed based on the center point of the 3D detection box projected onto the ground and considering the speed factor; fuse the 2D information data association matrix and the 3D information data association matrix by setting weights to obtain the final object association matrix; use the Hungarian matching algorithm to calculate the final object association matrix to obtain the object matching result; record the time when the target obtains the detection information of the previous frame, and delete the corresponding target if the object matching result exceeds the threshold;

[0012] Step 3: Incorporate the object's motion state into the Kalman filter dynamic algorithm. Dynamically adjust the transfer matrix F parameter of the motion model in the Kalman filter state prediction equation according to the target motion state, and update the speed and acceleration in the transfer matrix based on the motion states of the target in the previous and current frames. At the same time, introduce the noise matrix Q of the Kalman filter error covariance prediction equation, and its parameters are set according to dynamic probability and camera motion compensation requirements, including the process compensation factor along the x-axis and the process compensation factor along the y-axis, to accurately simulate camera motion noise and ensure the accurate update of the target motion model during the tracking process;

[0013] Step 4: Manage the long-term and short-term trajectory memories of the matched objects; and optimize the Kalman filter state prediction equation of the matched objects, and adjust and supplement the motion model formula.

[0014] The specific steps of Step 1 are as follows:

[0015] Step 11: Convert the laser information obtained by the laser sensor into a laser frame, and convert the image obtained by the camera sensor into an image frame;

[0016] Step 12: The detector obtains the 2D detection box information of all image frames, including the coordinates and confidence levels of the 2D detection boxes; the detector obtains the 3D detection box information of each target in the laser frame, including the coordinates and confidence levels of the 3D detection boxes.

[0017] The specific steps of Step 2 are as follows:

[0018] Step 21: Obtain the laser extrinsic matrix, camera intrinsic matrix, and camera extrinsic matrix;

[0019] Step 22: Use the bottom center point of the 2D detection box as the projection point, and project it onto the ground through the camera intrinsic matrix and camera extrinsic matrix;

[0020] Step 23: Use the center point of the 3D detection box as the projection point, fix the Z-axis as the ground height, and project it onto the ground through the laser extrinsic matrix;

[0021] Step 24: Calculate the Euclidean distance between the 2D projection point and the 3D projection point of the current frame to fuse multi-sensor data; combine the Euclidean distances of the 2D projection points across frames to match and associate the image trajectory with the detection result, and use the Euclidean distance of the 3D projection points to match and associate the point cloud trajectory with the detection result to achieve cross-frame target tracking and obtain the object matching result of the final object association matrix of the current frame.

[0022] The specific steps of Step 3 are as follows:

[0023] Step 31: Calculate the dynamic target motion state of the objects that have been successfully matched multiple times according to the threshold setting;

[0024] Step 32: Dynamically set the transfer matrix F parameter of the adjustment motion model for each trajectory according to the target motion state;

[0025] Step 33: Introduce the noise matrix Q to simulate the camera motion noise and perform compensation;

[0026] Step 34: Calibrate to obtain a predicted bounding box and covariance matrix that better fit the true position of the target.

[0027] The above Step 31 sets the corresponding motion state for each object, and the specific operations are as follows:

[0028] Step 311: Set the trajectory association times threshold T assoc , and screen for stable targets that meet the target successful association times N track ≥T assoc ;

[0029] Step 312: Calculate the target instantaneous velocity and acceleration from the historical frame sequence, where Δx is the adjacent displacement;

[0030] Step 313: Calculate the formula M = (1 + α0v t )Δt and where α0 is the velocity scale factor and α1 is the acceleration scale factor; the default values are α0 = α1 = 0.5 and are adjusted based on different scene types.

[0031] Step 314: Establish a state mapping rule: if v t ≤∈ v and a t ≤∈ a , then mark it as the stationary state; if a t >∈ a , mark it as the accelerating state; otherwise, it is the uniform velocity state; ∈ v , ∈ a is the preset noise tolerance threshold.

[0032] The above Step 32 dynamically sets the transfer matrix F parameter of the adjustment motion model for each trajectory according to the target motion state for each object; the specific operations are as follows:

[0033] Step 321: Construct a 6D transfer matrix F, whose non-zero elements are related to the state variables (x, y, v x , v y , a x , a y );

[0034] The F matrix is as follows:

[0035]

[0036] Step 322: Dynamically adjust the matrix parameters according to the status flag in Step 314:

[0037] Static state: Let M = 0, N = 0, degenerate to F = diag(1, 1, 0, 0, 0, 0), and only retain position prediction;

[0038] Accelerating state: According to M = α0v t Δt, Update, and the matrix retains the acceleration term;

[0039] Uniform velocity state: Fix M = vΔt, N = 0, and simplify to a constant velocity model;

[0040] Step 323: Perform matrix conditional update: If the states are consistent for K consecutive frames, lock the current F parameter until the state jumps, where K is the model switching delay coefficient.

[0041] Step 33 introduces the noise matrix Q to simulate the camera motion noise for compensation, which specifically includes the following operations:

[0042] Step 331: Calculate the motion noise intensity based on the camera sensor data, and dynamically calculate the x-axis compensation factor σ x and the y-axis compensation factor σ y ;

[0043] Step 332: Construct the noise transfer matrix G t , whose elements include the time square term and the linear term Δt, reflecting the cumulative effect of noise over time;

[0044]

[0045] Step 333: Generate the process noise matrix through and dynamically inject it into the Kalman filter error covariance prediction equation;

[0046] Step 334: Design a noise suppression strategy: If the camera is detected to be stationary, reduce the Q t weight to avoid overfitting.

[0047] Step 4 is as follows:

[0048] Step 41: When the target is occluded or the number of consecutive unmatched frames exceeds the threshold, trigger trajectory deletion and store it in the cache queue. The deleted trajectory retains 10 - 20 seconds of historical data, and the historical data includes position, velocity, and motion model parameters;

[0049] Step 42: Maintain the last motion parameters of the target unchanged during occlusion;

[0050] Step 43: Compare the new detected target with the cached trajectory: If the predicted position deviation Δ ≤ threshold, restore the original ID;

[0051] Step 44: Failure handling, when the deviation is too large: Automatically reset the motion model or destroy the cache.

[0052] An autonomous mobile robot multi-target tracking system based on a dynamic adaptive motion model, comprising:

[0053] A detection module, used to generate 2D detection boxes, obtain object detection information at the image level, generate 3D detection boxes, and obtain detection information at the laser level;

[0054] A data association module, used for: respectively constructing a 2D information data association matrix and a 3D information data association matrix; the 2D information association matrix is constructed based on the Euclidean distance of the midpoint at the bottom of the 2D detection box projected onto the ground, and the 3D information association matrix is constructed based on the center point of the 3D detection box projected onto the ground and considering the speed factor; obtaining the final object association matrix through specific weight fusion, and calculating the final association matrix using the Hungarian matching algorithm; recording the time when the target obtains the detection information of the previous frame, and deleting the corresponding target in the long-term and short-term memory modules if it exceeds the threshold, to achieve long-term and short-term tracking control;

[0055] A trajectory prediction module, used for: integrating the object motion state into the Kalman filter dynamic algorithm, dynamically adjusting the transfer matrix F parameter of the motion model in the Kalman filter state prediction equation according to the target motion state, where the parameters involving speed and acceleration are updated according to the previous frame state of the target and the calculated dynamic probability; at the same time, introducing a noise matrix Q, the parameters of which are set according to the vehicle dynamic probability and the camera motion compensation requirements, including the process compensation factors along the x and y axes, to accurately simulate the camera motion noise and ensure the accurate update of the target motion model during the tracking process;

[0056] A long-term and short-term memory module, used for optimizing, adjusting, and supplementing the formulas related to the motion model based on the Kalman filter state prediction matrix, enhancing the accuracy of target state prediction, ensuring the continuity and stability of tracking, and reducing the risk of tracking loss.

[0057] Compared with the prior art, the multi-target tracking method for an autonomous mobile robot based on a dynamic adaptive motion model of the present invention has the following remarkable beneficial effects:

[0058] Excellent motion adaptability: The present invention abandons the traditional simple motion assumption model and adopts a dynamic adaptive motion model. By analyzing the state and dynamic probability of the previous frame of the target in real time, the speed scale factor and acceleration scale factor are accurately calculated, and the transition matrix F parameter of the motion model is dynamically updated. Whether it is the sudden turning or sudden stop of a forklift, the irregular variable-speed walking of a worker, or the complex path switching of a mobile robot, it can quickly adapt to the target motion changes and accurately predict its position. In contrast, when faced with such complex motions, traditional algorithms have large prediction deviations and poor tracking effects due to rigid models, making it difficult to meet the requirements of industrial logistics scenarios for dynamic target tracking.

[0059] Powerful anti-interference performance: Aiming at the vibration interference and camera motion noise of industrial logistics robots, the present invention introduces a carefully designed noise matrix Q and a compensation mechanism. The intensity of motion noise is accurately estimated using camera IMU data and optical flow features, the x / y axis compensation factors are calculated, a noise transfer matrix is constructed, and a process noise matrix is generated and injected into the Kalman prediction covariance. This effectively cancels out the noise interference caused by the chassis vibration due to the change in the robot load and the camera's own motion, ensuring accurate target positioning and stable tracking. In contrast, existing technologies lack targeted measures and are prone to target positioning drift and tracking loss in a vibrating environment, seriously affecting the accuracy of logistics operations.

[0060] Efficient dense target processing ability: The data association module and the long-term and short-term memory modules of the present invention work together to effectively solve the problem of identity confusion of dense targets. The data association module constructs and fuses 2D and 3D information association matrices through multi-sensor fusion technology, and combines the Hungarian matching algorithm to achieve accurate data association; when targets are occluded or interleaved, the long-term and short-term memory modules use a cache queue and an intelligent trajectory management strategy to maintain the target identity information and trajectory continuity. Traditional algorithms are limited by feature extraction and association methods and are prone to incorrect associations in dense target scenarios, interfering with the logistics operation process and reducing the operation efficiency.

[0061] Significant overall efficiency improvement: The present invention adopts an efficient data processing and intelligent target screening mechanism. The Hungarian matching algorithm of the data association module quickly matches data, and the trajectory deletion and caching strategies of the long-term and short-term memory modules avoid the accumulation of invalid data and reduce the waste of computing resources. The algorithm runs fast and has strong real-time performance, meeting the requirements of industrial logistics scenarios for the rapid response of robots. Existing technologies are prone to computing bottlenecks when processing a large amount of target data, resulting in tracking delays and being unable to adapt to the high efficiency requirements of logistics operations, affecting industrial production and distribution processes.

[0062] In summary, the multi-target tracking algorithm for autonomous mobile robots based on a dynamic adaptive motion model of the present invention has prominent advantages in industrial logistics scenarios, can effectively improve the operation performance of robots, and strongly promotes the intelligent development of industrial logistics. Brief Description of the Drawings

[0063] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. Among them:

[0064] Figure 1 It is a schematic flowchart of the multi-target tracking method for an autonomous mobile robot based on a dynamic adaptive motion model according to the embodiment of the present invention;

[0065] Figure 2 It is a schematic flowchart of the data association method for the multi-target tracking method of an autonomous mobile robot based on a dynamic adaptive motion model according to the embodiment of the present invention;

[0066] Figure 3 It is a schematic flowchart of updating the motion model of an autonomous mobile robot based on a dynamic adaptive motion model according to the embodiment of the present invention

[0067] Figure 4 It is a schematic flowchart of the long-term and short-term memory modules of an autonomous mobile robot based on a dynamic adaptive motion model according to the embodiment of the present invention Detailed Embodiments

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0069] Through innovative design, the tracking accuracy, stability and reliability are greatly improved, providing a solid technical support for the efficient and accurate operation of autonomous mobile robots in complex logistics operation environments.

[0070] The core system of the present invention mainly consists of a detection module, a data association module, a trajectory prediction module, and a long-term and short-term memory module.

[0071] Each module closely interacts and operates collaboratively to jointly achieve the precise multi-target tracking function, forming an organic whole.

[0072] The detection module is the front-end perception unit of the tracking system, undertaking the key information collection task.

[0073] At the image level, advanced 2D detection box technology and deep learning models, such as convolutional neural networks (CNNs), are used to deeply analyze and accurately extract features such as the contour, position, and texture of the target, and to determine the two-dimensional coordinate position and detection confidence of the target in the image.

[0074] At the laser level, 3D detection box technology is combined with lidar point cloud data processing algorithms to collect three-dimensional spatial information of the target.

[0075] In actual operation, the original sensor signals are first processed through filtering, amplification, analog-to-digital conversion, etc., and converted into laser frame and image frame formats.

[0076] Then, with the help of high-precision detectors, such as advanced models like YOLO or Faster R-CNN, the frame data is deeply scanned to obtain accurate detection box information, providing a data foundation for subsequent processes.

[0077] The data association module is at the core hub position of the system.

[0078] When constructing the 2D information association matrix, the midpoint at the bottom of the 2D detection box is selected as the projection point, and projected onto the ground coordinate system through the camera's internal and external parameter matrices. The process involves operations such as matrix multiplication.

[0079] When constructing the 3D information association matrix, the center point of the 3D detection box (fixing the Z-axis as the ground height) is used as the projection point, projected onto the ground with the help of the laser external parameter matrix, and the speed factor is incorporated.

[0080] After that, the 2D and 3D matrices are fused through reasonable weight distribution to obtain the final association matrix, and the Hungarian matching algorithm is used to achieve accurate data association.

[0081] This module can also record the time when the target obtained the detection information in the previous frame. When the threshold is exceeded, the corresponding target is deleted in the long-term and short-term memory modules to avoid the accumulation of invalid data and improve the operation efficiency.

[0082] The trajectory prediction module is the innovation core of the present invention.

[0083] First, the motion state of the objects that have been associated multiple times is analyzed according to the threshold, and the instantaneous speed of the target is calculated through the historical frame sequence and the acceleration

[0084] Then, according to the state and dynamic probability of the target in the previous frame, according to the formula M=(1 + α0v t )Δt and the speed scale factor α0 and the acceleration scale factor α1 are generated (the default values are α0 = α1 = 0.5, which can be adjusted according to the scenario), and the transition matrix F is dynamically updated.

[0085] For example, when in the acceleration state, M = α0vt Δt, In the stationary state, M = 0, N = 0; in the uniform motion state, M = vΔt, N = 0.

[0086] Meanwhile, a noise matrix Q is introduced. By analyzing the camera IMU data and optical flow features, the motion noise intensity is estimated, and the compensation factors σ x and σ y on the x / y axes are calculated to construct a noise transfer matrix Through the process noise matrix is generated and injected into the Kalman prediction covariance to compensate for interference factors and improve the tracking accuracy.

[0087] The long-term and short-term memory modules play a key role in stabilizing and optimizing the tracking process.

[0088] Based on the Kalman filter prediction matrix, optimization algorithms and machine learning techniques are used to adjust the relevant formulas of the motion model, such as the transition matrix F, the noise matrix Q, etc.

[0089] When the target is occluded or the number of consecutive unmatched frames exceeds the threshold, the long-term and short-term memory management mechanism is triggered.

[0090] The deleted trajectories are stored in the cache queue and 10 - 20 seconds of historical data are retained.

[0091] During occlusion, the last motion parameters of the target are maintained unchanged. The newly detected target is compared with the cached trajectory. If the deviation is within the threshold, the original ID is restored; if the deviation is too large, the motion model is reset or the cache is destroyed to ensure continuous and stable tracking and reduce the risk of loss.

[0092] Each module works in coordination, enabling the algorithm of the present invention to effectively handle the complex situations in industrial logistics, overcome the defects of traditional algorithms, provide an efficient, accurate and stable solution for multi-target tracking of autonomous mobile robots, promote the intelligent development of industrial logistics, and improve the overall efficiency and quality.

[0093] In the implementation process of the present invention, first, the sensor system of the robot needs to be finely calibrated and configured to ensure that the laser sensor and the camera sensor can stably and accurately collect environmental data. Meanwhile, the relevant algorithm modules are integrated into the control system of the robot to build a complete multi-target tracking system architecture.

[0094] During actual operation, the detection module continuously receives data from sensors, quickly completes data preprocessing and format conversion, and extracts key information of the target using advanced target detection technology. The data association module then performs in-depth fusion and correlation analysis on this information, and based on carefully designed algorithms and parameters, establishes accurate correspondence relationships between targets. The trajectory prediction module, by leveraging historical data and real-time dynamic information, uses a dynamic adaptive motion model and Kalman filtering technology to perform high-precision prediction of the target's motion trajectory, and continuously adjusts and optimizes it according to new data feedback. The long-term and short-term memory module always monitors the entire tracking process, makes intelligent judgments and processes on the state changes of the target, effectively handles complex situations such as occlusion, target loss, and reappearance, and ensures the continuity and accuracy of tracking.

[0095] Through such a comprehensive and closely coordinated implementation process, the multi-target tracking algorithm of the present invention can fully exert its advantages in industrial logistics scenarios, significantly improve the multi-target tracking performance of autonomous mobile robots, provide a solid guarantee for the efficient and intelligent development of logistics operations, and strongly promote the advancement of the industrial logistics industry towards the intelligent direction.

[0096] Embodiment

[0097] Furthermore, in another embodiment, different types of industrial logistics scenarios were tested and optimized. Considering the layout characteristics of forklifts, shelves, and staff in the warehouse environment, an enhanced recognition function for the external shape features of forklifts was added to the detection module of the algorithm. Using the fine-tuning technology of the target detection model in deep learning, by collecting a large amount of forklift image data for model training, the detection module can more accurately identify different parts and postures of forklifts, thereby improving the detection accuracy of forklift targets in complex warehouse backgrounds.

[0098] In the data association module, to more accurately restore the trajectory and identity information of the target in the case of occlusion caused by shelves, the tracking errors caused by occlusion are effectively reduced.

[0099] In the trajectory prediction module, according to the common movement paths and speed ranges of forklifts and staff in the warehouse, multiple groups of different movement mode templates were preset. During the actual tracking process, by quickly judging the initial movement state of the target, the most matching template is selected to initialize the movement model, which speeds up the convergence speed of the algorithm, improves the timeliness and accuracy of trajectory prediction, and further enhances the performance of the entire multi-target tracking algorithm in the warehouse logistics scenario.

[0100] In another embodiment, for the scenario of multi-robot collaborative operation and frequent entry and exit of cargo transport vehicles in a logistics park, a distributed architecture of multi-sensor fusion is adopted in the detection module to achieve omni-directional and multi-level perception of different types of targets. By designing an efficient data fusion algorithm, the data of two sensors are deeply fused, improving the reliability and stability of detection.

[0101] In the data association module, considering the large number of targets and complex movement trajectories in the logistics park, the Hungarian algorithm is improved. A heuristic search strategy based on target priority and movement trend is introduced. When performing data association, key targets (such as vehicles transporting important goods) and targets with obvious movement trends are processed first, reducing the computational complexity and the possibility of incorrect association, and improving the efficiency and accuracy of data association.

[0102] In the trajectory prediction module, map information is combined. By analyzing factors such as the position of the target in the park, surrounding roads and obstacles, and traffic flow, a more reasonable prediction and planning of the target's movement trajectory are carried out, effectively avoiding collisions and conflicts between targets, and enhancing the safety and smoothness of logistics operations in the logistics park.

[0103] Refer to Figures 1-3 , the embodiments of the present invention include the following steps:

[0104] Step 1: The detection module operates to synchronize the lidar point cloud and image data. Since the sampling frequencies and timestamps of the lidar and camera may be different, it is necessary to ensure the time alignment of the data through hardware or software synchronization methods, and then generate lidar frames and image frames respectively.

[0105] Perform dual-modal detection on each frame. For the image frame, input it into the trained YOLOv5 model for 2D object detection, and output the coordinates and confidence levels of the detection boxes in the image; for the lidar frame, input the lidar point cloud data into the trained PointPillars model for 3D object detection, and output the position, size, and confidence level of the detection boxes of the targets in the three-dimensional space. Save the 2D detection box set as Save the 3D detection box set as and save the coordinate and confidence information for subsequent association.

[0106] Step 2: The data association module operates to load the camera internal parameter K and lidar external parameter from the pre-stored calibration file to ensure the accuracy of the parameters and prepare for subsequent coordinate transformation and data association.

[0107] Calculate the 2D and 3D projection distance matrices. For the 2D projection distance matrix D 2D , take The center point at the bottom of the 2D detection box is projected onto the ground coordinate system through the internal and external parameters of the camera to obtain the projection point proj 2D (i), calculate the predicted position μ of the tracking trajectory in the ground coordinate system track (j), according to the formula D 2D (i,j) = ‖proj 2D (i) - μ track (j)‖² to calculate the element; for the 3D projection distance matrix D 3D , take The center point of the 3D detection box in is mapped to the world coordinate system through the external parameters of the laser to obtain the projection point proj 3D (i), calculate the predicted position, predicted velocity of the tracking trajectory in the world coordinate system and the velocity of the 3D detection box, according to the formula D 3D (i,j) = 0.7‖proj 3d (i) - μ track (j)‖² + 0.3‖v det (i) - v track (j)‖ to calculate the element.

[0108] Perform weighted fusion matching. According to D final = 0.6D 2D + 0.4D 3D Fuse the 2D and 3D association matrices to obtain the final association matrix D final , use the Hungarian algorithm to match D final to make the detection box correspond to the tracking trajectory and achieve cross-modal data association.

[0109] Step 3: Design the state transition matrix for the trajectory prediction module operation where Dynamically adjust α0 and α1 according to the target motion state (stationary, uniform motion, acceleration). Set α0 = 0 and α1 = 0 when stationary; set α0 = 1 and α1 = 0 when in uniform motion; adjust according to the acceleration magnitude when accelerating to improve the accuracy of target motion prediction.

[0110] According to the formula Generate the noise matrix, where Obtain the camera acceleration and angular velocity through IMU data, and calculate the compensation factor σ through processing and analysis x , σ y , adjust the noise matrix Q t to compensate for the camera motion noise interference.

[0111] Apply the dynamic Kalman filter to predict and update the target state, combined with the dynamic state transition matrix F k and the noise matrix Q considering camera motion compensation k to improve the accuracy of trajectory prediction.

[0112] Step 4: Long-term and short-term memory modules The long-term and short-term memory modules operate to maintain the last known motion parameters v when the target is short-term lost. last ,a last Make predictions to maintain tracking continuity when the target is temporarily occluded or the detection fails.

[0113] When the target is long-term lost, store its historical trajectory information in the cache queue, retaining historical data for T cacha ∈ [10s, 20s]. When a newly detected target appears, calculate the position deviation Δ between it and the cached trajectory. If Δ ≤ ∈, resume tracking with the original ID; if the deviation is too large, initiate a failure handling mechanism, such as resetting the motion model or destroying the cache, to avoid the impact of incorrect data on subsequent tracking.

[0114] To verify the feasibility and effectiveness of the present invention, the present invention is verified on the training set of KITTI to obtain preliminary results. In terms of multi-object tracking accuracy (HOTA, MOTA), the present invention outperforms the existing optimal methods, demonstrating the improved robustness of the algorithm to target occlusion and deformation. While maintaining the leading accuracy, the frame rate (FPS) is increased to 1105, significantly better than the existing methods, meeting the requirements of high real-time scenarios such as the driving of mobile robots.

[0115]

[0116]

[0117] The present invention discloses a multi-object tracking algorithm for mobile robots based on a motion model, belonging to the technical field of autonomous mobile robots, mainly including an initialization module, a prediction module, and a trajectory association module.

[0118] The detection module is mainly responsible for generating 2D detection frames to obtain target detection information at the image level, and at the same time generating 3D detection frames to obtain detection information at the laser level. Specifically, first convert the laser information obtained by the laser sensor into a laser frame, and convert the image obtained by the camera sensor into an image frame. Then, with the help of the detector, obtain the 2D detection frame coordinates of the image frame and the 3D detection frame coordinates of the laser frame.

[0119] The data association module constructs 2D and 3D information data association matrices respectively, and uses the Hungarian matching algorithm to achieve cross-frame target tracking after weight fusion. For the 2D association matrix, it is constructed based on the Euclidean distance from the midpoint of the bottom of the 2D detection box projected onto the ground; the 3D association matrix is constructed based on the projection of the center point of the 3D detection box onto the ground and combined with the speed factor. In this process, the projection coordinate transformation is realized through the laser extrinsic matrix, camera intrinsic and extrinsic matrices, the Euclidean distance between the 2D / 3D projection points of the current frame is calculated, and multi-sensor data is fused for cross-frame association. In addition, the detection time of the target in the previous frame is recorded, and if it exceeds the threshold, the target will be deleted in the long-term / short-term memory module.

[0120] The trajectory prediction module adopts the dynamic adaptive Kalman filtering algorithm and adjusts the model parameters in real time in combination with the target motion state. First, stable targets are selected according to the trajectory association times threshold, and their instantaneous speed and acceleration are calculated. Then, a 6D transition matrix F is constructed, and the matrix parameters are dynamically adjusted according to the motion state (static state, accelerating state, uniform motion state). When in the static state, the transition matrix degenerates into a diagonal matrix for only position prediction; when in the accelerating state, the acceleration term is retained; when in the uniform motion state, it is simplified to a constant speed model. At the same time, a noise matrix Q is introduced to compensate for the camera motion noise, which includes x / y axis process compensation factors. Finally, the predicted bounding box and covariance matrix are generated through Kalman filtering, and corrected in combination with camera motion compensation. In the autonomous driving scenario, the mobile robot needs to operate in a complex road environment. For example, on urban streets, it may face various traffic participants, such as pedestrians, vehicles, etc., whose motion patterns are complex and changeable. At the same time, the camera will also move due to the movement of the robot, and the fixed state transition matrix of the traditional Kalman filter is difficult to accurately perform state transition on the target, affecting the tracking effect.

[0121] The long-term and short-term memory modules optimize the motion model based on Kalman filtering to enhance tracking stability. When the trajectory is occluded or the unmatched exceeds the threshold, it will be stored in the cache queue, and the historical data for 10 - 20 seconds will be retained. During the occlusion period, the last motion parameters are maintained unchanged. For newly detected targets, they will be compared with the cached trajectories. If the predicted position deviation Δ ≤ threshold, the original ID will be restored; if the deviation is too large, the motion model will be automatically reset or the cache will be destroyed.

[0122] In the scenario of logistics industrial robots, the environment in the warehouse is also relatively complex. The motion conditions of targets such as goods and staff are diverse, and the motion state of the robot itself will also change frequently during the process of carrying goods, which will also cause the camera to face motion problems and affect the accuracy of target tracking.

[0123] The present invention effectively solves the above problems by combining Kalman filtering with an adaptive state transition matrix and camera motion compensation. The verification results on the KITTI dataset show that the algorithm can achieve accurate multi-object tracking in complex scenarios, not only improving the tracking effect of mobile robots in the autonomous driving scenario, but also demonstrating the reliability of the algorithm in the logistics industrial robot scenario, and can meet the multi-object tracking requirements of mobile robots in different scenarios.

[0124] The present invention has been described above in conjunction with relevant content. Although the present invention has been presented above in a preferred embodiment, it is not intended to limit the present invention. Anyone familiar with this technology can make various modifications and decorations without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention should be defined by the claims.

Claims

1. A multi-target tracking method for an autonomous mobile robot based on a dynamic adaptive motion model, characterized in that, Specifically, it includes the following steps: Step 1: Obtain object detection information at the image level through 2D detection boxes, and obtain detection information at the laser level through 3D detection boxes; Step 2: Construct a 2D information data association matrix and a 3D information data association matrix respectively; the 2D information association matrix is constructed based on the Euclidean distance from the midpoint at the bottom of the 2D detection box projected onto the ground; the 3D information association matrix is constructed based on the projection of the center point of the 3D detection box onto the ground and considering the speed factor; Fuse the 2D information data association matrix and the 3D information data association matrix by setting weights to obtain the final object association matrix; use the Hungarian matching algorithm to calculate the final object association matrix to obtain the object matching result; Record the time when the target obtained the detection information in the previous frame, and delete the corresponding target if the object matching result exceeds the threshold; Step 3: Incorporate the object motion state into the Kalman filter dynamic algorithm, dynamically adjust the transfer matrix F parameter of the motion model in the Kalman filter state prediction equation according to the target motion state, and update the speed and acceleration in the transfer matrix according to the motion states of the previous and current frames of the target; at the same time, introduce the noise matrix Q of the Kalman filter error covariance prediction equation, and its parameters are set according to the dynamic probability and the requirements of camera motion compensation, including the process compensation factor along the x-axis and the process compensation factor along the y-axis, to accurately simulate the camera motion noise and ensure the accurate update of the target motion model during the tracking process; Step 4: Perform long-term and short-term trajectory memory management on the already matched objects; and optimize the Kalman filter state prediction equation of the already matched objects, and adjust and supplement the motion model formula.

2. The multi-object tracking method for an autonomous mobile robot based on a dynamic adaptive motion model according to claim 1, wherein The specific steps of Step 1 include the following sub-steps: Step 11: Convert the laser information obtained by the laser sensor into a laser frame, and convert the image obtained by the camera sensor into an image frame; Step 12: The detector obtains the 2D detection box information of all image frames, including the coordinates and confidence levels of the 2D detection boxes; the detector obtains the 3D detection box information of each target in the laser frame, including the coordinates and confidence levels of the 3D detection boxes.

3. The multi-object tracking method for an autonomous mobile robot based on a dynamic adaptive motion model according to claim 1, wherein The specific steps in Step 2 include the following sub-steps: Step 21: Obtain the laser extrinsic matrix, the camera intrinsic matrix, and the camera extrinsic matrix; Step 22: Use the center point at the bottom of the 2D detection box as the projection point, and project it onto the ground through the camera intrinsic matrix and the camera extrinsic matrix; Step 23: Use the center point of the 3D detection box as the projection point, fix the Z-axis as the ground height, and project it onto the ground through the laser extrinsic matrix; Step 24: By calculating the Euclidean distance between the 2D projection point and the 3D projection point in the current frame, fuse the multi-sensor data; combine the Euclidean distances of the 2D projection points across frames to match the associated image trajectory with the detection result, and the Euclidean distances of the 3D projection points to match the associated point cloud trajectory with the detection result, realize cross-frame target tracking, and obtain the object matching result of the final object association matrix in the current frame.

4. The multi-target tracking method for an autonomous mobile robot based on a dynamic adaptive motion model according to claim 1, characterized in that The specific steps of Step 3 include the following sub-steps: Step 31: Calculate the dynamic state of the target motion for the objects that have been successfully matched multiple times according to the threshold setting; Step 32: Set the transfer matrix F parameter for adjusting the motion model for each trajectory according to the dynamic state of the target motion; Step 33: Introduce the noise matrix Q to simulate the camera motion noise and perform compensation. Step 34: Calibrate to obtain a predicted bounding box and covariance matrix that better fit the true position of the target.

5. The multi-target tracking method for an autonomous mobile robot based on a dynamic adaptive motion model according to claim 4, characterized in that, Step 31 sets the corresponding motion state for each object, which specifically includes the following operations: Step 311: Set the threshold T for the number of trajectory associations assoc , and filter out stable targets that meet the target successful association times N track ≥T assoc ; Step 312: Calculate the target instantaneous velocity from the historical frame sequence and the acceleration where Δx is the adjacent displacement; Step 313: Calculate the formula M = (1 + α0v t )Δt and α0 is the speed proportionality factor, and α1 is the acceleration proportionality factor; Step 314: Establish a state mapping rule: If v t ≤∈ v and a t ≤∈ a then it is marked as the stationary state; if a t >∈ a it is marked as the accelerating state; otherwise it is the uniform state; ∈ v ,∈ a is the preset noise tolerance threshold.

6. The multi-target tracking method for an autonomous mobile robot based on a dynamic adaptive motion model according to claim 5, wherein Step 32 dynamically sets the transition matrix F parameter for adjusting the motion model for each trajectory according to the target motion state for each object. Specifically, it includes the following operations: Step 321: Construct a 6D transition matrix F, whose non-zero elements are associated with state variables (x, y, v x , v y , a x , a y ); The F matrix is as follows: Step 322: Dynamically adjust the matrix parameters according to the state flag in Step 314: Stationary state: Let M = 0, N = 0, and degrade to F = diag(1, 1, 0, 0, 0, 0), only retaining position prediction. Accelerated state: According to M = α0v t Δt, Update, and the matrix retains the acceleration term; Uniform velocity state: Fix M = vΔt, N = 0, and simplify to a constant velocity model. Step 323: Execute matrix condition update: If the states are consistent for K consecutive frames, lock the current F parameter until the state changes. K is the model switching delay coefficient.

7. The multi-object tracking method for an autonomous mobile robot based on a dynamic adaptive motion model according to claim 4, characterized in that Step 33 introduces the noise matrix Q to simulate the camera motion noise and perform compensation, which specifically includes the following operations: Step 331: Calculate the motion noise intensity based on the camera sensor data, and dynamically calculate the x-axis compensation factor σ x and the y-axis compensation factor σ y ; Step 332: Construct the noise transfer matrix G t , whose elements contain the time squared term and the linear term Δt, reflecting the cumulative effect of noise over time; Step 333: By generate a process noise matrix and dynamically inject it into the Kalman filter error covariance prediction equation; Step 334: Design a noise suppression strategy: If the camera is detected to be stationary, reduce the Q t weight to avoid overfitting.

8. The multi-target tracking method for an autonomous mobile robot based on a dynamic adaptive motion model according to claim 1, wherein Step 4 is as follows: Step 41: When the target is occluded or the number of consecutive unmatched frames exceeds the threshold, trigger trajectory deletion and store it in the cache queue. The deleted trajectory retains 10 - 20 seconds of historical data, which includes position, velocity, and motion model parameters. Step 42: Maintain the last motion parameters of the target unchanged during occlusion. Step 43: Compare the newly detected target with the cached trajectory: If the predicted position deviation Δ ≤ threshold, restore the original ID. Step 44: Failure handling, when the deviation is too large: Automatically reset the motion model or destroy the cache.

9. An autonomous mobile robot multi-target tracking system based on a dynamic adaptive motion model, characterized in that, It includes: A detection module for generating 2D detection boxes, obtaining target detection information at the image level, generating 3D detection boxes, and obtaining detection information at the laser level. A data association module for: respectively constructing a 2D information data association matrix and a 3D information data association matrix; the 2D information association matrix is constructed based on the Euclidean distance from the midpoint at the bottom of the 2D detection box projected onto the ground, and the 3D information association matrix is constructed based on the center point of the 3D detection box projected onto the ground and considering the speed factor; obtaining the final object association matrix through specific weight fusion, and calculating the final association matrix using the Hungarian matching algorithm; recording the time when the target obtains the detection information of the previous frame, and deleting the corresponding target in the long-term and short-term memory modules if it exceeds the threshold, to achieve long-term and short-term tracking control. A trajectory prediction module for: integrating the object motion state into the Kalman filter dynamic algorithm, dynamically adjusting the transition matrix F parameter of the motion model in the Kalman filter state prediction equation according to the target motion state, where the speed and acceleration parameters are updated according to the previous frame state of the target and the calculated dynamic probability; at the same time, introducing the noise matrix Q, whose parameters are set according to the vehicle dynamic probability and camera motion compensation requirements, including the process compensation factors along the x and y axes, to accurately simulate the camera motion noise and ensure the accurate update of the target motion model during the tracking process. A long-term and short-term memory module for optimizing, adjusting, and supplementing the formulas related to the motion model based on the Kalman filter state prediction matrix, enhancing the accuracy of target state prediction, ensuring the coherence and stability of tracking, and reducing the risk of tracking loss.

Images