A multi-modal target recognition and tracking prediction method based on laser radar and binocular camera

Abstract

The application relates to a multi-modal target recognition and tracking prediction method based on a laser radar and a binocular camera, and belongs to the field of robot / unmanned system sensing recognition. The method comprises the following steps: target two-dimensional images obtained by a binocular camera and target point cloud data obtained by a laser radar are pretreated to obtain pretreated two-dimensional images and point cloud data; in step S2, two-dimensional target detection and tracking detection are carried out based on the pretreated two-dimensional images to obtain two-dimensional target detection and tracking results; in step S3, three-dimensional target detection is carried out based on the pretreated point cloud data and the two-dimensional target detection and tracking results to obtain three-dimensional detection results; and in step S4, target fusion tracking prediction is carried out based on pixel coordinates in the two-dimensional target detection and tracking results and pixel coordinates of vertexes of a target point cloud boundary in the three-dimensional detection results to obtain a target ID, a position, a category and a target point cloud boundary. The method realizes accurate target recognition and tracking.

Description

Technical Field

[0001] This invention relates to the field of perception and recognition technology for robotic / unmanned systems, and in particular to a multimodal target recognition and tracking prediction method based on lidar and binocular cameras. Background Technology

[0002] Target recognition and tracking technology is mainly used in the front-end perception module of unmanned systems. It is responsible for accurately identifying and classifying specified target objects in the environment and continuously tracking them.

[0003] Relying solely on camera image information for target identification and tracking, due to the lack of depth features, only provides the target's positional information in pixel coordinates. Although depth information in the environment can be extracted through inter-frame matching of monocular cameras, parallax of binocular cameras, or by directly introducing a depth camera, this method suffers from low accuracy at the depth scale in large scenes or scenarios with rapid movement of unmanned systems, and exhibits significant errors in the identification and localization of distant targets. Furthermore, matching-based depth extraction methods require narrowing the gap between matching frames and increasing the matching frequency in situations involving rapid movement of unmanned systems, resulting in slow operation and difficulty in meeting the real-time requirements of actual unmanned system operation.

[0004] LiDAR, as the most commonly used 3D sensor in the field of unmanned operations, is frequently applied to 3D target detection. Unlike 2D image detection, LiDAR-based 3D target detection requires processing point cloud information with fewer features and a more complex structure. Based on the performance of LiDAR, 3D target detection has higher accuracy at the depth scale in larger scenes, thus offering higher accuracy in target localization compared to 2D methods. Furthermore, because point cloud features are simpler than image features, 3D target detection can operate at a higher frequency. Currently, there are two main 3D target detection methods: one projects point cloud information onto a bird's-eye view and uses a 2D detection framework for detection; the drawback of this method is its heavy reliance on the detection and tracking accuracy of the 2D framework. The other method projects point cloud information onto regular 3D voxels for recognition and tracking; the drawback of this method is that the constructed voxels may not be optimal, and information loss can easily occur during quantization, leading to potentially inaccurate detection results. Another mainstream approach is to directly perform object detection on the original point cloud based on deep learning networks. It refines the 3D detection method in the point cloud by combining semantic features and local spatial features, and finally generates the 3D bounding box of the target directly in the original point cloud. The drawback of this learning-based method is that the original point cloud information is sparse, and the feature extraction process in the network is often not efficient enough.

[0005] The aforementioned two-dimensional and three-dimensional target detection and tracking methods have the following limitations: the lack of depth scale in a single two-dimensional target detection framework makes it difficult to adapt to the detection and localization of targets in large scenes; the mainstream projection-based three-dimensional detection methods have poor accuracy, while the three-dimensional detection methods based on raw point clouds are not efficient enough. Summary of the Invention

[0006] Based on the above analysis, the embodiments of the present invention aim to provide a multi-agent-based simulation system for steel enterprise production processes, in order to solve the technical problems of lack of depth scale in two-dimensional detection and inefficient processing of three-dimensional detection point clouds in existing methods.

[0007] The objective of this invention is mainly achieved through the following technical solutions:

[0008] This invention provides a multimodal target recognition and tracking prediction method based on lidar and binocular camera, comprising the following steps:

[0009] Step S1: Preprocess the target 2D image acquired by the binocular camera and the target point cloud data acquired by the lidar to obtain the preprocessed 2D image and point cloud data.

[0010] Step S2: Perform two-dimensional target detection and tracking based on the preprocessed two-dimensional image to obtain the two-dimensional target detection and tracking results;

[0011] Step S3: Perform three-dimensional target detection based on the preprocessed point cloud data and the two-dimensional target detection and tracking results to obtain three-dimensional detection results;

[0012] Step S4: Based on the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the target point cloud boundary in the three-dimensional detection results, perform target fusion tracking prediction to obtain the target ID, location, category, and target point cloud boundary.

[0013] Further, step S2 includes:

[0014] The preprocessed multi-frame two-dimensional images are sequentially input into a one-stage detection model for two-dimensional target detection to obtain two-dimensional target detection results; the two-dimensional target detection results include the targets included in each frame of two-dimensional images, the category of the targets, and the pixel coordinates and detection confidence of the corresponding targets;

[0015] Based on the detection confidence, the two-dimensional target detection results corresponding to each frame of image are divided into high-confidence targets, low-confidence targets, and targets between high-confidence and low-confidence.

[0016] The pixel coordinates of the high-confidence targets are input into a Kalman filter. The Kalman filter predicts the new position of the targets in the current frame based on the pixel coordinates of the historical positions of each high-confidence target, thereby obtaining the predicted detection box position of the high-confidence targets and defining an ID for each high-confidence target.

[0017] Based on the preprocessed 2D image of the current frame, obtain the true detection bounding box positions of all targets in the current frame;

[0018] Based on the real detection boxes in the current frame and the predicted detection boxes in the previous frame, and combined with the new targets detected based on the low-confidence targets, the Hungarian algorithm is used to continuously track the same target, so as to obtain the two-dimensional target detection and tracking results.

[0019] Furthermore, the first-stage target detection model employs YOLOv8 combined with NMS;

[0020] The YOLOv8 first performs initial detection on the preprocessed 2D image to obtain multiple bounding boxes and their corresponding target categories, pixel coordinates, and confidence scores.

[0021] The NMS performs post-processing on the YOLOv8 output to suppress redundant bounding boxes. By retaining the bounding box with the highest confidence and removing other bounding boxes that overlap significantly with it, a two-dimensional target detection result is obtained.

[0022] Furthermore, based on each ground truth detection box in the current frame and each predicted detection box in the previous frame, the Hungarian algorithm is used to calculate the IOU value between each ground truth detection box in the current frame and each predicted detection box in the previous frame in turn.

[0023] The target of the predicted detection box in the previous frame with the largest IOU value with the real detection box is taken as the best matching object, and the target ID of the best matching object is inherited to achieve continuous tracking and matching of the same target between frames.

[0024] If a target in the current frame cannot be found to match a target in the previous frame, it is determined that a target has been lost. In this case, all low-confidence targets in the most recent P-frames are retrieved and matched again with the target in the current frame that failed to match using the Hungarian algorithm. If the match is successful, the target in the current frame will inherit the ID of the low-confidence target, thus enabling the re-identification of the lost target.

[0025] If a matching target still cannot be found, assign a new ID to the target in the current frame and treat it as a newly appearing target.

[0026] Furthermore, based on the newly received i-frame preprocessed point cloud data and the aforementioned two-dimensional target detection and tracking results, three-dimensional target detection is performed, including:

[0027] Predefined prior geometric information for each target category;

[0028] Based on the pixel coordinates of the four intersection points of each target in the two-dimensional target detection and tracking results, calculate the length and width of the target's two-dimensional detection box in pixel coordinates;

[0029] Based on the prior geometric information and the length and width of the target's 2D detection box in pixel coordinates, the left side of the target pixel is converted into a 3D coordinate position P through similar coordinate transformation. c1 (x1,y1,z1); with P c1 Using (x1, y1, z1) as the center coordinates, and based on the preset length, width, and height according to the number of targets in the detection area and the scene complexity, a 3D bounding box Box1 is generated;

[0030] Traverse all points in the current frame's point cloud. If a point is located within the bounding box Box1, it is classified as a point to be detected.

[0031] Euclidean clustering is performed on all the points to be detected in the bounding box Box1 to obtain multiple point cloud clusters. The point cloud cluster with the highest confidence is selected from the multiple point cloud clusters. The boundary of the selected point cloud cluster is used as the reference, and the length, width and height of the prior geometric information of the target category are used to construct the three-dimensional bounding box Box2 to obtain the three-dimensional detection result.

[0032] The three-dimensional detection result is a three-dimensional bounding box (Box2).

[0033] Furthermore, the step of selecting the point cloud cluster with the highest confidence from multiple point cloud clusters includes:

[0034] Based on the size of each point cloud cluster in the bounding box Box1 and its relationship with the center point P of the bounding box Box1 c1 The weighted values ​​of each point cloud cluster are obtained by weighting the distances (x1, y1, z1).

[0035] Select the point cloud cluster with the highest weighted value.

[0036] Furthermore, determine the three-dimensional position P of the center of the three-dimensional bounding box Box2 within Box1. c2 (x2,y2,z2);

[0037] The P c2 (x2,y2,z2) and Box2 constitute the final 3D detection result.

[0038] Further, step S4 includes:

[0039] Initialize the extended Kalman filter (EKF) and set the initial state and covariance matrix of the target;

[0040] Based on the initial state and covariance matrix of the target, the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results are weighted and fused to obtain the fused target state information.

[0041] The state transition model and process noise are used to predict the next state of the target and update the covariance matrix.

[0042] The predicted next state of the target is compared with the observed value of the target in the actual 3D detection, the Kalman gain is calculated, and the target state and covariance matrix are updated to output the target state estimate and covariance matrix at time k+1.

[0043] The two-dimensional and three-dimensional detection results are processed in parallel at each time step, and the detection results are fused and updated using EKF to continuously track and locate the target.

[0044] Output the target ID, the target's position in the world coordinate system, the target's category, and the point cloud bounding box (Box2).

[0045] Furthermore, the pixel coordinates in the two-dimensional target detection and tracking results, and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results are weighted and fused as follows:

[0046]

[0047] in, The target state information is obtained after weighted fusion; α and β are the three-dimensional and two-dimensional detection weight coefficients, respectively; The pixel coordinates of the 8 bounding box vertices of Box2 in 3D detection; These are the target pixel coordinates predicted for two-dimensional tracking.

[0048] Furthermore, the geometric prior information includes the shape and size of the target category, wherein the size includes the prior length, width, and height of the target.

[0049] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0050] 1. Based on point cloud observation and two-dimensional detection results, this invention determines the accurate location of the target point cloud using two detection boxes, Box1 and Box2, of different sizes. By utilizing the precise three-dimensional spatial information provided by the lidar and the rich two-dimensional image details provided by the binocular camera, this invention can achieve accurate identification and positioning of the target, especially in complex environments, significantly improving detection accuracy compared to a single sensor system.

[0051] 2. By fusing two-dimensional image information and three-dimensional point cloud information for target tracking prediction, the target state information obtained by weighted fusion is used to correct the prediction process of the extended Kalman filter and optimize the tracking prediction results. By fusing two-dimensional and three-dimensional detection results through the extended Kalman filter, this invention can maintain stable tracking when the target moves rapidly or is partially occluded, and reduce target tracking loss caused by environmental interference.

[0052] 3. Based on two-dimensional detection, this invention integrates three-dimensional point cloud information for target detection, which can make up for the lack of depth scale when using a camera alone for detection, and obtain more accurate point cloud target positioning information.

[0053] 4. This invention directly uses the raw point cloud data after voxel downsampling, eliminating the need for projection processing of the point cloud and preserving more target features. It employs a non-learning method for 3D target detection, using two 3D detection boxes for clustering to quickly locate the target position, resulting in higher real-time performance.

[0054] 5. The method of this invention is designed with real-time processing in mind. Through optimized data processing flow and algorithms, it ensures rapid data processing and response even in scenarios where unmanned systems move rapidly or targets frequently appear and disappear. By dynamically adjusting the length, width, and height dimensions of the 3D bounding box Box1 according to the number of targets and scene complexity within the detection area, this invention can more flexibly adapt to different detection environments, optimize detection performance, and reduce false detections and missed detections.

[0055] 6. By accurately fusing two-dimensional and three-dimensional data and effectively predicting target tracking, this invention reduces the demand for computing resources, improves the system's efficiency, and makes the method applicable to resource-constrained unmanned systems.

[0056] 7. By using the Hungarian algorithm to optimize the target matching process, this invention can accurately distinguish and track multiple targets in multi-target tracking scenarios, and can maintain accurate tracking even when targets are close to or intersect each other.

[0057] 8. The method of the present invention is not limited to a specific type of environment or target, but has wide applicability and can adapt to a variety of different application scenarios, and can be flexibly applied across scenarios, such as autonomous driving, robot navigation, intelligent monitoring, etc.

[0058] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0059] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0060] Figure 1 This is a flowchart of a multimodal target recognition and tracking prediction method based on lidar and binocular camera in an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of the two-dimensional target tracking and prediction process in an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of two-stage target detection in three-dimensional target detection in an embodiment of the present invention. Detailed Implementation

[0063] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0064] This invention aims to solve the technical problems of missing depth scale in 2D detection and inefficient point cloud processing in 3D detection by improving the fusion method of 2D and 3D target detection. This enables accurate identification, tracking, and localization of any target in large-scale scenes.

[0065] In this invention, a binocular camera provides rich two-dimensional image information, and a lidar provides high-precision three-dimensional spatial information. The combined use of the two can significantly improve the environmental perception capability and target tracking accuracy of the unmanned system.

[0066] In this invention, binocular cameras and lidar serve as perception modules for unmanned systems or robots. Exemplarily, these sensors are installed on unmanned system platforms such as driverless cars, drones, automated guided vehicles, and service robots.

[0067] A specific embodiment of the present invention discloses a multimodal target recognition and tracking prediction method based on lidar and binocular cameras, such as... Figure 1 As shown, it includes the following steps:

[0068] Step S1: Preprocess the target 2D image acquired by the binocular camera and the target point cloud data acquired by the lidar to obtain the preprocessed 2D image and point cloud data.

[0069] Step S2: Perform two-dimensional target detection and tracking based on the preprocessed two-dimensional image to obtain the two-dimensional target detection and tracking results;

[0070] Step S3: Perform three-dimensional target detection based on the preprocessed point cloud data and the two-dimensional target detection and tracking results to obtain three-dimensional detection results;

[0071] Step S4: Based on the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the target point cloud boundary in the three-dimensional detection results, perform target fusion tracking prediction to obtain the target ID, location, category, and target point cloud boundary.

[0072] Step S1 is divided into steps S11-S12, specifically.

[0073] Step S11: The binocular camera continuously acquires two-dimensional images of the target, performs preprocessing, and obtains a preprocessed two-dimensional image.

[0074] The system continuously uses a binocular camera to capture two-dimensional images of the target and performs image preprocessing on these images.

[0075] The 2D image size is scaled to the input layer size of the first-stage object detection model and used as the input for subsequent 2D object detection steps.

[0076] For example, the two-dimensional image size is scaled down to 640×480 pixels.

[0077] Step S12: Simultaneously, use the lidar to acquire target point cloud data and perform point cloud data preprocessing to obtain preprocessed point cloud data.

[0078] The system receives raw point cloud data from the lidar and preprocesses it by performing voxel grid downsampling to obtain preprocessed point cloud data. The preprocessed point cloud data is essentially the original point cloud data after voxel grid downsampling; it's still the original point cloud, just with a reduced number of points.

[0079] The acquired raw point cloud data is downsampled using voxel meshes to reduce the amount of raw point cloud data while retaining important spatial information.

[0080] The purpose of step S1 is to preprocess and downsample the binocular camera images and lidar point cloud data to provide standardized, high-quality data input for subsequent multimodal target recognition and tracking prediction.

[0081] Step S2 includes:

[0082] The preprocessed multi-frame two-dimensional images are sequentially input into a one-stage detection model for two-dimensional target detection to obtain two-dimensional target detection results; the two-dimensional target detection results include the targets included in each frame of two-dimensional images, the category of the targets, and the pixel coordinates and detection confidence of the corresponding targets;

[0083] Based on the detection confidence, the two-dimensional target detection results corresponding to each frame of image are divided into high-confidence targets, low-confidence targets, and targets between high-confidence and low-confidence.

[0084] The pixel coordinates of the high-confidence targets are input into a Kalman filter. The Kalman filter predicts the new position of the targets in the current frame based on the pixel coordinates of the historical positions of each high-confidence target, thereby obtaining the predicted detection box position of the high-confidence targets and defining an ID for each high-confidence target.

[0085] Based on the preprocessed 2D image of the current frame, obtain the true detection bounding box positions of all targets in the current frame;

[0086] Based on the real detection boxes in the current frame and the predicted detection boxes in the previous frame, and combined with the new targets detected based on the low-confidence targets, the Hungarian algorithm is used to continuously track the same target, so as to obtain the two-dimensional target detection and tracking results.

[0087] Step S2 is divided into steps S21-S22.

[0088] Step S21: Perform two-dimensional target detection based on the preprocessed two-dimensional image of the target.

[0089] The preprocessed multi-frame two-dimensional images are sequentially input into a one-stage detection model for two-dimensional target detection to obtain the two-dimensional target detection results.

[0090] For example, the one-stage object detection model uses the YOLO (You Only Look Once) model. The YOLO model is integrated on the computing device.

[0091] The two-dimensional detector is initialized. During the initialization process, a detection result class is maintained to store the two-dimensional target detection results output by the first-stage target detection model.

[0092] This category includes the target's class, four pixel coordinates (x, y), and detection confidence. The pixel coordinates are the coordinates of the four intersection points of the detected target's bounding box (x, y). min ,y min ), (x max ,y min ), (x min ,y max ), (x max ,y max The confidence level of target detection is mainly evaluated by whether there is an object in the target bounding box and the intersection-union ratio of the predicted bounding box and the true bounding box.

[0093] The two-dimensional detector is a soft module integrated on a computing device.

[0094] Exemplarily, the present invention adopts the YOLOv8 version, which is integrated on a computing device.

[0095] The one-stage object detection model adopts YOLOv8 combined with NMS;

[0096] The YOLOv8 first performs an initial detection on the preprocessed two-dimensional image to obtain multiple bounding boxes and the corresponding object classes, pixel coordinates, and confidence levels;

[0097] The NMS performs post-processing on the output of the YOLOv8 to suppress redundant bounding boxes. By retaining the bounding box with the highest confidence level and removing other bounding boxes with high overlap with it, a two-dimensional object detection result is obtained.

[0098] The YOLO model first takes the preprocessed target two-dimensional image as input, then performs model inference on the input image, and performs non-maximum suppression NMS (Non-Maximum Suppression) on the basis of the model output results to screen out the final two-dimensional object detection result. The output two-dimensional object detection result includes the class of the object, 4 pixel coordinates, and detection confidence level, which are saved and maintained in the detection result class.

[0099] According to the detection confidence level, it is divided into high-confidence objects (confidence > m) and low-confidence objects (confidence < n), as well as objects between high-confidence and low-confidence;

[0100] m and n are preset values, and m > n; Exemplarily, m = 0.6 and n = 0.5.

[0101] The non-maximum suppression NMS is integrated on a computing device.

[0102] The YOLO model is an object detection model mainly used in the field of object detection. It can identify the object classes and positions in an image through one scan. It is a real-time object detection algorithm that can provide a high processing speed while ensuring the detection quality. The characteristics of the YOLO model include its simplicity and efficiency. It does not need to first find possible object regions and then classify them like traditional object detection methods. The YOLO model outputs the class of the object, the detection bounding box, the detection confidence level, and the class.

[0103] Non-Maximum Suppression (NMS) is a post-processing technique used to filter and optimize the output of the YOLO model. In object detection, NMS addresses the problem of multiple bounding boxes surrounding the same object. NMS suppresses redundant bounding boxes by retaining the bounding box with the highest confidence and removing other bounding boxes with high overlap, thereby reducing redundancy and false detections. After NMS processing, redundant bounding boxes are effectively removed, retaining only those most likely to contain the object.

[0104] The relationship between YOLO and NMS:

[0105] (1) Complementarity: YOLO may generate multiple overlapping bounding boxes when detecting targets, especially when the target is partially occluded in the image or there are similar objects nearby. NMS, as a post-processing step of YOLO output, helps YOLO improve detection accuracy and reduce redundancy by suppressing unnecessary bounding boxes.

[0106] (2) Process sequence: In the YOLO object detection process, the preprocessed 2D image is first processed using the YOLO model to obtain a series of bounding boxes and corresponding confidence scores. Then, NMS is applied to these outputs to remove overlapping bounding boxes and retain the bounding boxes most likely to contain the object.

[0107] Function: NMS helps improve the overall quality of YOLO detection results, reduce false positives and duplicate detections, thereby improving the accuracy and recall of the YOLO model.

[0108] The output of the 2D target detection includes: target category, 4 pixel coordinates, and detection confidence.

[0109] The purpose of using a stereo camera for 2D target detection is to provide unmanned systems with basic information such as the target's category, pixel coordinates, and detection confidence, so that subsequent steps can perform accurate target tracking and precise positioning in 3D space.

[0110] Step S22: Track and predict the two-dimensional target.

[0111] Two-dimensional target tracking prediction adopts a target matching and tracking method based on detection confidence.

[0112] like Figure 2 As shown, the two-dimensional target detection results of a single frame image are used as input to continuously track and match high-confidence targets;

[0113] Low-confidence targets are saved. When a subsequently tracked target is lost, the currently saved low-confidence target is retrieved again for re-identification and tracking (e.g., ...). Figure 2 (As shown by the dashed line in the middle);

[0114] At most, low-confidence targets from the most recent P-frame detection results are saved; for example, P = 30.

[0115] Unclassified categories (i.e., targets with confidence levels between m and n) are not processed in the current frame. For example, unclassified categories, such as those with confidence levels between 0.6 and 0.5, are not processed in the current frame.

[0116] (1) For high confidence thresholds (e.g., >0.6):

[0117] Targets with a detection confidence level greater than 0.6 are considered sufficiently reliable and can be directly used for target tracking. This is because the detection results for these targets have a high probability of being correct, and therefore can be directly used in the subsequent tracking process.

[0118] (2) For low confidence thresholds (e.g., <0.5):

[0119] Targets with a confidence level less than 0.5 are considered unreliable and insufficient for direct tracking. However, the detection results of these targets are not completely ignored but are saved. When a target is lost during tracking (i.e., it cannot be detected in subsequent frames), these low-confidence targets can be used as potential candidate targets for re-initializing or resuming tracking.

[0120] (3) For the medium confidence range (between 0.5 and 0.6):

[0121] Targets with a confidence level between 0.5 and 0.6 are typically in a "gray area," neither directly used for tracking nor completely ignored. This range has many false positives, and they occur frequently. Generally, using targets with lower confidence levels for re-identification is more efficient.

[0122] Targets with confidence levels between 0.5 and 0.6 are not used for tracking and detection to strike a balance between tracking accuracy and computational resources, while avoiding the introduction of too many potential erroneous tracking errors. This design decision is typically based on considerations of the needs and constraints of a specific application scenario.

[0123] This step, based on the target list classified by confidence level, yields a list including high-confidence targets and low-confidence targets. This prepares the target list for subsequent tracking and prediction, and determines which targets need to be tracked.

[0124] The initial target tracking and prediction is performed. In this part, a Kalman filter is introduced. The pixel coordinates of high-confidence targets in historical frames are input into the Kalman filter. Based on the pixel coordinate data of each target's historical position, the Kalman filter predicts the new position of the target in the current frame, and obtains the predicted detection box positions of all targets in the current frame in the next frame and their corresponding target IDs.

[0125] Based on each ground truth detection box in the current frame and each predicted detection box in the previous frame, the Hungarian algorithm is used to calculate the IOU (Intersection over Union) value between each ground truth detection box in the current frame and each predicted detection box in the previous frame.

[0126] The target of the predicted detection box in the previous frame with the largest IOU value with the real detection box is taken as the best matching object, and the target ID of the best matching object is inherited to achieve continuous tracking and matching of the same target between frames.

[0127] If a target in the current frame cannot be found to match a target in the previous frame, it is determined that a target has been lost. In this case, all low-confidence targets in the most recent P-frames are retrieved and matched again with the target in the current frame that failed to match using the Hungarian algorithm. If the match is successful, the target in the current frame will inherit the ID of the low-confidence target, thus enabling the re-identification of the lost target.

[0128] If a matching target still cannot be found, assign a new ID to the target in the current frame and treat it as a newly appearing target.

[0129] The process of continuously acquiring the next frame from the stereo camera as the current frame is repeated. After completing the 2D object detection for that frame,

[0130] The true bounding box positions of all targets in the current frame, and the predicted bounding box positions of all targets in the previous frame;

[0131] The IOU features of each detection box in the current frame and each predicted detection box in the previous frame are calculated sequentially to measure the degree of matching between targets in different frames.

[0132] Obtain the target matching result for the (k+1)th frame. Output all identified targets for the (k+1)th frame, with information for each target including: target ID, 4 pixel coordinates, and detection confidence score.

[0133] The purpose of step S2 is to achieve accurate identification, localization and continuous tracking of targets in two-dimensional images by combining target detection of the YOLO model and prediction of the Kalman filter, and to optimize the target matching process, so as to improve the target tracking performance of unmanned systems in complex environments.

[0134] Step S3, specifically.

[0135] On the newly received i-frame preprocessed point cloud data, and the two-dimensional target detection and tracking results, three-dimensional target detection is performed, including:

[0136] Predefined prior geometric information for each target category;

[0137] Based on the pixel coordinates of the four intersection points of each target in the two-dimensional target detection and tracking results, calculate the length and width of the target's two-dimensional detection box in pixel coordinates;

[0138] Based on the prior geometric information and the length and width of the target's 2D detection box in pixel coordinates, the left side of the target pixel is converted into a 3D coordinate position P through similar coordinate transformation. c1 (x1,y1,z1); with P c1 Using (x1, y1, z1) as the center coordinates, and based on the preset length, width, and height according to the number of targets in the detection area and the scene complexity, a 3D bounding box Box1 is generated;

[0139] Traverse all points in the current frame's point cloud. If a point is located within the bounding box Box1, it is classified as a point to be detected.

[0140] Euclidean clustering is performed on all the points to be detected in the bounding box Box1 to obtain multiple point cloud clusters. The point cloud cluster with the highest confidence is selected from the multiple point cloud clusters. The boundary of the selected point cloud cluster is used as the reference, and the length, width and height of the prior geometric information of the target category are used to construct the three-dimensional bounding box Box2 to obtain the three-dimensional detection result.

[0141] The three-dimensional detection result is a three-dimensional bounding box (Box2).

[0142] The prior information for each target category is pre-defined and set.

[0143] The geometric prior information includes the shape and size of the target category, where the size includes the target's prior length, width, and height.

[0144] The three-dimensional target detection method in this invention is performed in the original point cloud environment of voxel mesh downsampling, and is mainly divided into two detection stages: the preliminary detection stage and the secondary detection stage.

[0145] Save the latest received i-frame point cloud and perform voxel grid downsampling maintenance on it.

[0146] i is a preset value; for example, i is 5. Processing is performed every 5 frames of point cloud data. Specifically, each frame of point cloud data is first downsampled using a voxel grid, and then the point cloud data after 5 frames of voxel downsampling is retained.

[0147] First, the initial testing phase:

[0148] The preliminary detection is based on downsampled point cloud data and two-dimensional target detection and tracking results.

[0149] This step uses a similar coordinate transformation method to make a preliminary prediction of the target's 3D information within the processed point cloud scene.

[0150] Input the 2D target detection and tracking results (pixel coordinates of 4 intersection points), including the length and width of the 2D detection box of the target in pixel coordinates. Calculate the length and width of the 2D detection box in pixel coordinates using the pixel coordinates of the 4 intersection points of the 2D detection box.

[0151] The 3D coordinate position P of the target object is output after similar coordinate transformation. c1 (x1,y1,z1), as shown in formula (1).

[0152]

[0153] Where z represents the depth scale information of the target, i.e., z1; w and h are the prior width and prior height of the target, respectively; w bbox h bbox f represents the width and length of the two-dimensional detection bounding box of the target, respectively; x f y These are the focal lengths in the x and y directions, respectively; α is the width and height weighting coefficient; C lidar R represents the target's coordinates in the radar coordinate system. cam2pix R is the rotation matrix from the pixel coordinate system to the camera coordinate system. lidar2cam p is the rotation matrix from the camera coordinate system to the radar coordinate system; pix The pixel coordinates of the center point of the target detection box are converted to radar coordinates to obtain (x1, y1); p cam2lidar Q is the offset vector from the camera coordinate system to the radar coordinate system; odom p is a quaternion rotation matrix from the lidar coordinate system to the world coordinate system; odom C represents the position vector of the lidar in the world coordinate system. world This refers to the coordinates of the target in the world coordinate system.

[0154] Furthermore, a large 3D bounding box, Box1, is used to define the 3D detection information. The boundary information of the bounding box, including length, width, and height, is preset, such as... Figure 3 As shown in the dashed box.

[0155] If there are many targets in the detection area or the scene is complex, the length, width and height of the bounding box should be preset to be smaller.

[0156] Otherwise, if there are few targets in the detection area and the scene complexity is low, the length, width and height of the bounding box should be preset to be larger.

[0157] Adjust the length L, width W, and height H of Box1 according to the target quantity N and scene complexity C, as shown in formula (2):

[0158]

[0159] Among them, L max W max and H max η, μ, and θ represent the maximum length, width, and height of the target scene, respectively; η, μ, and θ are the adjustment coefficients for length L, width W, and height H, respectively; f(N,C) is a function defined according to the actual needs of the number of targets and scene complexity. It can be a simple linear function or a deep learning-based model to more accurately reflect the impact of the number of targets and scene complexity on the size of Box1.

[0160] For Box1, with P c1 Using (x1, y1, z1) as the center coordinates, a 3D bounding box Box1 is generated with preset length, width and height.

[0161] The preliminary prediction of the target object's 3D coordinate position Pc1 and 3D bounding box Box1.

[0162] Based on the pixel coordinates of the four intersection points of the two-dimensional detection box, Pc1(x1,y1,z1) is obtained. At this time, the detection result of each target includes: target ID, target category, four two-dimensional pixel coordinates, detection confidence, Pc1(x1,y1,z1), and Box1.

[0163] Continue with the second testing phase:

[0164] Based on the 3D bounding box Box1 obtained in the preliminary detection phase and the 3D coordinate position P of the target object preliminarily predicted, c1 (x1,y1,z1).

[0165] Traverse all points in the current frame's point cloud. If a point is located within the bounding box Box1, it is classified as a point to be detected.

[0166] Perform Euclidean clustering on all points to be detected in bounding box Box1 to obtain multiple point cloud clusters, such as... Figure 3 The shaded areas are shown in the image.

[0167] The step of selecting the point cloud cluster with the highest confidence from multiple point cloud clusters includes:

[0168] Based on the size of each point cloud cluster in the bounding box Box1 and its relationship with the center point P of the bounding box Box1 c1 The weighted values ​​of each point cloud cluster are obtained by weighting the distances (x1, y1, z1).

[0169] Select the point cloud cluster with the highest weighted value.

[0170] Based on the size of the clustered point cloud clusters (the number of points in the point cloud) and their distance from the center point of the bounding box Box1, a weighted calculation is performed to select the point cloud clusters with the highest confidence. Figure 3 The top right corner is marked with a P c2 The point cloud cluster centered on the point cloud is shown in Equation (3).

[0171]

[0172] Among them, P j For the j-th point cloud cluster; S j For point cloud cluster P j The number of points in C; j Represents point cloud cluster P j The center point; D j C is the center point of the point cloud cluster. j Distance from the center point Box1 of the bounding box; The center point C of the point cloud cluster is respectively j The coordinates of the bounding box Box1 and the center point coordinates; W j Represents point cloud cluster P j The credibility; λ is the weighting parameter.

[0173] Based on the boundary of the filtered point cloud cluster, maintain a smaller 3D bounding box Box2, such as Figure 3 As shown in the solid line frame, with the center P of this three-dimensional bounding box c2 (x2,y2,z2) is the final detection and localization result of the target.

[0174] Determine the 3D position P of the center of the 3D bounding box Box2 within Box1. c2 (x2,y2,z2);

[0175] The P c2 (x2,y2,z2) and Box2 constitute the final 3D detection result.

[0176] The final 3D detection result includes the target's 3D position and the corresponding bounding box Box2, where the data for Box2 consists of the coordinates of the 8 vertices of the cube Box2.

[0177] Step S4 includes:

[0178] Initialize the extended Kalman filter (EKF) and set the initial state and covariance matrix of the target;

[0179] Based on the initial state and covariance matrix of the target, the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results are weighted and fused to obtain the fused target state information.

[0180] The state transition model and process noise are used to predict the next state of the target and update the covariance matrix.

[0181] The predicted next state of the target is compared with the observed value of the target in the actual 3D detection, the Kalman gain is calculated, and the target state and covariance matrix are updated to output the target state estimate and covariance matrix at time k+1.

[0182] The two-dimensional and three-dimensional detection results are processed in parallel at each time step, and the detection results are fused and updated using EKF to continuously track and locate the target.

[0183] Output the target ID, the target's position in the world coordinate system, the target's category, and the point cloud bounding box (Box2).

[0184] By fusing 2D image information (pixel coordinates of the four intersection points of the 2D bounding box) and 3D point cloud information (pixel coordinates of the eight vertices of Box2), the target position is tracked and predicted again in the point cloud scene.

[0185] (1) First, initialize the extended Kalman filter EKF.

[0186] An extended Kalman filter (EKF) is introduced into the secondary detection stage of 3D object detection as a parallel process of the secondary detection stage.

[0187] The target fusion tracking and prediction system is a software module integrated on the computing device. Before the operation begins, the Kalman filter is initialized, and the initial state and covariance matrix of the target are set.

[0188]

[0189] in, It is the predicted state of the target at time k. It is the prediction covariance matrix at time k; x init P is the preset initial value for the target state. init is the preset initial value for the covariance matrix, and k is the timestamp.

[0190] (2) Weighted fusion of target states

[0191] The target state information for two-dimensional tracking prediction (pixel coordinates of the four intersection points of the two-dimensional detection box) and the three-dimensional detection state information (pixel coordinates of the eight intersection points of Box2).

[0192] After completing the secondary detection phase and identifying and locating the target object, the Kalman filter continuously maintains the target position (pixel coordinates). Before the EKF prediction phase, the target state information (4 pixel coordinates) of the 2D tracking prediction and the 3D detection state information (the point cloud coordinates of the 8 corner points of the Box2 bounding box) are weighted and fused to serve as the target state information for the prediction phase.

[0193] The pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results are weighted and fused as shown in formula (5):

[0194]

[0195] in, The target state information is obtained after weighted fusion; α and β are the three-dimensional and two-dimensional detection weight coefficients, respectively; The pixel coordinates of the 8 bounding box vertices of Box2 in 3D detection; These are the target pixel coordinates predicted for two-dimensional tracking.

[0196] The fused target state information is shown in formula (6).

[0197]

[0198] in, Represents the target state information of 3D detection This represents the target state information for two-dimensional tracking and prediction. This represents the weighted fusion of the target state information (a vector formed by concatenating the 4 pixel coordinates of the 2D detection box and the pixel coordinates of the 8 vertices of Box1); (x min ,x max ,y min ,y max ,z min ,z max ) represents the specific coordinates in the 3D detection state information (using the x, y, z range of the bounding box to maintain the coordinates of 8 bounding box corner points); The specific coordinates in the target state information of the two-dimensional tracking prediction are represented (the coordinates of the intersection of the four detection boxes are maintained using the x and y ranges of the detection boxes);

[0199] This part obtains the fused target state information;

[0200] (3) Predict the next state of the target.

[0201] Based on the fused target state information, the next state of the target is predicted using the state transition model and process noise, and the covariance matrix is ​​updated as shown in Equation (7).

[0202]

[0203] in, F is the predicted state of the target at time k+1. k It is the state transition matrix at time k. It is the prediction covariance matrix at time k+1, Q k It is the process noise covariance matrix at time k.

[0204] Output the predicted next state of the target.

[0205] (4) Update the target status.

[0206] Based on the predicted next target state and the actual target observations in 3D detection.

[0207] After predicting the next target state, the predicted value is compared with the observed value of the target in the actual 3D detection, and the Kalman gain is calculated.

[0208] Obtain the Kalman gain K k+1 Then, the target state and covariance matrix are updated as shown in Equation (8).

[0209]

[0210] Among them, H k+1 It is the observation matrix. For H k+1 The transpose matrix, R k+1 It is the observation noise covariance matrix, K k+1 It is the Kalman gain.

[0211] The state estimate of the target at time k+1 is shown in Equation (9).

[0212]

[0213] Where, x k+1 It is the target state estimate updated after time k+1, P k+1 It is the covariance matrix updated after time k+1, z k+1 I represents the target observation data at time k+1 (i.e., the target's position in the global coordinate system at time k+1), and I is the identity matrix.

[0214] We obtain the state estimate of the target at time k+1 and the covariance matrix at time k+1.

[0215] (5) Parallel processing and continuous tracking.

[0216] Output results based on 2D and 3D detection.

[0217] Each detector and filter works in parallel, with the detectors performing both two-dimensional and three-dimensional detection of the target;

[0218] Simultaneously, the extended Kalman filter (EKF) completes the fusion and updating of the output results of the two detectors.

[0219] By repeatedly predicting and updating the target at each time step, the target can be continuously tracked and located.

[0220] The final output includes the target's position in the world coordinate system, the target's category, the point cloud bounding box (Box2), and the target ID at the current moment.

[0221] In summary, the multimodal target recognition and tracking prediction method based on lidar and binocular camera of this invention has the following beneficial effects:

[0222] 1. Based on point cloud observation and two-dimensional detection results, this invention determines the accurate location of the target point cloud using two detection boxes, Box1 and Box2, of different sizes. By utilizing the precise three-dimensional spatial information provided by the lidar and the rich two-dimensional image details provided by the binocular camera, this invention can achieve accurate identification and positioning of the target, especially in complex environments, significantly improving detection accuracy compared to a single sensor system.

[0223] 2. By fusing two-dimensional image information and three-dimensional point cloud information for target tracking prediction, the target state information obtained by weighted fusion is used to correct the prediction process of the extended Kalman filter and optimize the tracking prediction results. By fusing two-dimensional and three-dimensional detection results through the extended Kalman filter, this invention can maintain stable tracking when the target moves rapidly or is partially occluded, and reduce target tracking loss caused by environmental interference.

[0224] 3. Based on two-dimensional detection, this invention integrates three-dimensional point cloud information for target detection, which can make up for the lack of depth scale when using a camera alone for detection, and obtain more accurate point cloud target positioning information.

[0225] 4. This invention directly uses the raw point cloud data after voxel downsampling, eliminating the need for projection processing of the point cloud and preserving more target features. It employs a non-learning method for 3D target detection, using two 3D detection boxes for clustering to quickly locate the target position, resulting in higher real-time performance.

[0226] 5. The method of this invention is designed with real-time processing in mind. Through optimized data processing flow and algorithms, it ensures rapid data processing and response even in scenarios where unmanned systems move rapidly or targets frequently appear and disappear. By dynamically adjusting the length, width, and height dimensions of the 3D bounding box Box1 according to the number of targets and scene complexity within the detection area, this invention can more flexibly adapt to different detection environments, optimize detection performance, and reduce false detections and missed detections.

[0227] 6. By accurately fusing two-dimensional and three-dimensional data and effectively predicting target tracking, this invention reduces the demand for computing resources, improves the system's efficiency, and makes the method applicable to resource-constrained unmanned systems.

[0228] 7. By using the Hungarian algorithm to optimize the target matching process, this invention can accurately distinguish and track multiple targets in multi-target tracking scenarios, and can maintain accurate tracking even when targets are close to or intersect each other.

[0229] 8. The method of the present invention is not limited to a specific type of environment or target, but has wide applicability and can adapt to a variety of different application scenarios, and can be flexibly applied across scenarios, such as autonomous driving, robot navigation, intelligent monitoring, etc.

[0230] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal target recognition and tracking prediction method based on lidar and binocular camera, characterized in that, Includes the following steps: Step S1: Preprocess the target 2D image acquired by the binocular camera and the target point cloud data acquired by the lidar to obtain the preprocessed 2D image and point cloud data. Step S2: Perform two-dimensional target detection and tracking based on the preprocessed two-dimensional image to obtain the two-dimensional target detection and tracking results; Step S3: Perform three-dimensional target detection based on the preprocessed point cloud data and the two-dimensional target detection and tracking results to obtain three-dimensional detection results; Step S4: Based on the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the target point cloud boundary in the three-dimensional detection results, perform target fusion tracking prediction to obtain the target ID, location, category and target point cloud boundary. Step S4 includes: Initialize the extended Kalman filter (EKF) and set the initial state and covariance matrix of the target; Based on the initial state and covariance matrix of the target, the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results are weighted and fused to obtain the fused target state information. The state transition model and process noise are used to predict the next state of the target and update the covariance matrix. The predicted next state of the target is compared with the observed value of the target in the actual 3D detection, the Kalman gain is calculated, and the target state information and covariance matrix are updated to output the target state estimate and covariance matrix at time k+1. The two-dimensional and three-dimensional detection results are processed in parallel at each time step, and the detection results are fused and updated using EKF to continuously track and locate the target. Output the target ID, the target's position in the world coordinate system, the target's category, and the point cloud bounding box (Box2).

2. The method according to claim 1, characterized in that, Step S2 includes: The preprocessed multi-frame two-dimensional images are sequentially input into a one-stage detection model for two-dimensional target detection to obtain two-dimensional target detection results; the two-dimensional target detection results include the targets included in each frame of two-dimensional images, the category of the targets, and the pixel coordinates and detection confidence of the corresponding targets; Based on the detection confidence, the two-dimensional target detection results corresponding to each frame of image are divided into high-confidence targets, low-confidence targets, and targets between high-confidence and low-confidence. The pixel coordinates of the high-confidence targets are input into a Kalman filter. The Kalman filter predicts the new position of the targets in the current frame based on the pixel coordinates of the historical positions of each high-confidence target, thereby obtaining the predicted detection box position of the high-confidence targets and defining an ID for each high-confidence target. Based on the preprocessed 2D image of the current frame, obtain the true detection bounding box positions of all targets in the current frame; Based on the real detection boxes in the current frame and the predicted detection boxes in the previous frame, and combined with the new targets detected based on the low-confidence targets, the Hungarian algorithm is used to continuously track the same target, so as to obtain the two-dimensional target detection and tracking results.

3. The method according to claim 2, characterized in that, The first-stage detection model uses YOLOv8 combined with NMS; The YOLOv8 first performs initial detection on the preprocessed 2D image to obtain multiple bounding boxes and their corresponding target categories, pixel coordinates, and confidence scores. The NMS performs post-processing on the YOLOv8 output to suppress redundant bounding boxes. By retaining the bounding box with the highest confidence and removing other bounding boxes that overlap significantly with it, a two-dimensional target detection result is obtained.

4. The method according to claim 2, characterized in that, Based on each ground truth detection box in the current frame and each predicted detection box in the previous frame, the IOU value between each ground truth detection box in the current frame and each predicted detection box in the previous frame is calculated sequentially using the Hungarian algorithm. The target of the predicted detection box in the previous frame with the largest IOU value with the real detection box is taken as the best matching object, and the target ID of the best matching object is inherited to achieve continuous tracking and matching of the same target between frames. If a target in the current frame cannot be found to match a target in the previous frame, it is determined that a target has been lost. In this case, all low-confidence targets in the most recent P-frames are retrieved and matched again with the target in the current frame that failed to match using the Hungarian algorithm. If the match is successful, the target in the current frame will inherit the ID of the low-confidence target, thus enabling the re-identification of the lost target. If a matching target still cannot be found, assign a new ID to the target in the current frame and treat it as a newly appearing target.

5. The method according to claim 1, characterized in that, On the newly received i-frame preprocessed point cloud data, and the two-dimensional target detection and tracking results, three-dimensional target detection is performed, including: Predefined prior geometric information for each target category; Based on the pixel coordinates of the four intersection points of each target in the two-dimensional target detection and tracking results, calculate the length and width of the target's two-dimensional detection box in pixel coordinates; Based on the prior geometric information and the length and width of the target's 2D detection box in pixel coordinates, the left side of the target pixel is converted into a 3D coordinate position P through similar coordinate transformation. c1 (x1,y1,z1); with P c1 Using (x1, y1, z1) as the center coordinates, and based on the preset length, width, and height according to the number of targets in the detection area and the scene complexity, a 3D bounding box Box1 is generated; Traverse all points in the current frame's point cloud. If a point is located within the bounding box Box1, it is classified as a point to be detected. Euclidean clustering is performed on all the points to be detected in the bounding box Box1 to obtain multiple point cloud clusters. The point cloud cluster with the highest confidence is selected from the multiple point cloud clusters. The boundary of the selected point cloud cluster is used as the reference, and the length, width and height of the prior geometric information of the target category are used to construct the three-dimensional bounding box Box2 to obtain the three-dimensional detection result. The three-dimensional detection result is a three-dimensional bounding box (Box2).

6. The method according to claim 5, characterized in that, The step of selecting the point cloud cluster with the highest confidence from multiple point cloud clusters includes: Based on the size of each point cloud cluster in the bounding box Box1 and its relationship with the center point P of the bounding box Box1 c1 The weighted values ​​of each point cloud cluster are obtained by weighting the distances (x1, y1, z1). Select the point cloud cluster with the highest weighted value.

7. The method according to claim 5, characterized in that, Determine the 3D position P of the center of the 3D bounding box Box2 within Box1. c2 (x2,y2,z2); The P c2 (x2,y2,z2) and Box2 constitute the final 3D detection result.

8. The method according to claim 1, characterized in that, The pixel coordinates in the two-dimensional target detection and tracking results, and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results are weighted and fused as follows: ； in, The target state information after weighted fusion; and These are the detection weight coefficients for three-dimensional and two-dimensional detection, respectively. The pixel coordinates of the 8 bounding box vertices of Box2 in 3D detection; These are the target pixel coordinates predicted for two-dimensional tracking.

9. The method according to claim 6, characterized in that, The prior geometric information includes the shape and size of the target category, where the size includes the target's prior length, width, and height.

Images